Original: P ROJECTED L ATENT M ARKOV C HAIN M ONTE C ARLO : C ONDITIONAL S AMPLING OF N ORMALIZING F LOWS
Revised: P ROJECTED L ATENT M ARKOV C HAIN M ONTE C ARLO : C ONDITIONAL S AMPLING OF N ORMALIZING F LOWS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce Projected Latent Markov Chain Monte Carlo (PL-MCMC), a technique for sampling from the exact conditional distributions learned by normalizing ﬂows.
Revised: We introduce Projected Latent Markov Chain Monte Carlo (PL-MCMC), a technique for sampling from the exact conditional distributions learned by normalizing ﬂows.

Original: As a conditional sampling method, PL-MCMC enables Monte Carlo
Revised: As a conditional sampling method, PL-MCMC enables Monte Carlo

Original: Expectation Maximization (MC-EM) training of normalizing ﬂows from incomplete data.
Revised: Expectation Maximization (MC-EM) training of normalizing ﬂows from incomplete data.

Original: Through experimental tests applying normalizing ﬂows to missing data tasks for a variety of data sets, we demonstrate the efﬁcacy of PL-MCMC for conditional sampling from normalizing ﬂows
Revised: Through experimental tests applying normalizing ﬂows to missing data tasks for a variety of data sets, we demonstrate the efﬁcacy of PL-MCMC for conditional sampling from normalizing ﬂows

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Conditional sampling from modeled joint probability distributions offers a statistical framework for approaching tasks involving missing and incomplete data.
Revised: Conditional sampling from modeled joint probability distributions offers a statistical framework for approaching tasks involving missing and incomplete data.

Original: Deep generative models have demonstrated an exceptional capability for approximating the distributions governing complex data.
Revised: Deep generative models have demonstrated an exceptional capability for approximating the distributions governing complex data.

Original: Brief analysis illustrates a fundamental guarantee for generative models: the inaccuracy (i.e. divergence from ground truth) of a generative model’s approximated joint distribution upper bounds the expected inaccuracies of the conditional distributions known by the model, as shown in Appendix A.
Revised: Brief analysis illustrates a fundamental guarantee for generative models: the inaccuracy (i.e. divergence from ground truth) of a generative model’s approximated joint distribution upper bounds the expected inaccuracies of the conditional distributions known by the model, as shown in Appendix A.

Original: Although this guarantee holds for all generative models, specialized variants are typically used to approach tasks involving the conditional distributions among modeled variables, due to the difﬁculty in accessing the conditional distributions known by unspecialized generative models.
Revised: Although this guarantee holds for all generative models, specialized variants are typically used to approach tasks involving the conditional distributions among modeled variables, due to the difﬁculty in accessing the conditional distributions known by unspecialized generative models.

Original: Quite often, otherwise well trained generative models possess a capability for conditional inference that is regrettably locked away from our access.
Revised: Quite often, otherwise well trained generative models possess a capability for conditional inference that is regrettably locked away from our access.

Original: Normalizing ﬂow architectures like RealNVP (Dinh et al., 2014) and GLOW (Kingma & Dhariwal, 2018) have demonstrated accurate and expressive generative performance, showing great promise for application to missing data tasks.
Revised: Normalizing ﬂow architectures like RealNVP (Dinh et al., 2014) and GLOW (Kingma & Dhariwal, 2018) have demonstrated accurate and expressive generative performance, showing great promise for application to missing data tasks.

Original: Additionally, by enabling the calculation of exact likelihoods, normalizing ﬂows offer convenient mathematical properties for approaching exact conditional sampling.
Revised: Additionally, by enabling the calculation of exact likelihoods, normalizing ﬂows offer convenient mathematical properties for approaching exact conditional sampling.

Original: We are therefore motivated to develop techniques for sampling from the exact conditional distributions known by normalizing ﬂows.
Revised: We are therefore motivated to develop techniques for sampling from the exact conditional distributions known by normalizing ﬂows.

Original: In this paper, we propose Projected Latent Markov Chain
Revised: In this paper, we propose Projected Latent Markov Chain

Original: Monte Carlo (PL-MCMC), a conditional sampling technique that takes advantage of the convenient mathematical structure of normalizing ﬂows by deﬁning a Markov Chain within a ﬂow’s latent space and accepting proposed transitions based on the likelihood of the resulting imputation.
Revised: Monte Carlo (PL-MCMC), a conditional sampling technique that takes advantage of the convenient mathematical structure of normalizing ﬂows by deﬁning a Markov Chain within a ﬂow’s latent space and accepting proposed transitions based on the likelihood of the resulting imputation.

Original: In principle,
Revised: In principle,

Original: PL-MCMC enables exact conditional sampling without requiring specialized architecture, training history, or external inference machinery.
Revised: PL-MCMC enables exact conditional sampling without requiring specialized architecture, training history, or external inference machinery.

Original: Our Contributions: We prove that a Metropolis-Hastings implementation of our proposed PLMCMC technique is asymptotically guaranteed to sample from the exact conditional distributions known by any normalizing ﬂow satisfying very mild positivity and smoothness requirements.
Revised: Our Contributions: We prove that a Metropolis-Hastings implementation of our proposed PLMCMC technique is asymptotically guaranteed to sample from the exact conditional distributions known by any normalizing ﬂow satisfying very mild positivity and smoothness requirements.

Original: We then describe how to use PL-MCMC to perform Monte Carlo Expectation Maximization (MC-EM) training of normalizing ﬂows from incomplete training data.
Revised: We then describe how to use PL-MCMC to perform Monte Carlo Expectation Maximization (MC-EM) training of normalizing ﬂows from incomplete training data.

Original: To illustrate and demonstrate aspects of the technique, we perform a series of experiments utilizing PL-MCMC to complete CIFARimages, CelebA images, and MNIST digits affected by missing data.
Revised: To illustrate and demonstrate aspects of the technique, we perform a series of experiments utilizing PL-MCMC to complete CIFARimages, CelebA images, and MNIST digits affected by missing data.

Original: Finally, we perform a series of experiments training non-specialized normalizing ﬂows to model MNIST digits and continuous UCI datasets from incomplete training data to verify the performance of the proposed method.
Revised: Finally, we perform a series of experiments training non-specialized normalizing ﬂows to model MNIST digits and continuous UCI datasets from incomplete training data to verify the performance of the proposed method.

Original: Through these experimental results, we ﬁnd that PL-MCMC holds great practical promise for tasks requiring conditional sampling from normalizing ﬂows.
Revised: Through these experimental results, we ﬁnd that PL-MCMC holds great practical promise for tasks requiring conditional sampling from normalizing ﬂows.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: A conditional variant of normalizing ﬂows has been introduced by Lu & Huang (2020) to model a single conditional distribution between architecturally ﬁxed sets of conditioned and conditioning variables.
Revised: A conditional variant of normalizing ﬂows has been introduced by Lu & Huang (2020) to model a single conditional distribution between architecturally ﬁxed sets of conditioned and conditioning variables.

Original: While quite capable of learning individual conditional distributions, conditional variants do not enable arbitrary conditional sampling from a joint model.
Revised: While quite capable of learning individual conditional distributions, conditional variants do not enable arbitrary conditional sampling from a joint model.

Original: Richardson et al.
Revised: Richardson et al.

Original: (2020) concurrently train a deterministic inference network alongside a normalizing ﬂow for inferring missing data.
Revised: (2020) concurrently train a deterministic inference network alongside a normalizing ﬂow for inferring missing data.

Original: Although such an inference network can produce deterministic imputations consistent with the distributions learned by a normalizing ﬂow, it cannot stochastically sample from the conditional distributions known by the ﬂow.
Revised: Although such an inference network can produce deterministic imputations consistent with the distributions learned by a normalizing ﬂow, it cannot stochastically sample from the conditional distributions known by the ﬂow.

Original: Li et al.
Revised: Li et al.

Original: (2019) introduce shared parameter approximations that allow the derivation of approximate conditional normalizing ﬂows, though these approximations do not guarantee exact sampling from the conditional distributions of a particular joint model.
Revised: (2019) introduce shared parameter approximations that allow the derivation of approximate conditional normalizing ﬂows, though these approximations do not guarantee exact sampling from the conditional distributions of a particular joint model.

Original: Similar techniques for approaching missing data with other generative models, such as generative adversarial networks (GANs) and variational auto-encoders (VAEs), have been introduced with similar limitations (Ivanov et al., 2018;
Revised: Similar techniques for approaching missing data with other generative models, such as generative adversarial networks (GANs) and variational auto-encoders (VAEs), have been introduced with similar limitations (Ivanov et al., 2018;

Original: Yoon et al., 2018; Li et al., 2018).
Revised: Yoon et al., 2018; Li et al., 2018).

Original: A MCMC procedure for sampling from the conditional distributions of VAEs has been introduced by Rezende et al.
Revised: A MCMC procedure for sampling from the conditional distributions of VAEs has been introduced by Rezende et al.

Original: (2014) and reﬁned by Mattei & Frellsen (2018).
Revised: (2014) and reﬁned by Mattei & Frellsen (2018).

Original: This procedure fundamentally relies on the many-to-many relationship between the latent and modeled data spaces of VAEs, and cannot be directly applied to normalizing ﬂows, wherein the latent state uniquely determines (and is uniquely determined by) the modeled data state.
Revised: This procedure fundamentally relies on the many-to-many relationship between the latent and modeled data spaces of VAEs, and cannot be directly applied to normalizing ﬂows, wherein the latent state uniquely determines (and is uniquely determined by) the modeled data state.

Original: By following an unconstrained Markov Chain within the latent space, PL-MCMC mirrors this VAE conditional sampling procedure within the context of normalizing ﬂows.
Revised: By following an unconstrained Markov Chain within the latent space, PL-MCMC mirrors this VAE conditional sampling procedure within the context of normalizing ﬂows.

Original: PL-MCMC leverages the probabilistic structure learned by a normalizing ﬂow to produce efﬁcient
Revised: PL-MCMC leverages the probabilistic structure learned by a normalizing ﬂow to produce efﬁcient

Original: Markov Chains.
Revised: Markov Chains.

Original: The utility of the mathematical structure of normalizing ﬂows for approaching
Revised: The utility of the mathematical structure of normalizing ﬂows for approaching

Original: Monte Carlo estimation via independence sampling has been demonstrated by M ¨ uller et al.
Revised: Monte Carlo estimation via independence sampling has been demonstrated by M ¨ uller et al.

Original: The probabilistic structure of normalizing ﬂows has also been shown to improve unconditional sampling from externally deﬁned distributions by Hoffman et al.
Revised: The probabilistic structure of normalizing ﬂows has also been shown to improve unconditional sampling from externally deﬁned distributions by Hoffman et al.

Original: In using this learned structure, we believe that PL-MCMC receives many of the beneﬁts of Adaptive Monte Carlo methods (Haario et al., 2001; Foreman-Mackey et al., 2013; Zhu, 2019), as explained in Appendix B.
Revised: In using this learned structure, we believe that PL-MCMC receives many of the beneﬁts of Adaptive Monte Carlo methods (Haario et al., 2001; Foreman-Mackey et al., 2013; Zhu, 2019), as explained in Appendix B.

Original: PL-MCMC’s unconstrained Markov Chain through the latent space is not the only conceivable option for sampling from the conditional distributions described by normalizing ﬂows.
Revised: PL-MCMC’s unconstrained Markov Chain through the latent space is not the only conceivable option for sampling from the conditional distributions described by normalizing ﬂows.

Original: As normalizing ﬂows enable exact joint likelihood calculations, we could employ MCMC methods through the modeled data space.
Revised: As normalizing ﬂows enable exact joint likelihood calculations, we could employ MCMC methods through the modeled data space.

Original: Dinh et al.
Revised: Dinh et al.

Original: (2014) demonstrate a stochastic conditional MAP inference that can be adapted to implement the unadjusted Langevin algorithm (Fredrickson et al., 2006; Durmus et al., 2019) or the Metropolis adjusted Langevin algorithm (Grenander & Miller, 1994).
Revised: (2014) demonstrate a stochastic conditional MAP inference that can be adapted to implement the unadjusted Langevin algorithm (Fredrickson et al., 2006; Durmus et al., 2019) or the Metropolis adjusted Langevin algorithm (Grenander & Miller, 1994).

Original: A constrained
Revised: A constrained

Original: Hamiltonian Monte Carlo approach has also been introduced in the context of conditional sampling from generative models by Graham et al.
Revised: Hamiltonian Monte Carlo approach has also been introduced in the context of conditional sampling from generative models by Graham et al.

Original: MCMC methods restricted to the modeled data space approach the normalizing ﬂow as a sort of blackbox oracle to be used only for calculations regarding data likelihood.
Revised: MCMC methods restricted to the modeled data space approach the normalizing ﬂow as a sort of blackbox oracle to be used only for calculations regarding data likelihood.

Original: By design, PL-MCMC leverages the ﬂow’s one-to-one mapping between latent and modeled data spaces, thereby taking better advantage of the probabilistic structure learned by our normalizing ﬂows to perform conditional sampling.
Revised: By design, PL-MCMC leverages the ﬂow’s one-to-one mapping between latent and modeled data spaces, thereby taking better advantage of the probabilistic structure learned by our normalizing ﬂows to perform conditional sampling.

Original:  T HE PL-MCMC A PPROACH
Revised:  T HE PL-MCMC A PPROACH

Original: We consider a normalizing ﬂow between latent space Ξ and modeled data space X , deﬁning the mappings f θ : Ξ (cid:55)→ X and f − 1 θ : X (cid:55)→ Ξ .
Revised: We consider a normalizing ﬂow between latent space Ξ and modeled data space X , deﬁning the mappings f θ : Ξ (cid:55)→ X and f − 1 θ : X (cid:55)→ Ξ .

Original: This normalizing ﬂow imposes the probability density p f,θ ( x ) onto all modeled data values x ∈ X .
Revised: This normalizing ﬂow imposes the probability density p f,θ ( x ) onto all modeled data values x ∈ X .

Original: By the pairing ( x M ; x O ) , we denote the missing and observed portion of a modeled data value with joint density p f,θ ( x M ; x O ) under our normalizing ﬂow.
Revised: By the pairing ( x M ; x O ) , we denote the missing and observed portion of a modeled data value with joint density p f,θ ( x M ; x O ) under our normalizing ﬂow.

Original: Our goal is to sample from the conditional density described by the normalizing ﬂow, p f,θ ( x M | x O ) .3. T HE P ROJECTED
Revised: Our goal is to sample from the conditional density described by the normalizing ﬂow, p f,θ ( x M | x O ) .

Original: [Table]
Revised: [Table]

Revised: L ATENT T ARGET D ISTRIBUTION

Original: Rather than targeting the conditional distribution of missing values directly, PL-MCMC targets a distribution of latent variables that, after mapping through the ﬂow’s transformation, marginalizes to the desired conditional distribution.
Revised: Rather than targeting the conditional distribution of missing values directly, PL-MCMC targets a distribution of latent variables that, after mapping through the ﬂow’s transformation, marginalizes to the desired conditional distribution.

Original: Let the Markov Chain be composed of latent state ξ ∈ Ξ , mapping to the modeled data pair f θ ( ξ )
Revised: Let the Markov Chain be composed of latent state ξ ∈ Ξ , mapping to the modeled data pair f θ ( ξ )

Original: = ( y M ; y O ) .
Revised: = ( y M ; y O ) .

Original: Let q be an arbitrary smooth density over observed variables, y O .
Revised: Let q be an arbitrary smooth density over observed variables, y O .

Original: PL-MCMC targets the distribution whose (unnormalized) density within the modeled data space is q ( y O ) p
Revised: PL-MCMC targets the distribution whose (unnormalized) density within the modeled data space is q ( y O ) p

Original: f,θ ( y M | x O ) .
Revised: f,θ ( y M | x O ) .

Original: Fundamentally, PL-MCMC is a marginal MCMC method (Van Dyk, 2010) that uses the otherwise observed attributes, y O , as auxiliary working variables to take full advantage of the probabilistic structure learned by the normalizing ﬂow.
Revised: Fundamentally, PL-MCMC is a marginal MCMC method (Van Dyk, 2010) that uses the otherwise observed attributes, y O , as auxiliary working variables to take full advantage of the probabilistic structure learned by the normalizing ﬂow.

Original: 3. D ESCRIPTION OF M ETROPOLIS -H ASTINGS PL-MCMC A LGORITHM
Revised: 3. D ESCRIPTION OF M ETROPOLIS -H ASTINGS PL-MCMC A LGORITHM

Original: For a Metropolis-Hastings implementation of PL-MCMC, we introduce a transition kernel g ( ξ (cid:48) | ξ ) for generating proposal latent states.
Revised: For a Metropolis-Hastings implementation of PL-MCMC, we introduce a transition kernel g ( ξ (cid:48) | ξ ) for generating proposal latent states.

Original: We sample a new proposal latent vector ξ (cid:48) ∼ g ( ξ (cid:48) | ξ ) , mapping to the modeled data pair f θ ( ξ (cid:48) )
Revised: We sample a new proposal latent vector ξ (cid:48) ∼ g ( ξ (cid:48) | ξ ) , mapping to the modeled data pair f θ ( ξ (cid:48) )

Original: =
Revised: =

Original: ( y (cid:48)
Revised: ( y (cid:48)

Original: M ; y (cid:48) O ) .
Revised: M ; y (cid:48) O ) .

Original: An illustrative diagram of the production of PLMCMC proposals is provided in Appendix B.
Revised: An illustrative diagram of the production of PLMCMC proposals is provided in Appendix B.

Original: This proposal is then accepted with probability:
Revised: This proposal is then accepted with probability:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: Sample u ∼

Original: [Table]
Revised: [Table]

Original: Sample u ∼ Uniform [0 , 1] ; if u < α then
Revised: [0 , 1] ; if u < α then

Original: [Table]
Revised: [Table]

Original: 3. T HEORETICAL J USTIFICATION OF THE A LGORITHM
Revised: 3. T HEORETICAL J USTIFICATION OF THE A LGORITHM

Original: Proposition.
Revised: Proposition.

Original: For a given x O , if g ( ξ (cid:48) | ξ ) , p f,θ ( y M ; y O ) , and q ( y O ) are positive for any choice of ( y M ; y O ) ∈ X and ξ (cid:48) , ξ ∈ Ξ and are the densities for absolutely continuous distributions, the
Revised: For a given x O , if g ( ξ (cid:48) | ξ ) , p f,θ ( y M ; y O ) , and q ( y O ) are positive for any choice of ( y M ; y O ) ∈ X and ξ (cid:48) , ξ ∈ Ξ and are the densities for absolutely continuous distributions, the

Original: PL-MCMC update procedure listed in Algorithm 1 yields a Markov Chain of latent states ξ whose corresponding modeled data pairs, f θ ( ξ ) =
Revised: PL-MCMC update procedure listed in Algorithm 1 yields a Markov Chain of latent states ξ whose corresponding modeled data pairs, f θ ( ξ ) =

Original: ( y M ; y O ) , converge to a distribution with y M having marginal density p f,θ ( y M | x O ) .
Revised: ( y M ; y O ) , converge to a distribution with y M having marginal density p f,θ ( y M | x O ) .

Original: Proof.
Revised: Proof.

Original: Under these assumptions, the diffeomorphism (i.e, an invertible and differentiable mapping) provided by the ﬂow f θ allows us to interpret the latent transition kernel g ( ξ (cid:48) | ξ ) as the transition kernel g ( f − 1 θ ( y (cid:48) )
Revised: Under these assumptions, the diffeomorphism (i.e, an invertible and differentiable mapping) provided by the ﬂow f θ allows us to interpret the latent transition kernel g ( ξ (cid:48) | ξ ) as the transition kernel g ( f − 1 θ ( y (cid:48) )

Original: | f − 1 θ ( y ))
Revised: | f − 1 θ ( y ))

Original: | det ∂f − 1 θ ∂ y (cid:48) | within the modeled data space that is positive for all y , y (cid:48) ∈
Revised: | det ∂f − 1 θ ∂ y (cid:48) | within the modeled data space that is positive for all y , y (cid:48) ∈

Original: X and is the density for an absolutely continuous distribution.
Revised: X and is the density for an absolutely continuous distribution.

Original: Additionally, we note:
Revised: Additionally, we note:

Original: [Equation]
Revised: [Equation]

Original: The diffeomorphism provided by the ﬂow f θ also guarantees that
Revised: The diffeomorphism provided by the ﬂow f θ also guarantees that

Original: q ( y O ) p
Revised: q ( y O ) p

Original: f,θ ( y M | x O ) is positive for all ( y M ;
Revised: f,θ ( y M | x O ) is positive for all ( y M ;

Original: y O ) ∈ X
Revised: y O ) ∈ X

Original: and is the density for an absolutely continuous distribution.
Revised: and is the density for an absolutely continuous distribution.

Original: The procedure listed in Algorithm 1 therefore describes a Metropolis-Hastings update satisfying the conditions described by Tsvetkov et al.
Revised: The procedure listed in Algorithm 1 therefore describes a Metropolis-Hastings update satisfying the conditions described by Tsvetkov et al.

Original: The paired values f θ ( ξ ) =
Revised: The paired values f θ ( ξ ) =

Original: ( y M ; y O ) obtained through iterated application of Algorithm 1 thus converge to a target distribution with (unnormalized) density q ( y O ) p
Revised: ( y M ; y O ) obtained through iterated application of Algorithm 1 thus converge to a target distribution with (unnormalized) density q ( y O ) p

Original: f,θ ( y M | x O ) .
Revised: f,θ ( y M | x O ) .

Original: The requirements for convergence are very mild and are satisﬁed by the most common choices for latent, transition proposal, and auxiliary distributions (e.g. multivariate normal distributions).
Revised: The requirements for convergence are very mild and are satisﬁed by the most common choices for latent, transition proposal, and auxiliary distributions (e.g. multivariate normal distributions).

Original: We note that the eventual convergence of the PL-MCMC update towards the desired conditional distribution is not inﬂuenced by our choice of the auxiliary distribution q .
Revised: We note that the eventual convergence of the PL-MCMC update towards the desired conditional distribution is not inﬂuenced by our choice of the auxiliary distribution q .

Original: However, the choice of this auxiliary distribution can affect the rate of convergence.
Revised: However, the choice of this auxiliary distribution can affect the rate of convergence.

Original: We have found agreeable performance by selecting q to be an independent normal distribution centered on the conditioning values x O .
Revised: We have found agreeable performance by selecting q to be an independent normal distribution centered on the conditioning values x O .

Original: This guides the Markov Chain towards reasonable samples more quickly by leveraging learned dependencies between the observed and missing components of the modeled data.
Revised: This guides the Markov Chain towards reasonable samples more quickly by leveraging learned dependencies between the observed and missing components of the modeled data.

Original:  T RAINING N ORMALIZING F LOWS FROM M ISSING D ATA
Revised:  T RAINING N ORMALIZING F LOWS FROM M ISSING D ATA

Original: With PL-MCMC providing samples from the conditional distributions of normalizing ﬂows, a natural application of the technique is in MC-EM training (Dempster et al., 1977; Wei & Tanner, 1990;
Revised: With PL-MCMC providing samples from the conditional distributions of normalizing ﬂows, a natural application of the technique is in MC-EM training (Dempster et al., 1977; Wei & Tanner, 1990;

Original: Neath et al., 2013) of normalizing ﬂows from incomplete data.
Revised: Neath et al., 2013) of normalizing ﬂows from incomplete data.

Original: MC-EM training involves imputing missing values within the training set via conditional sampling of our current model, and then updating the parameters of our model to best ﬁt the newly imputed training set.
Revised: MC-EM training involves imputing missing values within the training set via conditional sampling of our current model, and then updating the parameters of our model to best ﬁt the newly imputed training set.

Original: As described within
Revised: As described within

Original: Appendix C, this leads to Algorithm 2, with PL-MCMC ( x O,i ; p f,θ , q i ) denoting the distribution obtained by following an implementation of PL-MCMC with auxiliary density q i (deﬁned in 3.1) and train being any training procedure that returns ﬂow parameters θ approximately maximizing the likelihood of a complete data training set.
Revised: Appendix C, this leads to Algorithm 2, with PL-MCMC ( x O,i ; p f,θ , q i ) denoting the distribution obtained by following an implementation of PL-MCMC with auxiliary density q i (deﬁned in 3.1) and train being any training procedure that returns ﬂow parameters θ approximately maximizing the likelihood of a complete data training set.

Original: For our experimental tests, PL-MCMC is obtained through iterated application of Algorithm 1.
Revised: For our experimental tests, PL-MCMC is obtained through iterated application of Algorithm 1.

Original: Algorithm 2:of
Revised: Algorithm 2:

Revised: Training of

Original: Flow
Revised: Flow

Original: Input: Incomplete training data X train = { x O, 1 , x O, 2 , . . .
Revised: Input: Incomplete training data X train = { x O, 1 , x O, 2 , . . .

Original: , x O,T } .
Revised: , x O,T } .

Original: Auxiliary densities q i .
Revised: Auxiliary densities q i .

Original: Normalizing ﬂow training procedure train .
Revised: Normalizing ﬂow training procedure train .

Original: Parameterized ﬂow architecture f θ .
Revised: Parameterized ﬂow architecture f θ .

Original: while training do for i ← 1 to T do
Revised: while training do for i ← 1 to T do

Original: Sample y M,i ∼ PL-MCMC ( x O,i ; p f,θ , q i ) ; end
Revised: Sample y M,i ∼ PL-MCMC ( x O,i ; p f,θ , q i ) ; end

Original: X (cid:48) train = { ( y M, 1 ; x O, 1 ) , ( y M, 2 ; x O, 2 ) , . . .
Revised: X (cid:48) train = { ( y M, 1 ; x O, 1 ) , ( y M, 2 ; x O, 2 ) , . . .

Original: , ( y M,T ; x O,T ) } ; θ ← train ( f, X (cid:48) train ) ; end
Revised: , ( y M,T ; x O,T ) } ; θ ← train ( f, X (cid:48) train ) ; end

Original: Intuitively, this procedure relies on conditional inference to “boost” the accuracy of our current model for the joint distribution governing the training data.
Revised: Intuitively, this procedure relies on conditional inference to “boost” the accuracy of our current model for the joint distribution governing the training data.

Original: At each step of Algorithm 2, X (cid:48) train represents samples from an approximation of the modeled data’s ground truth distribution.
Revised: At each step of Algorithm 2, X (cid:48) train represents samples from an approximation of the modeled data’s ground truth distribution.

Original: We ﬁt θ to model this approximate joint distribution.
Revised: We ﬁt θ to model this approximate joint distribution.

Original: After conditional inference with the new normalizing ﬂow using PL-MCMC, the next iteration of X (cid:48) train represents samples from a distribution with a smaller divergence from the ground truth distribution, as discussed in Appendix A. Importantly, this
Revised: After conditional inference with the new normalizing ﬂow using PL-MCMC, the next iteration of X (cid:48) train represents samples from a distribution with a smaller divergence from the ground truth distribution, as discussed in Appendix A. Importantly, this

Original: MC-EM training procedure assumes that data is missing at random (Little & Rubin, 2019).
Revised: MC-EM training procedure assumes that data is missing at random (Little & Rubin, 2019).

Original:  Q UALITATIVE E XPERIMENTAL R ESULTS
Revised:  Q UALITATIVE E XPERIMENTAL R ESULTS

Original: For a qualitative examination of the performance of PL-MCMC, we focus on conditionally sampling missing data using normalizing ﬂows that have been trained from complete data.
Revised: For a qualitative examination of the performance of PL-MCMC, we focus on conditionally sampling missing data using normalizing ﬂows that have been trained from complete data.

Original: We must note that the the purpose of PL-MCMC is to sample from a model’s conditional distributions, which may not coincide with accurately replicating the ground truth values of missing data.
Revised: We must note that the the purpose of PL-MCMC is to sample from a model’s conditional distributions, which may not coincide with accurately replicating the ground truth values of missing data.

Original: These qualitative experiments are therefore intended to illustrate aspects of the operation of PL-MCMC and to provide a visual veriﬁcation of the method’s performance.
Revised: These qualitative experiments are therefore intended to illustrate aspects of the operation of PL-MCMC and to provide a visual veriﬁcation of the method’s performance.

Original: Further details of these experiments and examples of unconditioned samples from the normalizing ﬂows are provided in Appendix D.
Revised: Further details of these experiments and examples of unconditioned samples from the normalizing ﬂows are provided in Appendix D.

Original: 5. C ONDITIONAL I NFERENCE WITH CIFAR-10 I MAGES
Revised: 5. C ONDITIONAL I NFERENCE WITH CIFAR-10 I MAGES

Original: We ﬁrst consider sampling a missing central quarter of CIFAR-10 (Krizhevsky et al., 2009) images ( 32 × 32 full color images) using a normalizing ﬂow following the GLOW architecture (Kingma &
Revised: We ﬁrst consider sampling a missing central quarter of CIFAR-10 (Krizhevsky et al., 2009) images ( 32 × 32 full color images) using a normalizing ﬂow following the GLOW architecture (Kingma &

Original: Dhariwal, 2018).
Revised: Dhariwal, 2018).

Original: To bolster our claim that PL-MCMC does not require specially trained models, we utilize a publicly available pre-trained model (van Amersfoort, 2019) for this experiment.
Revised: To bolster our claim that PL-MCMC does not require specially trained models, we utilize a publicly available pre-trained model (van Amersfoort, 2019) for this experiment.

Original: Initial and ﬁnal completions provided by the Markov Chain are illustrated in Figure 1.
Revised: Initial and ﬁnal completions provided by the Markov Chain are illustrated in Figure 1.

Original: [Figure]
Revised: [Figure]

Original: The initial state of the Markov Chain is constructed by ﬁlling pixels with RGB values randomly selected from the observed subset.
Revised: The initial state of the Markov Chain is constructed by ﬁlling pixels with RGB values randomly selected from the observed subset.

Original: Latent space transitions are generated via small perturbations within the absolute coordinates of the latent space.
Revised: Latent space transitions are generated via small perturbations within the absolute coordinates of the latent space.

Original: PL-MCMC is carried out for 25,000 proposals.
Revised: PL-MCMC is carried out for 25,000 proposals.

Original: Example progressions of completions are provided in Figure 2.
Revised: Example progressions of completions are provided in Figure 2.

Original: In comparison with unconditioned samples, the PL-MCMC completions appear reasonable, given the capabilities of the underlying model, and highlight the perceptual beneﬁt provided by conditioned sampling.
Revised: In comparison with unconditioned samples, the PL-MCMC completions appear reasonable, given the capabilities of the underlying model, and highlight the perceptual beneﬁt provided by conditioned sampling.

Original: 5. C ONDITIONAL I NFERENCE WITH C ELEB A I MAGES
Revised: 5. C ONDITIONAL I NFERENCE WITH C ELEB A I MAGES

Original: Next we consider sampling a missing right half of CelebA (Liu et al., 2015) images (aligned, cropped, and resized to 64 × 64 full color images) using a normalizing ﬂow following the GLOW architecture (Kingma & Dhariwal, 2018).
Revised: Next we consider sampling a missing right half of CelebA (Liu et al., 2015) images (aligned, cropped, and resized to 64 × 64 full color images) using a normalizing ﬂow following the GLOW architecture (Kingma & Dhariwal, 2018).

Original: To bolster our claim that PL-MCMC does not require specially trained models, we utilize a publicly available pre-trained model (Yuki-Chai, 2019) for this experiment.
Revised: To bolster our claim that PL-MCMC does not require specially trained models, we utilize a publicly available pre-trained model (Yuki-Chai, 2019) for this experiment.

Original: Initial and ﬁnal completions provided by the Markov Chain are illustrated in Figure 3.
Revised: Initial and ﬁnal completions provided by the Markov Chain are illustrated in Figure 3.

Original: [Figure]
Revised: [Figure]

Original: The initial state of the Markov Chain is constructed by sampling from the normalizing ﬂow at reduced variance.
Revised: The initial state of the Markov Chain is constructed by sampling from the normalizing ﬂow at reduced variance.

Original: Latent space transitions are generated via small perturbations within relative coordinates of the latent space.
Revised: Latent space transitions are generated via small perturbations within relative coordinates of the latent space.

Original: PL-MCMC is carried out for 25,000 proposals.
Revised: PL-MCMC is carried out for 25,000 proposals.

Original: Example progressions of completions are provided in Figure 4.
Revised: Example progressions of completions are provided in Figure 4.

Original: The progression of PL-MCMC completions clearly demonstrates how deﬁning a Markov Chain through the ﬂow’s latent space encourages proposing alterations to semantically meaningful attributes.
Revised: The progression of PL-MCMC completions clearly demonstrates how deﬁning a Markov Chain through the ﬂow’s latent space encourages proposing alterations to semantically meaningful attributes.

Original: C ONDITIONAL I NFERENCE WITH MNIST D IGITS
Revised: C ONDITIONAL I NFERENCE WITH MNIST D IGITS

Original: Finally, we consider sampling missing portions of MNIST (LeCun et al., 1998) digits ( 28 ×monochrome images) using a normalizing ﬂow following a variant of the NICE architecture (Dinh et al., 2014) under a variety of data missingness mechanisms.
Revised: Finally, we consider sampling missing portions of MNIST (LeCun et al., 1998) digits ( 28 ×monochrome images) using a normalizing ﬂow following a variant of the NICE architecture (Dinh et al., 2014) under a variety of data missingness mechanisms.

Original: The missingness mechanisms considered are independent missingness (I.M.), patch missingness (P.M.), and square observation (S.O.), at missingness rates of 0 . 6 , 0 . 6 , and 0 .
Revised: The missingness mechanisms considered are independent missingness (I.M.), patch missingness (P.M.), and square observation (S.O.), at missingness rates of 0 . 6 , 0 . 6 , and 0 .

Original: 8 respectively.
Revised: 8 respectively.

Original: Final completions and conditional expectations as obtained by averaging the ﬁnal completions of 20 independent PL-MCMC chains are illustrated in Figure 5.
Revised: Final completions and conditional expectations as obtained by averaging the ﬁnal completions of 20 independent PL-MCMC chains are illustrated in Figure 5.

Original: [Figure]
Revised: [Figure]

Original: The initial state of the Markov Chain is constructed by sampling from the normalizing ﬂow at reduced variance.
Revised: The initial state of the Markov Chain is constructed by sampling from the normalizing ﬂow at reduced variance.

Original: Latent space transitions are generated by a mixture of small perturbations within the absolute coordinates of the latent space and resampling at reduced variance.
Revised: Latent space transitions are generated by a mixture of small perturbations within the absolute coordinates of the latent space and resampling at reduced variance.

Original: PL-MCMC is performed over 2,000 proposals.
Revised: PL-MCMC is performed over 2,000 proposals.

Original: Example progressions of completions are provided in Figure 6.
Revised: Example progressions of completions are provided in Figure 6.

Original:  Q UANTITATIVE E XPERIMENTAL R ESULTS
Revised:  Q UANTITATIVE E XPERIMENTAL R ESULTS

Original: As an analytical description of the conditional distributions of non-specialized normalizing ﬂows is infeasible, it is difﬁcult to quantify how well PL-MCMC succeeds in sampling from its intended distributions.
Revised: As an analytical description of the conditional distributions of non-specialized normalizing ﬂows is infeasible, it is difﬁcult to quantify how well PL-MCMC succeeds in sampling from its intended distributions.

Original: Given the extreme dependence of Algorithm 2 on accurate conditional sampling from PLMCMC for effective training, we therefore quantify the performance of normalizing ﬂows trained from incomplete data as an indication for whether PL-MCMC produces sufﬁciently accurate and efﬁcient samples to remain useful for real-world missing data tasks.
Revised: Given the extreme dependence of Algorithm 2 on accurate conditional sampling from PLMCMC for effective training, we therefore quantify the performance of normalizing ﬂows trained from incomplete data as an indication for whether PL-MCMC produces sufﬁciently accurate and efﬁcient samples to remain useful for real-world missing data tasks.

Original: We also test the sampling efﬁciency of PL-MCMC independently of considerations regarding sampling accuracy.
Revised: We also test the sampling efﬁciency of PL-MCMC independently of considerations regarding sampling accuracy.

Original: Further details of these experiments are provided in Appendix E.
Revised: Further details of these experiments are provided in Appendix E.

Original: 6. T RAINING FROM I NCOMPLETE MNIST D IGITS
Revised: 6. T RAINING FROM I NCOMPLETE MNIST D IGITS

Original: In this experiment, we consider training models of MNIST digits from training sets affected by a variety of data missingness mechanisms and imputing test sets affected by the same missingness mechanisms.
Revised: In this experiment, we consider training models of MNIST digits from training sets affected by a variety of data missingness mechanisms and imputing test sets affected by the same missingness mechanisms.

Original: The data missingness mechanisms used are are independent missingness (I.M.), patch missingness (P.M.), and square observation (S.O.), with missingess rates of 0 . 3 , 0 . 6 , and 0 .
Revised: The data missingness mechanisms used are are independent missingness (I.M.), patch missingness (P.M.), and square observation (S.O.), with missingess rates of 0 . 3 , 0 . 6 , and 0 .

Original: As imputation performance measures, we consider per-pixel reconstruction RMSE and Fr´echet Inception Distance (Heusel et al., 2017).
Revised: As imputation performance measures, we consider per-pixel reconstruction RMSE and Fr´echet Inception Distance (Heusel et al., 2017).

Original: As comparison, we include results for imputing using pixel wise observed means and using the convolutional variant of MisGAN (Li et al., 2018).
Revised: As comparison, we include results for imputing using pixel wise observed means and using the convolutional variant of MisGAN (Li et al., 2018).

Original: Our normalizing ﬂow is a variant of the NICE architecture.
Revised: Our normalizing ﬂow is a variant of the NICE architecture.

Original: We performed MC-EM training of the normalizing ﬂow for a total of 1,000 epochs.
Revised: We performed MC-EM training of the normalizing ﬂow for a total of 1,000 epochs.

Original: Inference with normalizing ﬂows is performed using a PL-MCMC chain of 2 , 000 proposals.
Revised: Inference with normalizing ﬂows is performed using a PL-MCMC chain of 2 , 000 proposals.

Original: Our reported results within Table 1 reﬂect performance across ﬁfteen distinct pairings of training and test sets (models trained, where applicable, from three distinct training sets and each tested on ﬁve distinct test sets).
Revised: Our reported results within Table 1 reﬂect performance across ﬁfteen distinct pairings of training and test sets (models trained, where applicable, from three distinct training sets and each tested on ﬁve distinct test sets).

Original: For PL-MCMC, our results reﬂect imputationperformance using individual conditional samples (Ind.) and using the average of 10 conditional samples (Avg.) for test set completion.
Revised: For PL-MCMC, our results reﬂect imputation using individual conditional samples (Ind.) and using the average of 10 conditional samples (Avg.) for test set completion.

Original: [Table]
Revised: [Table]

Original: As RMSE and FID score are measures of distortion and divergence
Revised: As RMSE and FID score are measures of distortion and divergence

Original: [Table]
Revised: [Table]

Original: respectively
Revised: respectively

Original: [Table]
Revised: [Table]

Original: a single imputation estimate cannot simultaneously optimize both (Blau & Michaeli, 2018).
Revised: a single imputation estimate cannot simultaneously optimize both (Blau & Michaeli, 2018).

Original: MisGAN primarily focuses on minimizing imputation FID, while our MC-EM training favors reducing reconstruction RMSE.
Revised: MisGAN primarily focuses on minimizing imputation FID, while our MC-EM training favors reducing reconstruction RMSE.

Original: Our results highlight a potential advantage of performing imputation via sampling from conditional distributions.
Revised: Our results highlight a potential advantage of performing imputation via sampling from conditional distributions.

Original: With its deterministic imputation procedure, MisGAN is dedicated to minimizing FID and cannot reduce reconstruction RMSE by averaging multiple reconstructions.
Revised: With its deterministic imputation procedure, MisGAN is dedicated to minimizing FID and cannot reduce reconstruction RMSE by averaging multiple reconstructions.

Original: With PL-MCMC sampling, we can choose, to some degree, whether to minimize FID by imputing with a single sample from the ﬂow’s conditional distribution or to minimize RMSE by averaging across multiple samples.
Revised: With PL-MCMC sampling, we can choose, to some degree, whether to minimize FID by imputing with a single sample from the ﬂow’s conditional distribution or to minimize RMSE by averaging across multiple samples.

Original: These results demonstrate that PL-MCMC is able to sample from the conditional distributions of normalizing ﬂows sufﬁciently well to acceptably train normalizing ﬂows from MNIST digits affected by a variety of data missingness mechanisms and rates.
Revised: These results demonstrate that PL-MCMC is able to sample from the conditional distributions of normalizing ﬂows sufﬁciently well to acceptably train normalizing ﬂows from MNIST digits affected by a variety of data missingness mechanisms and rates.

Original: 6. T RAINING FROM I NCOMPLETE UCI D ATASETS
Revised: 6. T RAINING FROM I NCOMPLETE UCI D ATASETS

Original: In this experiment, we consider training models of various continuous UCI datasets (Bache & Lichman, 2013) affected by 50% uniformly missing values.
Revised: In this experiment, we consider training models of various continuous UCI datasets (Bache & Lichman, 2013) affected by 50% uniformly missing values.

Original: As a performance measure, we consider normalized MSE of imputing missing values within the training set.
Revised: As a performance measure, we consider normalized MSE of imputing missing values within the training set.

Original: As comparison, we include results for imputing using variable-wise observed means, using the missForest (Stekhoven & B¨uhlmann, 2012) R package with default settings, and using VAEs via MIWAE (Mattei & Frellsen, 2019).
Revised: As comparison, we include results for imputing using variable-wise observed means, using the missForest (Stekhoven & B¨uhlmann, 2012) R package with default settings, and using VAEs via MIWAE (Mattei & Frellsen, 2019).

Original: Our normalizing ﬂow is a variant of the NICE architecture.
Revised: Our normalizing ﬂow is a variant of the NICE architecture.

Original: We performed MC-EM training of the normalizing ﬂow for a total of 1,000 epochs.
Revised: We performed MC-EM training of the normalizing ﬂow for a total of 1,000 epochs.

Original: For inference, the PL-MCMC chain is run for 1, proposals.
Revised: For inference, the PL-MCMC chain is run for 1, proposals.

Original: Our reported results within Table 2 reﬂect performance across ﬁve distinct training sets.
Revised: Our reported results within Table 2 reﬂect performance across ﬁve distinct training sets.

Original: For PL-MCMC, our results reﬂect imputation performance using individual conditional samples (Ind.) and using the average of 25 conditional samples (Avg.) for test set completion.
Revised: For PL-MCMC, our results reﬂect imputation performance using individual conditional samples (Ind.) and using the average of 25 conditional samples (Avg.) for test set completion.

Original: [Table]
Revised: [Table]

Original: In all cases
Revised: In all cases

Original: [Table]
Revised: [Table]

Original: the MC-EM trained normalizing ﬂows perform at least as well as missForest and closely match MIWAE for estimating conditional expectations.
Revised: the MC-EM trained normalizing ﬂows perform at least as well as missForest and closely match MIWAE for estimating conditional expectations.

Original: We can conclude that, while there is some potential room for improvement in capturing the exact ground truth conditional distributions, MCEM training of normalizing ﬂows with PL-MCMC produces imputations comparable to those from current methods for this particular task.
Revised: We can conclude that, while there is some potential room for improvement in capturing the exact ground truth conditional distributions, MCEM training of normalizing ﬂows with PL-MCMC produces imputations comparable to those from current methods for this particular task.

Original: 6. S AMPLING E FFICIENCY FOR I NFERENCE OF MNIST D IGIT
Revised: 6. S AMPLING E FFICIENCY FOR I NFERENCE OF MNIST D IGIT

Original: Here we consider the task of estimating the conditional expectation for the missing region of a single
Revised: Here we consider the task of estimating the conditional expectation for the missing region of a single

Original: MNIST digit using the average of 100 independent Markov Chains.
Revised: MNIST digit using the average of 100 independent Markov Chains.

Original: We also use this experiment as an opportunity to explore the effect on conditional sampling performance produced by different choices for PL-MCMC’s auxiliary distribution and the transition proposal distribution.
Revised: We also use this experiment as an opportunity to explore the effect on conditional sampling performance produced by different choices for PL-MCMC’s auxiliary distribution and the transition proposal distribution.

Original: The RMSE versus proposal number of conditional means estimated via Gibbs sampling within the modeled data space and PL-MCMC with varying auxiliary distributions are compared in Figure 7.
Revised: The RMSE versus proposal number of conditional means estimated via Gibbs sampling within the modeled data space and PL-MCMC with varying auxiliary distributions are compared in Figure 7.

Original: Statistics are gathered from 10 distinct replications of the experiment.
Revised: Statistics are gathered from 10 distinct replications of the experiment.

Original: These results demonstrate that PL-MCMC can offer signiﬁcant performance gains over comparable
Revised: These results demonstrate that PL-MCMC can offer signiﬁcant performance gains over comparable

Original: MCMC methods conﬁned to the modeled data space.
Revised: MCMC methods conﬁned to the modeled data space.

Original: Even when using an improper uniform distribution as the auxiliary density (effectively omitting q from the acceptance probability calculation in
Revised: Even when using an improper uniform distribution as the auxiliary density (effectively omitting q from the acceptance probability calculation in

Original: Algorithm 1), PL-MCMC can accelerate conditional sampling by leveraging the ﬂow’s latent space to propose more effective proposal transitions.
Revised: Algorithm 1), PL-MCMC can accelerate conditional sampling by leveraging the ﬂow’s latent space to propose more effective proposal transitions.

Original: Depending on the characteristics of the normalizing ﬂow’s conditional distribution, selecting a more restrictive auxiliary distribution can greatly accelerate sampling even further.
Revised: Depending on the characteristics of the normalizing ﬂow’s conditional distribution, selecting a more restrictive auxiliary distribution can greatly accelerate sampling even further.

Original: As the results with auxiliary distributions with standard deviations of σ a = 10 − 3 and σ a = 10 − 6 closely overlap, there may be some concern that the auxiliary distri- bution might dominate PL-MCMC’s behavior and reduce the procedure to a simple search in the latent space to best rebuild the observed data, starting around σ a = 10 − 3 .
Revised: As the results with auxiliary distributions with standard deviations of σ a = 10 − 3 and σ a = 10 − 6 closely overlap, there may be some concern that the auxiliary distri- bution might dominate PL-MCMC’s behavior and reduce the procedure to a simple search in the latent space to best rebuild the observed data, starting around σ a = 10 − 3 .

Original: While this concern may be warranted when using exceedingly strong choices for the auxiliary distribution, analysis demonstrates (Appendix E.4) that this is not the case for our results with σ a = 10 − 3 .
Revised: While this concern may be warranted when using exceedingly strong choices for the auxiliary distribution, analysis demonstrates (Appendix E.4) that this is not the case for our results with σ a = 10 − 3 .

Original: The RMSE versus proposal number of conditional means estimated via Gibbs sampling within the modeled data space and PL-MCMC with varying transition proposal distributions are compared in Figure 8.
Revised: The RMSE versus proposal number of conditional means estimated via Gibbs sampling within the modeled data space and PL-MCMC with varying transition proposal distributions are compared in Figure 8.

Original: From these experiments, we offer a few preliminary conclusions regarding the effect of the auxiliary and transition proposal distributions on conditional sampling performance with PL-MCMC.
Revised: From these experiments, we offer a few preliminary conclusions regarding the effect of the auxiliary and transition proposal distributions on conditional sampling performance with PL-MCMC.

Original: In the Metropolis-Hastings implementation of PL-MCMC, the transition proposal distribution behaves much as one would expect for the transition proposal distributions of any Metropolis-Hastings procedures.
Revised: In the Metropolis-Hastings implementation of PL-MCMC, the transition proposal distribution behaves much as one would expect for the transition proposal distributions of any Metropolis-Hastings procedures.

Original: Increasing the proposal distribution’s scale will accelerate initial convergence, but may encounter problems traversing concentrated regions of the target distributions.
Revised: Increasing the proposal distribution’s scale will accelerate initial convergence, but may encounter problems traversing concentrated regions of the target distributions.

Original: To some target distribution dependent point (around σ a = 10 − 1 . 5 in Figure 7), strengthening the auxiliary distribution will continue to accelerate initial sampling.
Revised: To some target distribution dependent point (around σ a = 10 − 1 . 5 in Figure 7), strengthening the auxiliary distribution will continue to accelerate initial sampling.

Original: Beyond this point, further strengthening of the auxiliary distribution can be detrimental to sampling performance, as the auxiliary distribution becomes mismatched to the intrinsic coupling between missing and observed values modeled by the ﬂow’s conditional distribution.
Revised: Beyond this point, further strengthening of the auxiliary distribution can be detrimental to sampling performance, as the auxiliary distribution becomes mismatched to the intrinsic coupling between missing and observed values modeled by the ﬂow’s conditional distribution.

Original: Additional comparisons, including comparisons with respect to approximate computational cost, are provided within Appendix E.
Revised: Additional comparisons, including comparisons with respect to approximate computational cost, are provided within Appendix E.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: The mathematical structure of normalizing ﬂows is exceptionally convenient for approaching conditional sampling via MCMC.
Revised: The mathematical structure of normalizing ﬂows is exceptionally convenient for approaching conditional sampling via MCMC.

Original: By leveraging this mathematical structure, our proposed PL-MCMC technique enables asymptotically exact conditional inference with normalizing ﬂows, without requiring specialized architecture, training history, or external inference machinery.
Revised: By leveraging this mathematical structure, our proposed PL-MCMC technique enables asymptotically exact conditional inference with normalizing ﬂows, without requiring specialized architecture, training history, or external inference machinery.

Original: The particular implementations used in our experiments are primarily intended to serve as proof-of-concept illustrations of the PL-MCMC technique.
Revised: The particular implementations used in our experiments are primarily intended to serve as proof-of-concept illustrations of the PL-MCMC technique.

Original: Further research would be necessary to determine optimal choices of auxiliary distributions,transition proposal distributions, and MC-EM training procedures.
Revised: Further research would be necessary to determine optimal choices of auxiliary distributions,transition proposal distributions, and MC-EM training procedures.

Original: Sampling performance may be improved by replacing Metropolis-Hastings proposals with a more sophisticated technique, such as Hamiltonian Monte Carlo.
Revised: Sampling performance may be improved by replacing Metropolis-Hastings proposals with a more sophisticated technique, such as Hamiltonian Monte Carlo.

Original: Our experimental results demonstrate that, even when implemented with a naive Metropolis-Hastings procedure, PL-MCMC enables effective sampling from its intended distributions under practical settings.
Revised: Our experimental results demonstrate that, even when implemented with a naive Metropolis-Hastings procedure, PL-MCMC enables effective sampling from its intended distributions under practical settings.

Original: We believe that, with the
Revised: We believe that, with the

Original: PL-MCMC technique, normalizing ﬂows hold great promise for approaching missing data tasks.
Revised: PL-MCMC technique, normalizing ﬂows hold great promise for approaching missing data tasks.

Revised: A CKNOWLEDGEMENTS

Revised: This work was supported by the Ofﬁce of Naval Research Grant No. N00014-18-1-2244.

Original: A K NOWING THE J OINT D ISTRIBUTION I MPLIES K NOWING THE C ONDITIONAL D ISTRIBUTIONS
Revised: A K NOWING THE J OINT D ISTRIBUTION I MPLIES K NOWING THE C ONDITIONAL D ISTRIBUTIONS

Original: In this section we establish a fundamental guarantee regarding the accuracy of the conditional distributions known by generative models.
Revised: In this section, we establish a fundamental guarantee regarding the accuracy of the conditional distributions known by generative models.

Original: Consider two random variables, x and y , with ground truth joint distribution p x y that we have approximated by a generative model with joint distribution q x y .
Revised: Consider two random variables, x and y , with ground truth joint distribution p x,y that we have approximated by a generative model with joint distribution q x,y .

Original: In the discrete case  we may expand through the deﬁnition of the Kullback-Liebler divergence between our generative model and the ground truth to ﬁnd that:
Revised: In the discrete case [Equation] we may expand through the deﬁnition of the Kullback-Liebler divergence between our generative model and the ground truth to ﬁnd that:

Original: [Equation]
Revised: [Equation]

Original: From the non-negativity of the Kullback-Liebler divergence, we are then guaranteed:
Revised: From the non-negativity of the Kullback-Liebler divergence [Equation] we are then guaranteed:

Original: [Equation]
Revised: [Equation]

Original: With equality only when our generative model perfectly models the marginal distribution of x .
Revised: With equality only when our generative model perfectly models the marginal distribution of x .

Original: When considering the task of inferring y from observed values of x , the expected performance of our generative model in approximating the conditional distribution of y given x is no worse than its performance in approximating the full joint distribution between x and y .
Revised: When considering the task of inferring y from observed values of x [Equation] the expected performance of our generative model in approximating the conditional distribution of y given x is no worse than its performance in approximating the full joint distribution between x and y .

Original: Therefore, if we know that a generative model is a good approximation of the joint distribution governing some set of random variables, then it must also know good approximations of the conditional distributions among those random variables.
Revised: Therefore, if we know that a generative model is a good approximation of the joint distribution governing some set of random variables, then it must also know good approximations of the conditional distributions among those random variables.

Original: This inequality also serves as a justiﬁcation for Monte Carlo Expectation Maximization training.
Revised: This inequality also serves as a justiﬁcation for Monte Carlo Expectation Maximization training.

Original: When using our modeled distribution q x,y to impute an incomplete training set, the newly imputed training set is sampled from the distribution q y | x p x .
Revised: When using our modeled distribution q x,y to impute an incomplete training set, the newly imputed training set is sampled from the distribution q y | x p x .

Original: We can easily see that:
Revised: We can easily see that:

Original: [Equation]
Revised: [Equation]

Original: In the asymptotic limit of dataset size, conditionally inferring missing values within the dataset results in samples from a distribution whose divergence from ground truth is no worse than that of the original model.
Revised: In the asymptotic limit of dataset size, conditionally inferring missing values within the dataset results in samples from a distribution whose divergence from ground truth is no worse than that of the original model.

Original: Assuming that the original model describes the distribution of a previously imputed version of the training set, this implies that our newly training set is at least as reﬂective of the ground truth distribution as the previous training set.
Revised: Assuming that the original model describes the distribution of a previously imputed version of the training set, this implies that our newly training set is at least as reﬂective of the ground truth distribution as the previous training set.

Original: In practice, we ﬁnd that conditional imputation tends to improve divergence of the training set, which in turn allows MC-EM training to improve our model of the joint distribution.
Revised: In practice, we ﬁnd that conditional imputation tends to improve divergence of the training set, which in turn allows MC-EM training to improve our model of the joint distribution.

Original: B T HE A DVANTAGE OF L ATENT S PACE P ROPOSALS
Revised: B T HE A DVANTAGE OF L ATENT S PACE P ROPOSALS

Original: Here, we relay our intuition regarding the advantages of deﬁning a Markov Chain within the latent space of a normalizing ﬂow.
Revised: Here, we relay our intuition regarding the advantages of deﬁning a Markov Chain within the latent space of a normalizing ﬂow.

Original: This section provides a heuristic argument and therefore utilizes informal terminology to convey our current understanding.
Revised: This section provides a heuristic argument and therefore utilizes informal terminology to convey our current understanding.

Original: Take a normalizing ﬂow between latent space Ξ and modeled data space X , deﬁning the mappings f θ : Ξ (cid:55)→ X and f − 1 θ : X (cid:55)→ Ξ .
Revised: Take a normalizing ﬂow between latent space Ξ and modeled data space X , deﬁning the mappings f θ : Ξ (cid:55)→ X and f − 1 θ : X (cid:55)→ Ξ .

Original: This normalizing ﬂow imposes the probability density p f,θ ( x ) onto all modeled data values x ∈ X .
Revised: This normalizing ﬂow imposes the probability density p f,θ ( x ) onto all modeled data values x ∈ X .

Original: As practical applications of normalizing ﬂows primarily involve data embedded within a euclidean space, we will conﬁne this discussion to scenarios where latent and modeled data values are both points in R n for some n .
Revised: As practical applications of normalizing ﬂows primarily involve data embedded within a euclidean space, we will conﬁne this discussion to scenarios where latent and modeled data values are both points in R n for some n .

Original: In these cases, it is straightforward to discuss neighborhoods of ﬁxed radius around points within both the latent and modeled data spaces.
Revised: In these cases, it is straightforward to discuss neighborhoods of ﬁxed radius around points within both the latent and modeled data spaces.

Original: For now, let us consider the task of forming a Markov Chain for unconditionally sampling from the density p f,θ ( x ) .
Revised: For now, let us consider the task of forming a Markov Chain for unconditionally sampling from the density p f,θ ( x ) .

Original: For simplicity, let us only consider proposal perturbations within some ﬁxed radius of the Markov Chain’s current state.
Revised: For simplicity, let us only consider proposal perturbations within some ﬁxed radius of the Markov Chain’s current state.

Original: With the one-to-one mapping provided by the ﬂow, we have the option of tracking and perturbing the current state within either the latent space or the modeled data space.
Revised: With the one-to-one mapping provided by the ﬂow, we have the option of tracking and perturbing the current state within either the latent space or the modeled data space.

Original: When considering the neighborhood of data points, probability mass within the modeled data space is often non-isotropic for highly structured data.
Revised: When considering the neighborhood of data points, probability mass within the modeled data space is often non-isotropic for highly structured data.

Original: However, probability mass is nearly isotropic within the latent space in the neighborhood of the latent representations of data points, assuming that the distribution on latent space states has been appropriately chosen (as is the case for the commonly used multivariate normal or logistic distributions).
Revised: However, probability mass is nearly isotropic within the latent space in the neighborhood of the latent representations of data points, assuming that the distribution on latent space states has been appropriately chosen (as is the case for the commonly used multivariate normal or logistic distributions).

Original: As a result, performing an isotropic perturbation within the latent space results in proposals that are about as likely as the starting state.
Revised: As a result, performing an isotropic perturbation within the latent space results in proposals that are about as likely as the starting state.

Original: Within the modeled data space, even a very small isotropic perturbation can produce proposals that are far more unlikely than the starting state.
Revised: Within the modeled data space, even a very small isotropic perturbation can produce proposals that are far more unlikely than the starting state.

Original: As an example, suppose our normalizing ﬂow was well trained to model a set of high-ﬁdelity images.
Revised: As an example, suppose our normalizing ﬂow was well trained to model a set of high-ﬁdelity images.

Original: If our proposals within the modeled data space were created by adding independent Gaussian perturbations to pixel values, we would almost always inject noise into the image and proposals within the modeled data space would be tend to be unlikely, low-ﬁdelity images.
Revised: If our proposals within the modeled data space were created by adding independent Gaussian perturbations to pixel values, we would almost always inject noise into the image and proposals within the modeled data space would be tend to be unlikely, low-ﬁdelity images.

Original: With the assumption that transitions between equally likely states are usually accepted and transitions to much more unlikely states are usually rejected, we should expect latent space proposals to be accepted more frequently compared to modeled data space proposals.
Revised: With the assumption that transitions between equally likely states are usually accepted and transitions to much more unlikely states are usually rejected, we should expect latent space proposals to be accepted more frequently compared to modeled data space proposals.

Original: As an intuition, we could say that perturbations within the ﬂow’s latent space are semantically meaningful for the modeled data set.
Revised: As an intuition, we could say that perturbations within the ﬂow’s latent space are semantically meaningful for the modeled data set.

Original: As demonstrated by Hoffman et al.
Revised: As demonstrated by Hoffman et al.

Original: (2019), the normalizing ﬂow inherently transforms the modeled probability distribution in a manner that is well suited to exploration using naive, isotropic proposals.
Revised: (2019), the normalizing ﬂow inherently transforms the modeled probability distribution in a manner that is well suited to exploration using naive, isotropic proposals.

Original: This is related to Adaptive Monte Carlo methods (Haario et al., 2001; Foreman-Mackey et al., 2013; Zhu, 2019), which attempt to transform the proposal density to most effectively explore a ﬁxed distribution.
Revised: This is related to Adaptive Monte Carlo methods (Haario et al., 2001; Foreman-Mackey et al., 2013; Zhu, 2019), which attempt to transform the proposal density to most effectively explore a ﬁxed distribution.

Original: With latent space transitions in normalizing ﬂows, it is as though the modeled data distribution has been transformed so as to be best explored by a ﬁxed proposal density.
Revised: With latent space transitions in normalizing ﬂows, it is as though the modeled data distribution has been transformed so as to be best explored by a ﬁxed proposal density.

Original: With PL-MCMC we are concerned with making effective proposals with respect to a conditional distribution.
Revised: With PL-MCMC we are concerned with making effective proposals with respect to a conditional distribution.

Original: Even when attempting to sample a conditional distribution, utilizing latent space proposals remains beneﬁcial.
Revised: Even when attempting to sample a conditional distribution, utilizing latent space proposals remains beneﬁcial.

Original: Deﬁne a projection operator via proj x O ( y M ; y O )
Revised: Deﬁne a projection operator via proj x O ( y M ; y O )

Original: =
Revised: =

Original: y M ; x O , which simply replaces the observed component of a y ∈ X with the conditioning values x O .
Revised: y M ; x O , which simply replaces the observed component of a y ∈ X with the conditioning values x O .

Original: The elements of a proposed transition within a Metropolis-Hastings implementation of PL-MCMC are illustrated in Figure 9.
Revised: The elements of a proposed transition within a Metropolis-Hastings implementation of PL-MCMC are illustrated in Figure 9.

Original: Averaging over possible pieces of observed data x O , we expect to ﬁnd that the set of likely completions, x M , under the conditional density p f,θ ( x M | x O ) remains essentially a subset of the set of likely values under the marginalized density p f,θ ( x M ) .
Revised: Averaging over possible pieces of observed data x O , we expect to ﬁnd that the set of likely completions, x M , under the conditional density p f,θ ( x M | x O ) remains essentially a subset of the set of likely values under the marginalized density p f,θ ( x M ) .

Original: If a proposed x M is unlikely under p f,θ ( x M ) , we expect it to be unlikely under p f,θ ( x M | x O ) .
Revised: If a proposed x M is unlikely under p f,θ ( x M ) , we expect it to be unlikely under p f,θ ( x M | x O ) .

Original: Hence, with PL-MCMC, the advantages of latent space proposals carry through to the conditional inference setting, as the resulting proposed completions can remain likely under p f,θ ( x M ) .
Revised: Hence, with PL-MCMC, the advantages of latent space proposals carry through to the conditional inference setting, as the resulting proposed completions can remain likely under p f,θ ( x M ) .

Original: Returning to the example of modeling a set of high-ﬁdelity images, suppose we observe the left half of these images.
Revised: Returning to the example of modeling a set of high-ﬁdelity images, suppose we observe the left half of these images.

Original: In general, we would believe that the set of likely right half completions conditioned on the observed left half is well covered by set of likely right halves that we see across the entire distribution of images.
Revised: In general, we would believe that the set of likely right half completions conditioned on the observed left half is well covered by set of likely right halves that we see across the entire distribution of images.

Original: Perturbing the pixel values of the right half injects noise, tending to produce a low-ﬁdelity right half, which results in an unlikely, noisy image when combined with the observed left half.
Revised: Perturbing the pixel values of the right half injects noise, tending to produce a low-ﬁdelity right half, which results in an unlikely, noisy image when combined with the observed left half.

Original: Following the PL-MCMC latent perturbations, proposed right halves may be more able to at least remain the high-ﬁdelity right halves of high-ﬁdelity images.
Revised: Following the PL-MCMC latent perturbations, proposed right halves may be more able to at least remain the high-ﬁdelity right halves of high-ﬁdelity images.

Original: By employing latent space proposals to sample from p f,θ ( x M | x O ) , PL-MCMC can more easily propose completions x M that could plausibly have been taken from likely members of the modeled data distribution.
Revised: By employing latent space proposals to sample from p f,θ ( x M | x O ) , PL-MCMC can more easily propose completions x M that could plausibly have been taken from likely members of the modeled data distribution.

Original: Of course, for conditional inference, we also need to produce samples that are well matched to the observed data.
Revised: Of course, for conditional inference, we also need to produce samples that are well matched to the observed data.

Original: While latent space proposals assist in making meaningful and efﬁcient transitions within a Markov Chain, PL-MCMC ultimately relies on the auxiliary distribution, q , and guaranteed convergence to the correct conditional distribution to effectively sample from typical completions of the observed data.
Revised: While latent space proposals assist in making meaningful and efﬁcient transitions within a Markov Chain, PL-MCMC ultimately relies on the auxiliary distribution, q , and guaranteed convergence to the correct conditional distribution to effectively sample from typical completions of the observed data.

Original: C D ETAILS OF MC-EM T RAINING
Revised: C D ETAILS OF MC-EM T RAINING

Original: In this section we review the derivation of Monte Carlo Expectation maximization (Dempster et al., 1977; Wei & Tanner, 1990; Neath et al., 2013) in the context of its use with PL-MCMC.
Revised: In this section we review the derivation of Monte Carlo Expectation maximization (Dempster et al., 1977; Wei & Tanner, 1990; Neath et al., 2013) in the context of its use with PL-MCMC.

Original: Suppose we are presented with a training set of T observed values (not all missing the same entries), X train = { x O, 1 , x O, 2 , . . .
Revised: Suppose we are presented with a training set of T observed values (not all missing the same entries), X train = { x O, 1 , x O, 2 , . . .

Original: , x O,T } .
Revised: , x O,T } .

Original: Ideally, under the assumption that data values are missing at random (Little & Rubin, 2019), we’d wish to ﬁnd the ﬂow parameters θ that maximize the log-likelihood of X train :
Revised: Ideally, under the assumption that data values are missing at random (Little & Rubin, 2019), we’d wish to ﬁnd the ﬂow parameters θ that maximize the log-likelihood of X train :

Original: [Equation]
Revised: [Equation]

Original: Yet the complexity of the normalizing ﬂow makes an analytical computation of the marginal likelihoods of observed data entirely impractical.
Revised: Yet the complexity of the normalizing ﬂow makes an analytical computation of the marginal likelihoods of observed data entirely impractical.

Original: We therefore utilize the Expecation-Maximization (EM) algorithm
Revised: We therefore utilize the Expecation-Maximization (EM) algorithm

Original: (Dempster et al., 1977) to approach this optimization.
Revised: (Dempster et al., 1977) to approach this optimization.

Original: Following Dempster et al.
Revised: Following Dempster et al.

Original: (1977), we deﬁne Q ( θ (cid:48) | θ ) to be:
Revised: (1977), we deﬁne Q ( θ (cid:48) | θ ) to be:

Original: [Equation]
Revised: [Equation]

Original: PL-MCMC can be immediately applied to approximate these expectations.
Revised: PL-MCMC can be immediately applied to approximate these expectations.

Original: Let the set Y =
Revised: Let the set Y =

Original: { y M, 1 , y M, 2 , . . .
Revised: { y M, 1 , y M, 2 , . . .

Original: , y M,T } be created by sampling each y M,i ∼ p f,θ ( y M,i | x O,i ) using a PL-MCMC chain as described previously.
Revised: , y M,T } be created by sampling each y M,i ∼ p f,θ ( y M,i | x O,i ) using a PL-MCMC chain as described previously.

Original: We may now use the approximation:
Revised: We may now use the approximation:

Original: [Equation]
Revised: [Equation]

Original: In principle, we would then update θ following;
Revised: In principle, we would then update θ following;

Original: [Equation]
Revised: [Equation]

Original: In practice, it is more feasible to continue to train the ﬂow on the conditionally imputed version of
Revised: In practice, it is more feasible to continue to train the ﬂow on the conditionally imputed version of

Original: X train .
Revised: X train .

Original: With X (cid:48) train = { ( y M, 1 ; x O, 1 ) , ( y M, 2 ; x O, 2 ) , . . .
Revised: With X (cid:48) train = { ( y M, 1 ; x O, 1 ) , ( y M, 2 ; x O, 2 ) , . . .

Original: , ( y M,T ; x O,T ) } denoting our newly imputed training set and train being any training procedure that returns ﬂow parameters θ approximately maximizing the likelihood of a complete data training set, we rely on the approximation that: 
Revised: , ( y M,T ; x O,T ) } denoting our newly imputed training set and train being any training procedure that returns ﬂow parameters θ approximately maximizing the likelihood of a complete data training set, we rely on the approximation that: argmax θ (cid:48)

Original: [Equation]
Revised: Q ( θ (cid:48) | θ ) ≈ train ( f, X (cid:48) train ) .

Original: This approximation immediately leads to our described algorithm for the MC-EM training of normalizing ﬂows using PL-MCMC.
Revised: This approximation immediately leads to our described algorithm for the MC-EM training of normalizing ﬂows using PL-MCMC.

Original: D D ETAILS R EGARDING Q UALITATIVE E XPERIMENTS D. D ETAILS R EGARDING C ONDITIONAL I NFERENCE WITH CIFAR-10 I MAGES
Revised: D D ETAILS R EGARDING Q UALITATIVE E XPERIMENTS D. D ETAILS R EGARDING C ONDITIONAL I NFERENCE WITH CIFAR-10 I MAGES

Original: In this experiment, we infer a missing central quarter (an 8 × 8 pixel square) of CIFAR-
Revised: In this experiment, we infer a missing central quarter (an 8 × 8 pixel square) of CIFAR-

Original: (Krizhevsky et al., 2009) images.
Revised: (Krizhevsky et al., 2009) images.

Original: The CIFAR-10 dataset is composed of 60 , 000 full color 32 ×images of 10 distinct classes of objects, with 6 , 000 images provided for each class.
Revised: The CIFAR-10 dataset is composed of 60 , 000 full color 32 ×images of 10 distinct classes of objects, with 6 , 000 images provided for each class.

Original: The standard training and test set split for the CIFAR-10 dataset is 50 , 000 and 10 , 000 images, respectively.
Revised: The standard training and test set split for the CIFAR-10 dataset is 50 , 000 and 10 , 000 images, respectively.

Original: Our chosen normalizing ﬂow is a variant of the GLOW (Kingma & Dhariwal, 2018) architecture.
Revised: Our chosen normalizing ﬂow is a variant of the GLOW (Kingma & Dhariwal, 2018) architecture.

Original: We utilized a publicly available, pre-trained model (van Amersfoort, 2019) for this experiment.
Revised: We utilized a publicly available, pre-trained model (van Amersfoort, 2019) for this experiment.

Original: In the terminology of Kingma & Dhariwal (2018), the model has a depth of ﬂow (K) of 32 and a total of 3 levels (L) and ﬂow layers utilize 512 hidden channels.
Revised: In the terminology of Kingma & Dhariwal (2018), the model has a depth of ﬂow (K) of 32 and a total of 3 levels (L) and ﬂow layers utilize 512 hidden channels.

Original: The model was reportedly trained for a total of 1 , 500 epochs using Adamax with a learning rate of 5 × 10 − 4 and a batchsize of 64 .
Revised: The model was reportedly trained for a total of 1 , 500 epochs using Adamax with a learning rate of 5 × 10 − 4 and a batchsize of 64 .

Original: We presume, but cannot guarantee, that the model was trained on the standard 50 , 000 example CIFAR10 training set.
Revised: We presume, but cannot guarantee, that the model was trained on the standard 50 , 000 example CIFAR10 training set.

Original: Examples of unconditioned samples from this model are provided within Figure 10, as obtained with the standard sampling variance, σ = 1 . 

Revised: Examples of unconditioned samples from this model are provided within Figure 10, as obtained with the standard sampling variance, σ = 1 .

Original: (temperature T = 1 . 0 , in the terminology of Kingma & Dhariwal (2018)).
Revised: (temperature T = 1 . 0 , in the terminology of Kingma & Dhariwal (2018)).

Original: From these unconditioned samples, it is clear that the model has not collapsed to memorizing the training set.
Revised: From these unconditioned samples, it is clear that the model has not collapsed to memorizing the training set.

Original: The particular implementation of the normalizing ﬂow most easily provided access to a coordinate dependent representation of the latent space, which we call absolute coordinates for the latent space.
Revised: The particular implementation of the normalizing ﬂow most easily provided access to a coordinate dependent representation of the latent space, which we call absolute coordinates for the latent space.

Original: Fundamentally, the Markov Chain within PL-MCMC may utilize any convenient representation of the latent space, so long as a diffeomorphism still maps that representation back to the modeled data.
Revised: Fundamentally, the Markov Chain within PL-MCMC may utilize any convenient representation of the latent space, so long as a diffeomorphism still maps that representation back to the modeled data.

Original: For this CIFAR-10 experiment, we chose to employ a Markov Chain within the architecture’s absolute coordinates.
Revised: For this CIFAR-10 experiment, we chose to employ a Markov Chain within the architecture’s absolute coordinates.

Original: As a general note, the qualiﬁer “absolute” merely refers to the representation favored by the ﬂow’s implementation, while the qualiﬁer “relative” refers to the representation best coinciding with the chosen prior distribution for the ﬂow.
Revised: As a general note, the qualiﬁer “absolute” merely refers to the representation favored by the ﬂow’s implementation, while the qualiﬁer “relative” refers to the representation best coinciding with the chosen prior distribution for the ﬂow.

Original: The terms only reﬂect aspects of our practical usage of the representations, as there is no theoretically favored diffeomorphic representation of the latent space.
Revised: The terms only reﬂect aspects of our prac- tical usage of the representations, as there is no theoretically favored diffeomorphic representation of the latent space.

Original: For inference, we selected images from the standard CIFAR-10 test set.
Revised: For inference, we selected images from the standard CIFAR-10 test set.

Original: During inference with
Revised: During inference with

Original: PL-MCMC, latent space transitions are generated within the above mentioned absolute coordinates of the ﬂow’s latent space by small perturbations from the current latent state.
Revised: PL-MCMC, latent space transitions are generated within the above mentioned absolute coordinates of the ﬂow’s latent space by small perturbations from the current latent state.

Original: When perturbing the latent state, proposals are generated following a perturbation kernel, g p ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼
Revised: When perturbing the latent state, proposals are generated following a perturbation kernel, g p ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼

Original: N ( ξ , σ 2 p I ) .
Revised: N ( ξ , σ 2 p I ) .

Original: The auxiliary distribution, q , is chosen to target y O ∼ N ( x O , σ 2 a I ) .
Revised: The auxiliary distribution, q , is chosen to target y O ∼ N ( x O , σ 2 a I ) .

Original: For this experiment, we select σ p = 0 .
Revised: For this experiment, we select σ p = 0 .

Original: 01 and σ a = 1 × 10 − 3 .
Revised: 01 and σ a = 1 × 10 − 3 .

Original: PL-MCMC is carried out over 25,000 proposals.
Revised: PL-MCMC is carried out over 25,000 proposals.

Original: D. D ETAILS R EGARDING C ONDITIONAL I NFERENCE WITH C ELEB A I MAGES
Revised: D. D ETAILS R EGARDING C ONDITIONAL I NFERENCE WITH C ELEB A I MAGES

Original: In this experiment, we infer a missing right half (a 64 × 32 pixel rectangle) of CelebA (Liu et al., 2015) images.
Revised: In this experiment, we infer a missing right half (a 64 × 32 pixel rectangle) of CelebA (Liu et al., 2015) images.

Original: The CelebA dataset is composed of 202 , 599 full color images of celebrity faces.
Revised: The CelebA dataset is composed of 202 , 599 full color images of celebrity faces.

Original: We utilize the aligned and cropped version of the CelebA dataset, resized to a size of 64 × 64 .
Revised: We utilize the aligned and cropped version of the CelebA dataset, resized to a size of 64 × 64 .

Original: Our chosen normalizing ﬂow is a variant of the GLOW (Kingma & Dhariwal, 2018) architecture.
Revised: Our chosen normalizing ﬂow is a variant of the GLOW (Kingma & Dhariwal, 2018) architecture.

Original: We utilized a publicly available, pre-trained model (Yuki-Chai, 2019) for this experiment.
Revised: We utilized a publicly available, pre-trained model (Yuki-Chai, 2019) for this experiment.

Original: In the terminology of Kingma & Dhariwal (2018), the model has a depth of ﬂow (K) of 32 and a total of 3 levels (L) and ﬂow layers utilize 512 hidden channels.
Revised: In the terminology of Kingma & Dhariwal (2018), the model has a depth of ﬂow (K) of 32 and a total of 3 levels (L) and ﬂow layers utilize 512 hidden channels.

Original: The model was reportedly trained for a total of 1 , 500 epochs using Adamax with a learning rate of 1 × 10 − 3 and a batchsize of 12 .
Revised: The model was reportedly trained for a total of 1 , 500 epochs using Adamax with a learning rate of 1 × 10 − 3 and a batchsize of 12 .

Original: We believe that this model was trained on the entirety of the CelebA dataset, with no withheld test or validation set.
Revised: We believe that this model was trained on the entirety of the CelebA dataset, with no withheld test or validation set.

Original: Examples of unconditioned samples from this model are provided within Figuresand 12, as obtained with reduced and standard sampling variance, σ = 0 .
Revised: Examples of unconditioned samples from this model are provided within Figuresand 12, as obtained with reduced and standard sampling variance, σ = 0 .

Original: 5 and σ = 1 . 0 respectively (temperatures T = 0 .
Revised: 5 and σ = 1 . 0 respectively (temperatures T = 0 .

Original: 5 and T = 1 . 0 , in the terminology of Kingma & Dhariwal (2018)).
Revised: 5 and T = 1 . 0 , in the terminology of Kingma & Dhariwal (2018)).

Original: From these unconditioned samples, it is clear that the model has not collapsed to memorizing the training set.
Revised: From these unconditioned samples, it is clear that the model has not collapsed to memorizing the training set.

Original: As in the CIFAR-10 experiment, the particular implementation of the normalizing ﬂow most easily provided access to a coordinate dependent representation of the latent space, which we call absolute coordinates for the latent space.
Revised: As in the CIFAR-10 experiment, the particular implementation of the normalizing ﬂow most easily provided access to a coordinate dependent representation of the latent space, which we call absolute coordinates for the latent space.

Original: For this CelebA experiment, we chose to employ a
Revised: For this CelebA experiment, we chose to employ a

Original: Markov Chain within what we call relative coordinates of the latent space.
Revised: Markov Chain within what we call relative coordinates of the latent space.

Original: For this architecture, we have convenient access to three subsets of latent variables, ξ 1 , ξ 2 , and ξ 3 within absolute coordinates.
Revised: For this architecture, we have convenient access to three subsets of latent variables, ξ 1 , ξ 2 , and ξ 3 within absolute coordinates.

Original: Fixing ξ 1 (resp. ﬁxing ξ 1 and ξ 2 ) there is an invertible transformation h 2 ( ξ 2 ; ξ 1 ) (resp.
Revised: Fixing ξ 1 (resp. ﬁxing ξ 1 and ξ 2 ) there is an invertible transformation h 2 ( ξ 2 ; ξ 1 ) (resp.

Original: h 3 ( ξ 3 ; ξ 1 , ξ 2 ) ) such that h 2 ( ξ 2 ; ξ 1 ) ∼ N (0 , I ) (resp.
Revised: h 3 ( ξ 3 ; ξ 1 , ξ 2 ) ) such that h 2 ( ξ 2 ; ξ 1 ) ∼ N (0 , I ) (resp.

Original: h 3 ( ξ 3 ; ξ 1 , ξ 2 ) ∼ N (0 , I ) )
Revised: h 3 ( ξ 3 ; ξ 1 , ξ 2 ) ∼ N (0 , I ) )

Original: under our ﬂow’s prior.
Revised: under our ﬂow’s prior.

Original: The Markov Chain in relative coordinates simply follows and proposes transitions for the triplet ( ξ 1 , h 2 ( ξ 2 ; ξ 1 ) , h 3 ( ξ 3 ; ξ 1 , ξ 2 )) .
Revised: The Markov Chain in relative coordinates simply follows and proposes transitions for the triplet ( ξ 1 , h 2 ( ξ 2 ; ξ 1 ) , h 3 ( ξ 3 ; ξ 1 , ξ 2 )) .

Original: As it appears that no test set had been withheld during training of the model, we selected images at random from the full dataset for our experiment.
Revised: As it appears that no test set had been withheld during training of the model, we selected images at random from the full dataset for our experiment.

Original: During inference with PL-MCMC, latent space transitions are generated within relative coordinates of the ﬂow’s latent space by small perturbations from the current latent state.
Revised: During inference with PL-MCMC, latent space transitions are generated within relative coordinates of the ﬂow’s latent space by small perturbations from the current latent state.

Original: When perturbing the latent state, proposals are generated following a perturbation kernel, g p ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( ξ , σ 2 p I ) .
Revised: When perturbing the latent state, proposals are generated following a perturbation kernel, g p ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( ξ , σ 2 p I ) .

Original: The auxiliary distribution, q , is chosen to target y O ∼ N ( x O , σ 2 a I ) .
Revised: The auxiliary distribution, q , is chosen to target y O ∼ N ( x O , σ 2 a I ) .

Original: For this experiment, we select σ p = 0 .
Revised: For this experiment, we select σ p = 0 .

Original: 01 and σ a = 1 × 10 − 3 .
Revised: 01 and σ a = 1 × 10 − 3 .

Original: PL-MCMC is carried out over 25,000 proposals.
Revised: PL-MCMC is carried out over 25,000 proposals.

Original: D. D ETAILS R EGARDING C ONDITIONAL I NFERENCE WITH MNIST D IGITS
Revised: D. D ETAILS R EGARDING C ONDITIONAL I NFERENCE WITH MNIST D IGITS

Original: In this experiment, we infer missing portions of MNIST (LeCun et al., 1998) digits.
Revised: In this experiment, we infer missing portions of MNIST (LeCun et al., 1998) digits.

Original: The MNIST dataset is composed of 70 , 000 monochrome 28 × 28 images of handwritten digits.
Revised: The MNIST dataset is composed of 70 , 000 monochrome 28 × 28 images of handwritten digits.

Original: The standard training and test set split for the MNIST dataset is 60 , 000 and 10 , 000 images, respectively.
Revised: The standard training and test set split for the MNIST dataset is 60 , 000 and 10 , 000 images, respectively.

Original: Our data missingness mechanisms are independent missingness, where pixels are lost uniformly at random, patch missingness, where randomly located contiguous rectangular blocks are missing, and square observation, where only a randomly located contiguous square is observed.
Revised: Our data missingness mechanisms are independent missingness, where pixels are lost uniformly at random, patch missingness, where randomly located contiguous rectangular blocks are missing, and square observation, where only a randomly located contiguous square is observed.

Original: Our chosen normalizing ﬂow is a variant of the NICE (Dinh et al., 2014) architecture.
Revised: Our chosen normalizing ﬂow is a variant of the NICE (Dinh et al., 2014) architecture.

Original: Our implementation is a modiﬁcation of that by Mu (2019).
Revised: Our implementation is a modiﬁcation of that by Mu (2019).

Original: In the terminology of Kingma & Dhariwal (2018), the model has a depth of ﬂow (K) of 5 and a total of 4 levels (L) and intermediate ﬂow layers have a dimension of 1000 .
Revised: In the terminology of Kingma & Dhariwal (2018), the model has a depth of ﬂow (K) of 5 and a total of 4 levels (L) and intermediate ﬂow layers have a dimension of 1000 .

Original: The ﬂow utilizes an independent logistic prior distribution.
Revised: The ﬂow utilizes an independent logistic prior distribution.

Original: Rather than splitting even and odd pixels within coupling layers, we split between two randomly selected partitions that are chosen at the time of the ﬂow’s initialization and remain ﬁxed for all layers of the ﬂow.
Revised: Rather than splitting even and odd pixels within coupling layers, we split between two randomly selected partitions that are chosen at the time of the ﬂow’s initialization and remain ﬁxed for all layers of the ﬂow.

Original: Of course, better performance would be expected by selecting a ﬂow architecture that best suits the spatial organization of image data.
Revised: Of course, better performance would be expected by selecting a ﬂow architecture that best suits the spatial organization of image data.

Original: However, random partitioning ensures that the expected performance of the ﬂow remains independent of the spatial structuring of the data.
Revised: However, random partitioning ensures that the expected performance of the ﬂow remains independent of the spatial structuring of the data.

Original: The normalizing ﬂow is trained for 1000 epochs over the standard 60 , 000 element MNIST training set using RMSprop with a learning rate of 1 × 10 − 5 and a momentum of 0 .
Revised: The normalizing ﬂow is trained for 1000 epochs over the standard 60 , 000 element MNIST training set using RMSprop with a learning rate of 1 × 10 − 5 and a momentum of 0 .

Original: 9 and a batch size of 200 .
Revised: 9 and a batch size of 200 .

Original: The data is pre-processed by performing pixel-wise whitening of the dataset (subtracting pixel-wise observed means and dividing by pixel-wise observed standard deviation).
Revised: The data is pre-processed by performing pixel-wise whitening of the dataset (subtracting pixel-wise observed means and dividing by pixel-wise observed standard deviation).

Original: The training procedure is intended to follow that used later for training from incomplete MNIST digits to serve as a baseline for comparison and is not intended to produce the best possible generative model of MNIST digits.
Revised: The training procedure is intended to follow that used later for training from incomplete MNIST digits to serve as a baseline for comparison and is not intended to produce the best possible generative model of MNIST digits.

Original: Examples of unconditioned samples from this model are provided within Figure 13, as obtained with the standard sampling variance (temperature T = 1 . 0 , in the terminology of Kingma & Dhariwal (2018))
Revised: Examples of unconditioned samples from this model are provided within Figure 13, as obtained with the standard sampling variance (temperature T = 1 . 0 , in the terminology of Kingma & Dhariwal (2018))

Original: During inference with PL-MCMC, latent space transitions are generated within the absolute coordinates of the ﬂow’s latent space, half of the time at random by small perturbations from the current latent state and half of the time entirely resampled at a reduced standard deviation.
Revised: During inference with PL-MCMC, latent space transitions are generated within the absolute coordinates of the ﬂow’s latent space, half of the time at random by small perturbations from the current latent state and half of the time entirely resampled at a reduced standard deviation.

Original: When perturbing the latent state, proposals are generated following a perturbation kernel, g p ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( ξ , σ 2 p I ) .
Revised: When perturbing the latent state, proposals are generated following a perturbation kernel, g p ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( ξ , σ 2 p I ) .

Original: When completely resampling the latent state, proposals are generated following a resampling kernel, g r ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( 0 , σ 2 r I ) .
Revised: When completely resampling the latent state, proposals are generated following a resampling kernel, g r ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( 0 , σ 2 r I ) .

Original: The auxiliary distribution, q , is chosen to target y O ∼ N ( x O , σ 2 a I ) .
Revised: The auxiliary distribution, q , is chosen to target y O ∼ N ( x O , σ 2 a I ) .

Original: For this experiment, we select σ p = 0 .
Revised: For this experiment, we select σ p = 0 .

Original: 05 , σ r = 0 .
Revised: 05 , σ r = 0 .

Original: 5 , and σ a = 1 × 10 − 3 .
Revised: 5 , and σ a = 1 × 10 − 3 .

Original: To simplify calculation of Metropolis-Hastings acceptance probabilities, we employ the assumption that small displacements in the latent space result from g p ( ξ (cid:48) | ξ ) while large displacements result from g r ( ξ (cid:48) | ξ ) , which is valid in the limit of σ p << σ r .
Revised: To simplify calculation of Metropolis-Hastings acceptance probabilities, we employ the assumption that small displacements in the latent space result from g p ( ξ (cid:48) | ξ ) while large displacements result from g r ( ξ (cid:48) | ξ ) , which is valid in the limit of σ p << σ r .

Original: Therefore, rather than utilize the true transition kernel g ( ξ (cid:48) | ξ ) , we simply assume that g ( ξ (cid:48) | ξ ) ∝ g p ( ξ (cid:48) | ξ ) following a perturbation and g ( ξ (cid:48) | ξ ) ∝ g r ( ξ (cid:48) | ξ ) following a resample.
Revised: Therefore, rather than utilize the true transition kernel g ( ξ (cid:48) | ξ ) , we simply assume that g ( ξ (cid:48) | ξ ) ∝ g p ( ξ (cid:48) | ξ ) following a perturbation and g ( ξ (cid:48) | ξ ) ∝ g r ( ξ (cid:48) | ξ ) following a resample.

Original: PL-MCMC is carried out over 2, proposals.
Revised: PL-MCMC is carried out over 2, proposals.

Original: To determine conditional expectations, the results of 20 independent PL-MCMC chains are averaged together.
Revised: To determine conditional expectations, the results of 20 independent PL-MCMC chains are averaged together.

Original: E D ETAILS R EGARDING Q UANTITATIVE E XPERIMENTS E. D ETAILS R EGARDING T RAINING FROM I NCOMPLETE MNIST D IGITS
Revised: E D ETAILS R EGARDING Q UANTITATIVE E XPERIMENTS E. D ETAILS R EGARDING T RAINING FROM I NCOMPLETE MNIST D IGITS

Original: In this experiment, we train normalizing ﬂows to model the distribution of MNIST digits from incomplete training data.
Revised: In this experiment, we train normalizing ﬂows to model the distribution of MNIST digits from incomplete training data.

Original: Our data missingness mechanisms are independent missingness, where pixels are lost uniformly at random, patch missingness, where randomly located contiguous rectangular blocks are missing, and square observation, where only a randomly located contiguous square is observed.
Revised: Our data missingness mechanisms are independent missingness, where pixels are lost uniformly at random, patch missingness, where randomly located contiguous rectangular blocks are missing, and square observation, where only a randomly located contiguous square is observed.

Original: For each missingness mechanism, we consider missingness rates of 0 . 3 , 0 . 6 , and 0 .
Revised: For each missingness mechanism, we consider missingness rates of 0 . 3 , 0 . 6 , and 0 .

Original: For training, we apply the missingness mechanism to the standard MNIST training set, resulting in a training set of 60 , 000 incomplete digits.
Revised: For training, we apply the missingness mechanism to the standard MNIST training set, resulting in a training set of 60 , 000 incomplete digits.

Original: For testing, we apply the missingness mechanism to the standard MNIST test set, resulting in a test set of 10 , 000 incomplete digits.
Revised: For testing, we apply the missingness mechanism to the standard MNIST test set, resulting in a test set of 10 , 000 incomplete digits.

Original: Our chosen normalizing ﬂow is a variant of the NICE (Dinh et al., 2014) architecture.
Revised: Our chosen normalizing ﬂow is a variant of the NICE (Dinh et al., 2014) architecture.

Original: In the terminology of Kingma & Dhariwal (2018), the model has a depth of ﬂow (K) of 5 and a total oflevels (L) and intermediate ﬂow layers have a dimension of 1000 .
Revised: In the terminology of Kingma & Dhariwal (2018), the model has a depth of ﬂow (K) of 5 and a total oflevels (L) and intermediate ﬂow layers have a dimension of 1000 .

Original: The ﬂow utilizes an independent logistic prior distribution.
Revised: The ﬂow utilizes an independent logistic prior distribution.

Original: Rather than splitting even and odd pixels within coupling layers, we split between two randomly selected partitions that are chosen at the time of the ﬂow’s initialization and remain ﬁxed for all layers of the ﬂow.
Revised: Rather than splitting even and odd pixels within coupling layers, we split between two randomly selected partitions that are chosen at the time of the ﬂow’s initialization and remain ﬁxed for all layers of the ﬂow.

Original: The ﬂow architecture and implementation is the same as used above for training from complete MNIST digits.
Revised: The ﬂow architecture and implementation is the same as used above for training from complete MNIST digits.

Original: Better performance could be obtained by selecting an architecture that best suits the spatial organization of image data or by scaling model complexity along with the missingness rate (at a missingness rate of 0 .
Revised: Better performance could be obtained by selecting an architecture that best suits the spatial organization of image data or by scaling model complexity along with the missingness rate (at a missingness rate of 0 .

Original: 9 , we have a tenth as much observed data available for training as in the complete data case).
Revised: 9 , we have a tenth as much observed data available for training as in the complete data case).

Original: In all cases, the normalizing ﬂow is trained for 1000 epochs using RMSprop with a learning rate of 1 × 10 − 5 and a momentum of 0 .
Revised: In all cases, the normalizing ﬂow is trained for 1000 epochs using RMSprop with a learning rate of 1 × 10 − 5 and a momentum of 0 .

Original: 9 and a batch size of 200 .
Revised: 9 and a batch size of 200 .

Original: The data is pre-processed by performing pixel-wise whitening of the dataset (subtracting pixel-wise observed means and dividing by pixel-wise observed standard deviation).
Revised: The data is pre-processed by performing pixel-wise whitening of the dataset (subtracting pixel-wise observed means and dividing by pixel-wise observed standard deviation).

Original: At each of the ﬁrst 50 epochs of training, missing pixels are resampled following an independent normal distribution, such that x M ∼ N ( 0 , I ) (in whitened coordinates).
Revised: At each of the ﬁrst 50 epochs of training, missing pixels are resampled following an independent normal distribution, such that x M ∼ N ( 0 , I ) (in whitened coordinates).

Original: Every 50 epochs thereafter, missing values are resampled using PL-MCMC as applied to the ﬂow being trained.
Revised: Every 50 epochs thereafter, missing values are resampled using PL-MCMC as applied to the ﬂow being trained.

Original: During inference with PL-MCMC, latent space transitions are generated within the absolute coordinates of the ﬂow’s latent space, half of the time at random by small perturbations from the current latent state and half of the time entirely resampled at a reduced standard deviation.
Revised: During inference with PL-MCMC, latent space transitions are generated within the absolute coordinates of the ﬂow’s latent space, half of the time at random by small perturbations from the current latent state and half of the time entirely resampled at a reduced standard deviation.

Original: When perturbing the latent state, proposals are generated following a perturbation kernel, g p ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( ξ , σ 2 p I ) .
Revised: When perturbing the latent state, proposals are generated following a perturbation kernel, g p ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( ξ , σ 2 p I ) .

Original: When completely resampling the latent state, proposals are generated following a resampling kernel, g r ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( 0 , σ 2 r I ) .
Revised: When completely resampling the latent state, proposals are generated following a resampling kernel, g r ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( 0 , σ 2 r I ) .

Original: The auxiliary distribution, q , is chosen to target y O ∼ N ( x O , σ 2 a I ) .
Revised: The auxiliary distribution, q , is chosen to target y O ∼ N ( x O , σ 2 a I ) .

Original: For computation of Metropolis-Hastings probabilities, we employ the same approximation as used when inferring MNIST digits within the qualitative experiments.
Revised: For computation of Metropolis-Hastings probabilities, we employ the same approximation as used when inferring MNIST digits within the qualitative experiments.

Original: Throughout training, we use σ p = 0 .
Revised: Throughout training, we use σ p = 0 .

Original: 05 and σ a = 1 × 10 − 3 .
Revised: 05 and σ a = 1 × 10 − 3 .

Original: Within the ﬁrst 500 epochs, we utilize σ p = 1 . 814 .
Revised: Within the ﬁrst 500 epochs, we utilize σ p = 1 . 814 .

Original: After the ﬁrst 500 epochs, we resample at reduced variance with σ p = 0 .
Revised: After the ﬁrst 500 epochs, we resample at reduced variance with σ p = 0 .

Original: These parameters and this training procedure were chosen because they provided acceptable performance when applied to training data with the moderate missingness rate of 0 . 6 .
Revised: These parameters and this training procedure were chosen because they provided acceptable performance when applied to training data with the moderate missingness rate of 0 . 6 .

Original: It would certainly be beneﬁcial to determine a more principled approach to their selection.
Revised: It would certainly be beneﬁcial to determine a more principled approach to their selection.

Original: After
Revised: After

Original: PL-MCMC sampling
Revised: PL-MCMC sampling

Original: , we clamp values between pixel-wise observed minimal and maximal values to produce a newly imputed training set for use in the next 50 epochs of training.
Revised: , we clamp values between pixel-wise observed minimal and maximal values to produce a newly imputed training set for use in the next 50 epochs of training.

Original: Figure 14 below illustrates the progression of how the training set is imputed over training.
Revised: Figure 14 below illustrates the progression of how the training set is imputed over training.

Original: [Figure]
Revised: [Figure]

Original: For testing, we utilize the normalizing ﬂows to reconstruct the 10 , 000 element incomplete test sets.
Revised: For testing, we utilize the normalizing ﬂows to reconstruct the 10 , 000 element incomplete test sets.

Original: For this reconstruction, we utilize PL-MCMC chains over 2 , 000 proposals following the same procedure as in training.
Revised: For this reconstruction, we utilize PL-MCMC chains over 2 , 000 proposals following the same procedure as in training.

Original: During testing, we use σ p = 0 .
Revised: During testing, we use σ p = 0 .

Original: 05 , σ p = 0 . 5 , and σ a = 1 × 10 − 3 .
Revised: 05 , σ p = 0 . 5 , and σ a = 1 × 10 − 3 .

Original: To collect statistics for performance measures, we train three normalizing ﬂows on distinctly prepared (i.e. resampled missingness patterns) training sets, each of which is tested on ﬁve distinctly prepared test sets.
Revised: To collect statistics for performance measures, we train three normalizing ﬂows on distinctly prepared (i.e. resampled missingness patterns) training sets, each of which is tested on ﬁve distinctly prepared test sets.

Original: We consider two methods of employing PL-MCMC to produce reconstructions of missing data.
Revised: We consider two methods of employing PL-MCMC to produce reconstructions of missing data.

Original: The ﬁrst method is imputation with a single sample from a PL-MCMC chain (imputation with a sample from the conditional distribution) and the second method is imputation with the average across multiple samples from independent PL-MCMC chains (imputation with the conditional mean).
Revised: The ﬁrst method is imputation with a single sample from a PL-MCMC chain (imputation with a sample from the conditional distribution) and the second method is imputation with the average across multiple samples from independent PL-MCMC chains (imputation with the conditional mean).

Original: As we average across the samples from 10 independent PL-MCMC chains for the second method, our reported statistics for the ﬁrst method encompass these additional 10 replications for individual sample imputation performance.
Revised: As we average across the samples from 10 independent PL-MCMC chains for the second method, our reported statistics for the ﬁrst method encompass these additional 10 replications for individual sample imputation performance.

Original: As imputation performance measures, we consider per-pixel reconstruction RMSE and Fr´echet Inception Distance (Heusel et al., 2017).
Revised: As imputation performance measures, we consider per-pixel reconstruction RMSE and Fr´echet Inception Distance (Heusel et al., 2017).

Original: To compute Fr´echet Inception Distance, we use the implementation provided along with that of MisGAN (Li, 2019).
Revised: To compute Fr´echet Inception Distance, we use the implementation provided along with that of MisGAN (Li, 2019).

Original: Reconstruction RMSE is recorded within the original, unwhitened representation of pixel data.
Revised: Reconstruction RMSE is recorded within the original, unwhitened representation of pixel data.

Original: When imputing T test examples with M i denoting the set of pixel indices that are missing for the i -th example and ˆ x i,j denoting our imputed estimate for the j -th pixel of the i -th example (having ground truth value x i,j ), our reconstruction
Revised: When imputing T test examples with M i denoting the set of pixel indices that are missing for the i -th example and ˆ x i,j denoting our imputed estimate for the j -th pixel of the i -th example (having ground truth value x i,j ), our reconstruction

Original: RMSE is calculated as:
Revised: RMSE is calculated as:

Original: [Equation]
Revised: [Equation]

Original: For comparison, we consider imputation using pixel-wise observed means and imputation using the convolutional variant of MisGAN (Li et al., 2018).
Revised: For comparison, we consider imputation using pixel-wise observed means and imputation using the convolutional variant of MisGAN (Li et al., 2018).

Original: We use the implementation of MisGAN provided by Li (2019).
Revised: We use the implementation of MisGAN provided by Li (2019).

Original: The MisGAN models are trained following the provided default parameters (500 epochs with a batch size of 64 with τ = 0 , α = 0 .
Revised: The MisGAN models are trained following the provided default parameters (500 epochs with a batch size of 64 with τ = 0 , α = 0 .

Original: 1 , β = 0 .
Revised: 1 , β = 0 .

Original: 1 , γ = 0 , maskgen = fusion , gp lambda = 10 , n critic = 5 , and n latent = 128 , with a three layer fully connected imputer network with 784 units in each layer).
Revised: 1 , γ = 0 , maskgen = fusion , gp lambda = 10 , n critic = 5 , and n latent = 128 , with a three layer fully connected imputer network with 784 units in each layer).

Original: E. D ETAILS R EGARDING T RAINING FROM I NCOMPLETE UCI D ATASETS
Revised: E. D ETAILS R EGARDING T RAINING FROM I NCOMPLETE UCI D ATASETS

Original: In this experiment, we train normalizing ﬂows to model the distributions of various continuous UCI datasets (Bache & Lichman, 2013) from incomplete training data.
Revised: In this experiment, we train normalizing ﬂows to model the distributions of various continuous UCI datasets (Bache & Lichman, 2013) from incomplete training data.

Original: In all cases, our data missingness mechanism is independent missingness with a missingness rate of 0 . 5 .
Revised: In all cases, our data missingness mechanism is independent missingness with a missingness rate of 0 . 5 .

Original: A summary of the UCI datasets used in this experiment is provided below in Table 3.
Revised: A summary of the UCI datasets used in this experiment is provided below in Table 3.

Original: For training, we apply the missingness mechanism to the entirety of a single copy of the dataset.
Revised: For training, we apply the missingness mechanism to the entirety of a single copy of the dataset.

Original: For testing, we attempt to reconstruct the missing portions of the training set.
Revised: For testing, we attempt to reconstruct the missing portions of the training set.

Original: [Table]
Revised: [Table]

Original: Our chosen normalizing ﬂows are variants of the NICE (Dinh et al., 2014) architecture.
Revised: Our chosen normalizing ﬂows are variants of the NICE (Dinh et al., 2014) architecture.

Original: In the terminology of Kingma & Dhariwal (2018), all models have a depth of ﬂow (K) of 5 and a total of 4 levels (L) and intermediate ﬂow layers have a dimension of 120 .
Revised: In the terminology of Kingma & Dhariwal (2018), all models have a depth of ﬂow (K) of 5 and a total of 4 levels (L) and intermediate ﬂow layers have a dimension of 120 .

Original: The ﬂows utilize an independent normal prior distribution.
Revised: The ﬂows utilize an independent normal prior distribution.

Original: Rather than splitting even and odd pixels within coupling layers, we split between two randomly selected partitions that are chosen at the time of the ﬂow’s initialization and remain ﬁxed for all layers of the ﬂow.
Revised: Rather than splitting even and odd pixels within coupling layers, we split between two randomly selected partitions that are chosen at the time of the ﬂow’s initialization and remain ﬁxed for all layers of the ﬂow.

Original: As our implementation works most easily with an even number of attributes, we copy the concrete attributes and data missingness to double the number of attributes.
Revised: As our implementation works most easily with an even number of attributes, we copy the concrete attributes and data missingness to double the number of attributes.

Original: By copying data missingness patterns, we ensure that the doubling does not introduce additional information for training.
Revised: By copying data missingness patterns, we ensure that the doubling does not introduce additional information for training.

Original: A summary of input attribute dimensions and training batch sizes is provided within Table 4.
Revised: A summary of input attribute dimensions and training batch sizes is provided within Table 4.

Original: [Table]
Revised: [Table]

Original: In all cases, the normalizing ﬂow is trained for 1000 epochs using Adamax with a learning rate of 0 .
Revised: In all cases, the normalizing ﬂow is trained for 1000 epochs using Adamax with a learning rate of 0 .

Original: 002 , β 1 = 0 .
Revised: 002 , β 1 = 0 .

Original: 9 , and β 2 = 0 .
Revised: 9 , and β 2 = 0 .

Original: We duplicate the training sets (data missingness patterns included) 10 times to form a larger training set without introducing additional information beyond that present in the original incomplete training set.
Revised: We duplicate the training sets (data missingness patterns included) 10 times to form a larger training set without introducing additional information beyond that present in the original incomplete training set.

Original: The batch sizes used are listed in Table 4.
Revised: The batch sizes used are listed in Table 4.

Original: The data is pre-processed by performing attribute-wise whitening of the dataset (subtracting attributewise observed means and dividing by attribute-wise observed standard deviation).
Revised: The data is pre-processed by performing attribute-wise whitening of the dataset (subtracting attributewise observed means and dividing by attribute-wise observed standard deviation).

Original: At each of the ﬁrst 50 epochs of training, missing attribute are resampled following an independent normal distribution, such that x M ∼ N ( 0 , I ) (in whitened coordinates).
Revised: At each of the ﬁrst 50 epochs of training, missing attribute are resampled following an independent normal distribution, such that x M ∼ N ( 0 , I ) (in whitened coordinates).

Original: Every 50 epochs thereafter, missing values are resampled using PL-MCMC as applied to the ﬂow being trained.
Revised: Every 50 epochs thereafter, missing values are resampled using PL-MCMC as applied to the ﬂow being trained.

Original: During inference with PLMCMC, latent space transitions are generated within the absolute coordinates of the ﬂow’s latent space, half of the time at random by small perturbations from the current latent state and half of the time entirely resampled at a reduced standard deviation.
Revised: During inference with PLMCMC, latent space transitions are generated within the absolute coordinates of the ﬂow’s latent space, half of the time at random by small perturbations from the current latent state and half of the time entirely resampled at a reduced standard deviation.

Original: When perturbing the latent state, proposals are generated following a perturbation kernel, g p ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( ξ , σ 2 p I ) .
Revised: When perturbing the latent state, proposals are generated following a perturbation kernel, g p ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( ξ , σ 2 p I ) .

Original: When completely resampling the latent state, proposals are generated following a resampling kernel, g r ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( 0 , σ 2 r I ) .
Revised: When completely resampling the latent state, proposals are generated following a resampling kernel, g r ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( 0 , σ 2 r I ) .

Original: The auxiliary distribution, q , is chosen to target y O ∼ N ( x O , σ 2 a I ) .
Revised: The auxiliary distribution, q , is chosen to target y O ∼ N ( x O , σ 2 a I ) .

Original: For computation of Metropolis-Hastings probabilities, we employ the same approximation as used when inferring MNIST digits within the qualitative experiments.
Revised: For computation of Metropolis-Hastings probabilities, we employ the same approximation as used when inferring MNIST digits within the qualitative experiments.

Original: Throughout training, we use σ p = 0 .
Revised: Throughout training, we use σ p = 0 .

Original: 01 , σ r = 1 . 0 , σ a = 1 × 10 − 3 .
Revised: 01 , σ r = 1 . 0 , σ a = 1 × 10 − 3 .

Original: These parameters and this training procedure were chosen because they provided acceptable performance across the UCI datasets considered.
Revised: These parameters and this training procedure were chosen because they provided acceptable performance across the UCI datasets considered.

Original: It would certainly be beneﬁcial to determine a more principled approach to their selection.
Revised: It would certainly be beneﬁcial to determine a more principled approach to their selection.

Original: After PL-MCMC sampling, we clamp values between attribute-wise observed minimal and maximal values to produce a newly imputed training set for use in the next 50 epochs of training.
Revised: After PL-MCMC sampling, we clamp values between attribute-wise observed minimal and maximal values to produce a newly imputed training set for use in the next 50 epochs of training.

Original: For testing, we utilize the normalizing ﬂows to reconstruct the missing values from their training sets.
Revised: For testing, we utilize the normalizing ﬂows to reconstruct the missing values from their training sets.

Original: For this reconstruction, we utilize PL-MCMC chains over 2 , 000 proposals following the same procedure as in training.
Revised: For this reconstruction, we utilize PL-MCMC chains over 2 , 000 proposals following the same procedure as in training.

Original: During testing, we use σ p = 0 .
Revised: During testing, we use σ p = 0 .

Original: 01 , σ p = 1 . 0 , and σ a = 1 × 10 − 3 .
Revised: 01 , σ p = 1 . 0 , and σ a = 1 × 10 − 3 .

Original: To collect statistics for performance measures, we train ﬁve normalizing ﬂows on distinctly prepared (i.e. resampled missingness patterns) training sets.
Revised: To collect statistics for performance measures, we train ﬁve normalizing ﬂows on distinctly prepared (i.e. resampled missingness patterns) training sets.

Original: We consider two methods of employing PLMCMC to produce reconstructions of missing data.
Revised: We consider two methods of employing PLMCMC to produce reconstructions of missing data.

Original: The ﬁrst method is imputation with a single sample from a PL-MCMC chain (imputation with a sample from the conditional distribution) and the second method is imputation with the average across multiple samples from independent PLMCMC chains (imputation with the conditional mean).
Revised: The ﬁrst method is imputation with a single sample from a PL-MCMC chain (imputation with a sample from the conditional distribution) and the second method is imputation with the average across multiple samples from independent PLMCMC chains (imputation with the conditional mean).

Original: As we average across the samples fromindependent PL-MCMC chains for the second method, our reported statistics for the ﬁrst method encompass these additional 25 replications for individual sample imputation performance.
Revised: As we average across the samples fromindependent PL-MCMC chains for the second method, our reported statistics for the ﬁrst method encompass these additional 25 replications for individual sample imputation performance.

Original: In the case of the concrete dataset, we consider the copied attribute values as an additional single sample from the conditional distribution that is also incorporated into averaging.
Revised: In the case of the concrete dataset, we consider the copied attribute values as an additional single sample from the conditional distribution that is also incorporated into averaging.

Original: As an imputation performance measure, we consider per-attribute normalized MSE.
Revised: As an imputation performance measure, we consider per-attribute normalized MSE.

Original: When imputing
Revised: When imputing

Original: T test examples with M i denoting the set of attribute indices that are missing for the i -th example and ˆ x i,j denoting our imputed estimate for the j -th attribute of the i -th example (having ground truth value x i,j ), our normalized MSE is calculated as:
Revised: T test examples with M i denoting the set of attribute indices that are missing for the i -th example and ˆ x i,j denoting our imputed estimate for the j -th attribute of the i -th example (having ground truth value x i,j ), our normalized MSE is calculated as:

Original: [Equation]
Revised: [Equation]

Original: where σ j denotes the ground truth standard deviation of the j -th attribute.
Revised: where σ j denotes the ground truth standard deviation of the j -th attribute.

Original: For comparison, we consider imputation using attribute-wise observed means, imputation using the missForest (Stekhoven & B¨uhlmann, 2012) R package with default parameters, and imputation with
Revised: For comparison, we consider imputation using attribute-wise observed means, imputation using the missForest (Stekhoven & B¨uhlmann, 2012) R package with default parameters, and imputation with

Original: VAEs using MIWAE (Mattei & Frellsen, 2019).
Revised: VAEs using MIWAE (Mattei & Frellsen, 2019).

Original: For imputation with missForrest, no data preprocessing is employed.
Revised: For imputation with missForrest, no data preprocessing is employed.

Original: We use the implementation of MIWAE provided by Mattei (2019).
Revised: We use the implementation of MIWAE provided by Mattei (2019).

Original: In all cases, the VAE architecture employed has an intrinsic dimension d of 10 , an encoder and decoder comprised of 3 layers each with 128 hidden units with ReLU activation functions, an independent normal prior, and a Student’s t distribution observation model.
Revised: In all cases, the VAE architecture employed has an intrinsic dimension d of 10 , an encoder and decoder comprised of 3 layers each with 128 hidden units with ReLU activation functions, an independent normal prior, and a Student’s t distribution observation model.

Original: In all cases, we utilize zero imputation as the MIWAE imputation function.
Revised: In all cases, we utilize zero imputation as the MIWAE imputation function.

Original: For training, we use 20 importance weights while for inference we use 10 , 000 importance weights.
Revised: For training, we use 20 importance weights while for inference we use 10 , 000 importance weights.

Original: We train the models using the provided default parameters ( 2 , 000 epochs using Adam with a learning rate of 0 .
Revised: We train the models using the provided default parameters ( 2 , 000 epochs using Adam with a learning rate of 0 .

Original: 001 and a batch size of 64 ).
Revised: 001 and a batch size of 64 ).

Original: In all cases, the data is pre-processed by performing attribute-wise whitening of the dataset (subtracting attribute-wise observed means and dividing by attribute-wise observed standard deviation).
Revised: In all cases, the data is pre-processed by performing attribute-wise whitening of the dataset (subtracting attribute-wise observed means and dividing by attribute-wise observed standard deviation).

Original: E. D ETAILS R EGARDING S AMPLING E FFICIENCY FOR I NFERENCE OF MNIST D IGIT
Revised: E. D ETAILS R EGARDING S AMPLING E FFICIENCY FOR I NFERENCE OF MNIST D IGIT

Original: In these experiments, we use MCMC to estimate the conditional expectation for the missing portion of a single MNIST digit.
Revised: In these experiments, we use MCMC to estimate the conditional expectation for the missing portion of a single MNIST digit.

Original: In this case, the data missingness mechanism is a checkerboard pattern masking half of the digit, as shown in Figure 15.
Revised: In this case, the data missingness mechanism is a checkerboard pattern masking half of the digit, as shown in Figure 15.

Original: To perform this inference, we use the same normalizing ﬂow as used in the qualitative experiments in Section 5.3, detailed within Appendix D.3.
Revised: To perform this inference, we use the same normalizing ﬂow as used in the qualitative experiments in Section 5.3, detailed within Appendix D.3.

Original: During inference with PL-MCMC, latent space transitions are generated within the absolute coordinates of the ﬂow’s latent space, half of the time at random by small perturbations from the current latent state and half of the time entirely resampled at a reduced standard deviation.
Revised: During inference with PL-MCMC, latent space transitions are generated within the absolute coordinates of the ﬂow’s latent space, half of the time at random by small perturbations from the current latent state and half of the time entirely resampled at a reduced standard deviation.

Original: When perturbing the latent state, proposals are generated following a perturbation kernel, g p ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( ξ , σ 2 p I ) .
Revised: When perturbing the latent state, proposals are generated following a perturbation kernel, g p ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( ξ , σ 2 p I ) .

Original: When completely resampling the latent state, proposals are generated following a resampling kernel, g r ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( 0 , σ 2 r I ) .
Revised: When completely resampling the latent state, proposals are generated following a resampling kernel, g r ( ξ (cid:48) | ξ ) , such that ξ (cid:48) ∼ N ( 0 , σ 2 r I ) .

Original: The auxiliary distribution, q , is chosen to target y O ∼ N ( x O , σ 2 a I ) .
Revised: The auxiliary distribution, q , is chosen to target y O ∼ N ( x O , σ 2 a I ) .

Original: For this experiment, unless otherwise speciﬁed, we select σ p = 0 .
Revised: For this experiment, unless otherwise speciﬁed, we select σ p = 0 .

Original: 01 , σ r = 1 . 0 , and σ a = 1 × 10 − 3 .
Revised: 01 , σ r = 1 . 0 , and σ a = 1 × 10 − 3 .

Original: For computation of Metropolis-Hastings probabilities, we employ the same approximation as used when inferring
Revised: For computation of Metropolis-Hastings probabilities, we employ the same approximation as used when inferring

Original: MNIST digits within the qualitative experiments.
Revised: MNIST digits within the qualitative experiments.

Original: Ideally, for comparison with PL-MCMC, we would consider employing a Metropolis-Hastings
Revised: Ideally, for comparison with PL-MCMC, we would consider employing a Metropolis-Hastings

Original: MCMC through the modeled data space proposing x (cid:48)
Revised: MCMC through the modeled data space proposing x (cid:48)

Original: M ∼ N ( x M , σ 2 MH I ) , for some appropriately chosen σ MH .
Revised: M ∼ N ( x M , σ 2 MH I ) , for some appropriately chosen σ MH .

Original: However, we found that naive Metropolis-Hastings through the modeled data space required such a small σ MH that the Markov Chain made no discernible progress.
Revised: However, we found that naive Metropolis-Hastings through the modeled data space required such a small σ MH that the Markov Chain made no discernible progress.

Original: We therefore resorted to per-missing-pixel Gibbs sampling, with missing pixel proposals generated following x (cid:48)
Revised: We therefore resorted to per-missing-pixel Gibbs sampling, with missing pixel proposals generated following x (cid:48)

Original: M,i ∼ N ( µ i , σ 2 i ) , with µ i and
Revised: M,i ∼ N ( µ i , σ 2 i ) , with µ i and

Original: σ i denoting the i -th missing pixel’s observed mean and standard deviation throughout the training set.
Revised: σ i denoting the i -th missing pixel’s observed mean and standard deviation throughout the training set.

Original: For both techniques, the initial state of Markov Chain (latent state for PL-MCMC and missing pixel values for Gibbs sampling) is determined by unconditional sampling from the model at standard variance (temperature T = 1 . 0 , in the terminology of Kingma & Dhariwal (2018)).
Revised: For both techniques, the initial state of Markov Chain (latent state for PL-MCMC and missing pixel values for Gibbs sampling) is determined by unconditional sampling from the model at standard variance (temperature T = 1 . 0 , in the terminology of Kingma & Dhariwal (2018)).

Original: In each replication, the conditional mean is estimated from the average of 100 independent Markov Chains.
Revised: In each replication, the conditional mean is estimated from the average of 100 independent Markov Chains.

Original: To gather statistics regarding the variance of estimated conditional mean RMSE, we perform 10 separate replications of the experiment.
Revised: To gather statistics regarding the variance of estimated conditional mean RMSE, we perform 10 separate replications of the experiment.

Original: As the evaluation of a PL-MCMC Metropolis-Hastings proposal involved two transformations through the normalizing ﬂow while the evaluation of a Gibbs sampling proposal involves only one transformation, it can be argued that each PL-MCMC proposal was twice as costly as each Gibbs sample.
Revised: As the evaluation of a PL-MCMC Metropolis-Hastings proposal involved two transformations through the normalizing ﬂow while the evaluation of a Gibbs sampling proposal involves only one transformation, it can be argued that each PL-MCMC proposal was twice as costly as each Gibbs sample.

Original: For this reason, Figures 16 and 17 perform the same comparisons as Figures 7 and 8, but with respect to the number of ﬂow transformations utilized in the Markov Chains.
Revised: For this reason, Figures 16 and 17 perform the same comparisons as Figures 7 and 8, but with respect to the number of ﬂow transformations utilized in the Markov Chains.

Original: We can see that, even accounting for the relative computational costs of the two methods, PL-MCMC offers signiﬁcantly improved sampling performance.
Revised: We can see that, even accounting for the relative computational costs of the two methods, PL-MCMC offers signiﬁcantly improved sampling performance.

Original: Figures 18 and 19 compare the effect of altering proposal distribution scale when the transition proposals are generated only by the perturbation kernel.
Revised: Figures 18 and 19 compare the effect of altering proposal distribution scale when the transition proposals are generated only by the perturbation kernel.

Original: For reference, the results from our default σ p = 0 . 01 , σ r = 1 . 0 , and σ a = 1 × 10 − 3 PL-MCMC implementation are also included in red.
Revised: For reference, the results from our default σ p = 0 . 01 , σ r = 1 . 0 , and σ a = 1 × 10 − 3 PL-MCMC implementation are also included in red.

Original: Figures 20 and 21 compare the effect of altering the scale of the transition proposal’s resampling kernel.
Revised: Figures 20 and 21 compare the effect of altering the scale of the transition proposal’s resampling kernel.

Original: E. R ESTRICTIVE A UXILIARY D ISTRIBUTIONS D O N OT N ECESSARILY R EDUCE PL-MCMC T O S TOCHASTIC S EARCH
Revised: E. R ESTRICTIVE A UXILIARY D ISTRIBUTIONS D O N OT N ECESSARILY R EDUCE PL-MCMC T O S TOCHASTIC S EARCH

Original: Given that our results from auxiliary distributions with standard deviations of σ a = 10 − 3 and σ a = 10 − 6 closely overlap we may be concerned that the auxiliary distribution might dominate
Revised: Given that our results from auxiliary distributions with standard deviations of σ a = 10 − 3 and σ a = 10 − 6 closely overlap we may be concerned that the auxiliary distribution might dominate

Original: PL-MCMC’s behavior
Revised: PL-MCMC’s behavior

Original: and reduce the procedure to a simple search in the latent space to best rebuild the observed data.
Revised: and reduce the procedure to a simple search in the latent space to best rebuild the observed data.

Original: To determine whether an auxiliary distribution with σ a = 10 − 3 overwhelmingly dominates the conditional sampling process we follow a PL-MCMC implementation with σ a = 10 − 3 and determine how often the its decisions regarding proposal acceptance would be contradicted by an implementation with σ a = ∞ .
Revised: To determine whether an auxiliary distribution with σ a = 10 − 3 overwhelmingly dominates the conditional sampling process we follow a PL-MCMC implementation with σ a = 10 − 3 and determine how often the its decisions regarding proposal acceptance would be contradicted by an implementation with σ a = ∞ .

Original: These results are provided within Figure
Revised: These results are provided within Figure

Original: Fundamentally, we see that, at least with σ a = 10 − 3 in this particular task, the two implementations are likely to agree in their decisions to accept or reject particular proposal transitions.
Revised: Fundamentally, we see that, at least with σ a = 10 − 3 in this particular task, the two implementations are likely to agree in their decisions to accept or reject particular proposal transitions.

Original: Choosing an auxiliary distribution with σ a = 10 − 3 does not overwhelmingly “bully” the Markov Chain into mindlessly reconstructing observed data.
Revised: Choosing an auxiliary distribution with σ a = 10 − 3 does not overwhelmingly “bully” the Markov Chain into mindlessly reconstructing observed data.

Original: Still, there is a substantial probability that this choice of auxiliary distribution will alter the decision, which is the mechanism by which the auxiliary distribution helps to guide the Markov Chain and improve conditional sampling performance.
Revised: Still, there is a substantial probability that this choice of auxiliary distribution will alter the decision, which is the mechanism by which the auxiliary distribution helps to guide the Markov Chain and improve conditional sampling performance.

Original: We suspect that these decision change probability computations could be useful for tuning the choice of auxiliary distribution.
Revised: We suspect that these decision change probability computations could be useful for tuning the choice of auxiliary distribution.