Original: P ERCEPTUAL G ENERATIVE A UTOENCODERS
Revised: P ERCEPTUAL G ENERATIVE A UTOENCODERS

Original: Anonymous authors Paper under double-blind review
Revised: CIFAR Fellow

Original: ABSTRACT
Revised: A

Revised: BSTRACT

Original: Modern generative models are usually designed to match target distributions directly in the data space, where  intrinsic dimensionality of data can be much lower than  ambient dimensionality.
Revised: Modern generative models are usually designed to match target distributions directly in the data space, where the intrinsic dimensionality of data can be much lower than the ambient dimensionality.

Original: We argue that this discrepancy may contribute to the difﬁculties in training generative models.
Revised: We argue that this discrepancy may contribute to the difﬁculties in training generative models.

Original: We therefore propose to map both the generated and target distributions to the latent space using the encoder of a standard autoencoder, and train the generator (or decoder) to match the target distribution in the latent space.
Revised: We therefore propose to map both the generated and target distributions to the latent space using the encoder of a standard autoencoder, and train the generator (or decoder) to match the target distribution in the latent space.

Original: The resulting method, perceptual generative autoencoder (PGA), is then incorporated with maximum likelihood or variational autoencoder (VAE) objective to train the generative model.
Revised: The resulting method, perceptual generative autoencoder (PGA), is then incorporated with maximum likelihood or variational autoencoder (VAE) objective to train the generative model.

Original: With maximum likelihood, PGA generalizes the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities.
Revised: With maximum likelihood, PGA generalizes the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities.

Original: When combined with VAE, PGA can generate sharper samples than vanilla VAE.
Revised: When combined with VAE, PGA can generate sharper samples than vanilla VAE.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent years have witnessed great interest in generative models, mainly due to the success of generative adversarial networks (GANs) (Goodfellow et al., 2014;
Revised: Recent years have witnessed great interest in generative models, mainly due to the success of generative adversarial networks (GANs) (Goodfellow et al., 2014;

Original: Radford et al., 2016; Karras et al., 2018; Brock et al., 2019).
Revised: Radford et al., 2016; Karras et al., 2018; Brock et al., 2019).

Original: Despite the prevalence, the adversarial nature of GANs can lead to a number of challenges, such as unstable training dynamics and mode collapse.
Revised: Despite the prevalence, the adversarial nature of GANs can lead to a number of challenges, such as unstable training dynamics and mode collapse.

Original: Since the advent of GANs, substantial efforts have been devoted to addressing these challenges (Salimans et al., 2016; Arjovsky et al., 2017; Gulrajani et al., 2017; Miyato et al., 2018), while non-adversarial approaches that are free of these issues have also gained attention.
Revised: Since the advent of GANs, substantial efforts have been devoted to addressing these challenges (Salimans et al., 2016; Arjovsky et al., 2017; Gulrajani et al., 2017; Miyato et al., 2018), while non-adversarial approaches that are free of these issues have also gained attention.

Original: Examples include variational autoencoders (VAEs) (Kingma & Welling, 2014), reversible generative models (Dinh et al., 2017; Kingma & Dhariwal, 2018), and Wasserstein autoencoders (WAEs) (Tolstikhin et al., 2018).
Revised: Examples include variational autoencoders (VAEs) (Kingma & Welling, 2014), reversible generative models (Dinh et al., 2017; Kingma & Dhariwal, 2018), and Wasserstein autoencoders (WAEs) (Tolstikhin et al., 2018).

Original: However, non-adversarial approaches often have signiﬁcant limitations.
Revised: However, non-adversarial approaches often have signiﬁcant limitations.

Original: For instance, VAEs tend to generate blurry samples, while reversible generative models require restricted neural network architectures or solving neural differential equations (Grathwohl et al., 2019).
Revised: For instance, VAEs tend to generate blurry samples, while reversible generative models require restricted neural network architectures or solving neural differential equations (Grathwohl et al., 2019).

Original: Furthermore, to use the change of variable formula, the latent space of a reversible model must have the same dimensionality as the data space, which is unreasonable considering that real-world, high-dimensional data (e.g., images) tends to lie on low-dimensional manifolds, and thus results in redundant latent dimensions and variability.
Revised: Furthermore, to use the change of variable formula, the latent space of a reversible model must have the same dimensionality as the data space, which is unreasonable considering that real-world, high-dimensional data (e.g., images) tends to lie on low-dimensional manifolds, and thus results in redundant latent dimensions and variability.

Original: Intriguingly, recent research (Arjovsky et al., 2017; Dai & Wipf, 2019) suggests that the discrepancy between the intrinsic and ambient dimensionalities of data also contributes to the difﬁculties in training GANs and VAEs.
Revised: Intriguingly, recent research (Arjovsky et al., 2017; Dai & Wipf, 2019) suggests that the discrepancy between the intrinsic and ambient dimensionalities of data also contributes to the difﬁculties in training GANs and VAEs.

Original: In this work, we present a novel framework for training autoencoder-based generative models, with non-adversarial losses and unrestricted neural network architectures.
Revised: In this work, we present a novel framework for training autoencoder-based generative models, with non-adversarial losses and unrestricted neural network architectures.

Original: Given a standard autoencoder and a target data distribution, instead of matching the target distribution in the data space, we map both the generated and target distributions to the latent space using the encoder, and train the generator (or decoder) to minimize the divergence between the mapped distributions.
Revised: Given a standard autoencoder and a target data distribution, instead of matching the target distribution in the data space, we map both the generated and target distributions to the latent space using the encoder, and train the generator (or decoder) to minimize the divergence between the mapped distributions.

Original: We prove, under mild assumptions, that by minimizing a form of latent reconstruction error, matching the target distribution in the latent space implies matching it in the data space.
Revised: We prove, under mild assumptions, that by minimizing a form of latent reconstruction error, matching the target distribution in the latent space implies matching it in the data space.

Original: We call this framework perceptual generative autoencoder (PGA) .
Revised: We call this framework perceptual generative autoencoder (PGA) .

Original: We show that PGA enables training generative autoencoders with maximum likelihood, without restrictions on architectures or latent dimensionalities.
Revised: We show that PGA enables training generative autoencoders with maximum likelihood, without restrictions on architectures or latent dimensionalities.

Original: In addition, when combined with VAE, PGA can generate sharper samples than vanilla VAE.
Revised: In addition, when combined with VAE, PGA can generate sharper samples than vanilla VAE.

Original:  M ETHODS 2. P ERCEPTUAL G ENERATIVE M ODEL
Revised:  M ETHODS 2. P ERCEPTUAL G ENERATIVE M ODEL

Original: Let f : R D → R H be the encoder parameterized by φ , and g : R H
Revised: Let f : R D → R H be the encoder parameterized by φ , and g : R H

Original: → R D be the decoder parameterized by θ .
Revised: → R D be the decoder parameterized by θ .

Original: Our goal is to obtain a generative model, which maps a simple prior distribution to the data distribution, D .
Revised: Our goal is to obtain a generative model, which maps a simple prior distribution to the data distribution, D .

Original: Throughout this paper, we use N ( 0 , I ) as the prior distribution.
Revised: Throughout this paper, we use N ( 0 , I ) as the prior distribution.

Original: For z ∈ R H , the output of the decoder, g ( z ) , lies in a manifold that is at most H -dimensional.
Revised: For z ∈ R H , the output of the decoder, g ( z ) , lies in a manifold that is at most H -dimensional.

Original: Therefore, if we train the autoencoder to minimize
Revised: Therefore, if we train the autoencoder to minimize

Original: [Equation]
Revised: [Equation]

Original:  where ˆ x = g ( f ( x )) , then ˆ x can be seen as a projection of the input data, x , onto the manifold of g ( z ) .
Revised: 1) where ˆ x = g ( f ( x )) , then ˆ x can be seen as a projection of the input data, x , onto the manifold of g ( z ) .

Original: Let ˆ D denote the distribution of ˆ x .
Revised: Let ˆ D denote the distribution of ˆ x .

Original: Given enough capacity of the encoder, ˆ D is the best approximation to D (in terms of L 2 distance), that we can obtain from the decoder, and thus can serve as a surrogate target for training the generator.
Revised: Given enough capacity of the encoder, ˆ D is the best approximation to D (in terms of L 2 distance), that we can obtain from the decoder, and thus can serve as a surrogate target for training the generator.

Original: Due to the difﬁculty of directly training the generator to match ˆ D , we seek to map ˆ D to the latent space, and train the generator to match the mapped distribution, ˆ H , in the latent space.
Revised: Due to the difﬁculty of directly training the generator to match ˆ D , we seek to map ˆ D to the latent space, and train the generator to match the mapped distribution, ˆ H , in the latent space.

Original: To this end, we reuse the encoder for mapping ˆ D to ˆ H , and train the generator
Revised: To this end, we reuse the encoder for mapping ˆ D to ˆ H , and train the generator

Original: such that h ( · ) =
Revised: such that h ( · ) =

Original: f ( g ( · )) maps
Revised: f ( g ( · )) maps

Original: N ( 0 , I ) to ˆ H .
Revised: N ( 0 , I ) to ˆ H .

Original: In addition, to ensure that g maps N ( 0 , I ) to ˆ D , we minimize the following latent reconstruction loss with respect to (w.r.t.)
Revised: In addition, to ensure that g maps N ( 0 , I ) to ˆ D , we minimize the following latent reconstruction loss with respect to (w.r.t.)

Original: φ :
Revised: φ :

Original: [Equation]
Revised: [Equation]

Original: Formally, let Z ( x ) be the set of all z ’s that are mapped to the same x by the decoder, we have the following theorem:
Revised: Formally, let Z ( x ) be the set of all z ’s that are mapped to the same x by the decoder, we have the following theorem:

Original: Theorem 1.
Revised: Theorem 1.

Original: Assuming the convexity of Z ( x ) for all x ∈ R D , and sufﬁcient capacity of the encoder; for z ∼ N ( 0 , I ) , if Eq.
Revised: Assuming the convexity of Z ( x ) for all x ∈ R D , and sufﬁcient capacity of the encoder; for z ∼ N ( 0 , I ) , if Eq.

Original: (2) is minimized and h ( z ) ∼ ˆ H , then g ( z ) ∼ ˆ D .
Revised: (2) is minimized and h ( z ) ∼ ˆ H , then g ( z ) ∼ ˆ D .

Original: Proof.
Revised: Proof.

Original: We ﬁrst show that any different x ’s generated by the decoder are mapped to different z ’s by the encoder.
Revised: We ﬁrst show that any different x ’s generated by the decoder are mapped to different z ’s by the encoder.

Original: Let x 1 = g
Revised: Let x 1 = g

Original: ( z 1 ) , x 2 = g
Revised: ( z 1 ) , x 2 = g

Original: ( z 2 ) , and x 1 (cid:54) = x 2 .
Revised: ( z 2 ) , and x 1 (cid:54) = x 2 .

Original: Since the encoder has sufﬁcient capacity and Eq.
Revised: Since the encoder has sufﬁcient capacity and Eq.

Original: (2) is minimized, we have f ( x 1 ) =
Revised: (2) is minimized, we have f ( x 1 ) =

Original: E [ z 1 ] and f ( x 2 ) =
Revised: E [ z 1 ] and f ( x 2 ) =

Original: E [ z 2 ] .
Revised: E [ z 2 ] .

Original: By deﬁnition, z 1 ∈ Z ( x 1 ) , z 2 ∈ Z ( x 2 ) .
Revised: By deﬁnition, z 1 ∈ Z ( x 1 ) , z 2 ∈ Z ( x 2 ) .

Original: Therefore, given the convexity of Z ( x 1 ) and Z ( x 2 ) , f ( x 1 ) ∈ Z ( x 1 ) and f ( x 

Revised: Therefore, given the convexity of Z ( x 1 ) and Z ( x 2 ) , f ( x 1 ) ∈ Z ( x 1 ) and f ( x

Original: ) ∈
Revised: ) ∈

Original: Z ( x 2 ) .
Revised: Z ( x 2 ) .

Original: Since Z ( x 1 ) ∩ Z ( x 2 )
Revised: Since Z ( x 1 ) ∩ Z ( x 2 )

Original: =
Revised: =

Original: Ø , we have f ( x 1 ) (cid:54) =
Revised: Ø , we have f ( x 1 ) (cid:54) =

Original: f
Revised: f

Original: ( x 2 ) .
Revised: ( x 2 ) .

Original: For z ∼ N ( 0 , I ) , denote the distributions of g ( z ) and h ( z ) , respectively, by (cid:101) D and (cid:101) H .
Revised: For z ∼ N ( 0 , I ) , denote the distributions of g ( z ) and h ( z ) , respectively, by (cid:101) D and (cid:101) H .

Original: We then consider the case where (cid:101)
Revised: We then consider the case where (cid:101)

Original: D
Revised: D

Original: and ˆ D are discrete distributions.
Revised: and ˆ D are discrete distributions.

Original: If g ( z ) (cid:28) ˆ D , then there exists an x , such that p (cid:101) H ( f ( x ))
Revised: If g ( z ) (cid:28) ˆ D , then there exists an x , such that p (cid:101) H ( f ( x ))

Original: =
Revised: =

Original: p (cid:101) D ( x ) (cid:54) =
Revised: p (cid:101) D ( x ) (cid:54) =

Original: p ˆ D ( x ) =
Revised: p ˆ D ( x ) =

Original: p ˆ H ( f ( x )) , contradicting that h ( z ) ∼ ˆ H .
Revised: p ˆ H ( f ( x )) , contradicting that h ( z ) ∼ ˆ H .

Original: The result still holds when (cid:101) D and ˆ D approach continuous distributions.
Revised: The result still holds when (cid:101) D and ˆ D approach continuous distributions.

Original: Note that the two distributions compared in Theorem 1, (cid:101) D and ˆ D , are mapped respectively from
Revised: Note that the two distributions compared in Theorem 1, (cid:101) D and ˆ D , are mapped respectively from

Original: N ( 0 , I ) and H .
Revised: N ( 0 , I ) and H .

Original: While N ( 0 , I ) is supported on the whole R H , there can be z ’s with low probabilities in N ( 0 , I ) , but with high probabilities in H , which are not well covered by Eq.
Revised: While N ( 0 , I ) is supported on the whole R H , there can be z ’s with low probabilities in N ( 0 , I ) , but with high probabilities in H , which are not well covered by Eq.

Original: Therefore, it is sometimes helpful to minimize another latent reconstruction loss on H :
Revised: Therefore, it is sometimes helpful to minimize another latent reconstruction loss on H :

Original: [Equation]
Revised: [Equation]

Original: By Theorem 1, the problem of training the generative model reduces to training h to map N ( 0 , I ) to ˆ H , which we refer to as the perceptual generative model.
Revised: By Theorem 1, the problem of training the generative model reduces to training h to map N ( 0 , I ) to ˆ H , which we refer to as the perceptual generative model.

Original: In the subsequent subsections, we present a maximum likelihood approach, as well as a VAE-based approach, to train the perceptual generative model.
Revised: In the subsequent subsections, we present a maximum likelihood approach, as well as a VAE-based approach, to train the perceptual generative model.

Original: 2. A M AXIMUM L IKELIHOOD A PPROACH
Revised: 2. A M AXIMUM L IKELIHOOD A PPROACH

Original: We ﬁrst assume the invertibility of h .
Revised: We ﬁrst assume the invertibility of h .

Original: For ˆ x ∼ ˆ D , let ˆ H be the distribution of f (ˆ x ) .
Revised: For ˆ x ∼ ˆ D , let ˆ H be the distribution of f (ˆ x ) .

Original: We can train h directly with maximum likelihood using the change of variable formula as
Revised: We can train h directly with maximum likelihood using the change of variable formula as

Original: [Equation]
Revised: [Equation]

Original: Ideally, we would like to maximize Eq.
Revised: Ideally, we would like to maximize Eq.

Original: (4) w.r.t.
Revised: (4) w.r.t.

Original: the parameters of the generator (or decoder), θ .
Revised: the parameters of the generator (or decoder), θ .

Original: However, directly optimizing the ﬁrst term in Eq.
Revised: However, directly optimizing the ﬁrst term in Eq.

Original: (4) requires computing z
Revised: (4) requires computing z

Original: = h − 1 (ˆ z ) , which is usually unknown.
Revised: = h − 1 (ˆ z ) , which is usually unknown.

Original: Nevertheless, for ˆ z ∼ ˆ H , we have h − 1 (ˆ z ) =
Revised: Nevertheless, for ˆ z ∼ ˆ H , we have h − 1 (ˆ z ) =

Original: f ( x ) and x ∼ D , and thus we can minimize the following loss function w.r.t.
Revised: f ( x ) and x ∼ D , and thus we can minimize the following loss function w.r.t.

Original: φ instead:
Revised: φ instead:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: To avoid computing the Jacobian for the second term in Eq.
Revised: To avoid computing the Jacobian for the second term in Eq.

Original: (4), which is slow for unrestricted architectures, we approximate the Jacobian determinant and derive a loss function for the decoder as
Revised: (4), which is slow for unrestricted architectures, we approximate the Jacobian determinant and derive a loss function for the decoder as

Original: [Equation]
Revised: [Equation]

Original: (cid:19)(cid:12)(cid:12)(cid:12)(cid:12)(cid:21) , (6) where S ( (cid:15) ) is a uniform distribution on a small hypersphere of radius (cid:15) .
Revised: , (6) where S ( (cid:15) ) is a uniform distribution on a small hypersphere of radius (cid:15) .

Original: When (cid:15) → 0 , the approximation forms an upper bound on the right-hand side (r.h.s.) of Eq.
Revised: When (cid:15) → 0 , the approximation forms an upper bound on the right-hand side (r.h.s.) of Eq.

Original: (6), and becomes tight if h is close to the identity function.
Revised: (6), and becomes tight if h is close to the identity function.

Original: Intuitively, Eq.
Revised: Intuitively, Eq.

Original: (5) attracts the latent representations of data samples to the origin, while Eq.
Revised: (5) attracts the latent representations of data samples to the origin, while Eq.

Original: (6) expands the volume occupied by each sample in the latent space.
Revised: (6) expands the volume occupied by each sample in the latent space.

Original: The above discussion relies on the assumption that h is invertible, which is not necessarily true for unrestricted architectures.
Revised: The above discussion relies on the assumption that h is invertible, which is not necessarily true for unrestricted architectures.

Original: If h ( z ) is not invertible for some z , the logarithm of the Jacobian determinant at z becomes inﬁnite, in which case Eq.
Revised: If h ( z ) is not invertible for some z , the logarithm of the Jacobian determinant at z becomes inﬁnite, in which case Eq.

Original: (4) cannot be optimized.
Revised: (4) cannot be optimized.

Original: Nevertheless, since (cid:107) h ( z + δ ) − h ( z ) (cid:107) 22 is unlikely to be zero if the model is properly initialized, the approximation in
Revised: Nevertheless, since (cid:107) h ( z + δ ) − h ( z ) (cid:107) 22 is unlikely to be zero if the model is properly initialized, the approximation in

Original: Eq.
Revised: Eq.

Original: (6) remains ﬁnite, and thus can be optimized regardless.
Revised: (6) remains ﬁnite, and thus can be optimized regardless.

Original: To summarize, we train the autoencoder to obtain a generative model by minimizing the following loss function:
Revised: To summarize, we train the autoencoder to obtain a generative model by minimizing the following loss function:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where α , β , and γ are hyperparameters to be tuned.
Revised: where α , β , and γ are hyperparameters to be tuned.

Original: We refer to this approach as maximum likelihood
Revised: We refer to this approach as maximum likelihood

Original: PGA (LPGA).
Revised: PGA (LPGA).

Original: 2. A VAE- BASED A PPROACH
Revised: 2. A VAE- BASED A PPROACH

Original: The original VAE is trained by maximizing the evidence lower bound on log p ( x ) as
Revised: The original VAE is trained by maximizing the evidence lower bound on log p ( x ) as

Original: [Equation]
Revised: [Equation]

Original: 8) where p ( x | z ) is modeled with the decoder, and q ( z | x ) is modeled with the encoder.
Revised: 8) where p ( x | z ) is modeled with the decoder, and q ( z | x ) is modeled with the encoder.

Original: In our case, we would like to modify Eq.
Revised: In our case, we would like to modify Eq.

Original: (8) in a way that helps maximize log p (ˆ z ) .
Revised: (8) in a way that helps maximize log p (ˆ z ) .

Original: Therefore, we replace p ( x | z ) on the r.h.s.
Revised: Therefore, we replace p ( x | z ) on the r.h.s.

Original: of Eq.
Revised: of Eq.

Original: (8) with p (ˆ z | z ) , and derive a lower bound on log p (ˆ z ) as
Revised: (8) with p (ˆ z | z ) , and derive a lower bound on log p (ˆ z ) as

Original: [Equation]
Revised: [Equation]

Original: Similar to the original VAE, we make the assumption that q ( z | x ) and p (ˆ z | z ) are Gaussian; i.e., q ( z | x ) =
Revised: Similar to the original VAE, we make the assumption that q ( z | x ) and p (ˆ z | z ) are Gaussian; i.e., q ( z | x ) =

Original: N (cid:16) z (cid:12)(cid:12)(cid:12) µ φ ( x ) ,  σ 2 φ ( x ) (cid:17) , and p (ˆ z | z ) =N (cid:0)
Revised: N (cid:16) z (cid:12)(cid:12)(cid:12) µ φ ( x ) , diag (cid:16) σ 2 φ ( x ) (cid:17)(cid:17) , and p (ˆ z | z ) =

Original:  ˆ z (cid:12)(cid:12) µ θ,φ ( z ) , σ 2 I (cid:1) .
Revised: N (cid:0) ˆ z (cid:12)(cid:12) µ θ,φ ( z ) , σ 2 I (cid:1) .

Original: Here, µ φ ( · ) = f ( · ) , µ θ,φ ( · ) = h ( · ) , and σ > 0 is a tunable scalar.
Revised: Here, µ φ ( · ) = f ( · ) , µ θ,φ ( · ) = h ( · ) , and σ > 0 is a tunable scalar.

Original: The VAE variant is trained by minimizing
Revised: The VAE variant is trained by minimizing

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that we slightly abuse the notation, since z is deterministic for the losses in Eqs.
Revised: Note that we slightly abuse the notation, since z is deterministic for the losses in Eqs.

Original: (2) and (3), but is stochastic (with additive Gaussian noise) in Eq.
Revised: (2) and (3), but is stochastic (with additive Gaussian noise) in Eq.

Original: Accordingly, the overall loss function is given by
Revised: Accordingly, the overall loss function is given by

Original: [Equation]
Revised: [Equation]

Original: We refer to this approach as variational PGA (VPGA).
Revised: We refer to this approach as variational PGA (VPGA).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate the performance of LPGA and VPGA on three image datasets, MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky & Hinton, 2009), and CelebA (Liu et al., 2015).
Revised: In this section, we evaluate the performance of LPGA and VPGA on three image datasets, MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky & Hinton, 2009), and CelebA (Liu et al., 2015).

Original: For
Revised: For

Original: CelebA, we employ the discriminator and generator architecture of DCGAN (Radford et al., 2016) for the encoder and decoder of PGA.
Revised: CelebA, we employ the discriminator and generator architecture of DCGAN (Radford et al., 2016) for the encoder and decoder of PGA.

Original: For comparison, the same architecture is used in the recent literature on WAEs (Tolstikhin et al., 2018; Kolouri et al., 2019).

Original: We half the number of ﬁlters (i.e., 64 ﬁlters for the ﬁrst convolutional layer) for faster experiments, while more ﬁlters are observed to improve performance.
Revised: We half the number of ﬁlters (i.e., 64 ﬁlters for the ﬁrst convolutional layer) for faster experiments, while more ﬁlters are observed to improve performance.

Original: Due to smaller input sizes, we reduce the number of convolutional layers accordingly for MNIST and CIFAR-10, and add a fully-connected layer of 1024 units for MNIST, as done in (Chen et al., 2016).
Revised: Due to smaller input sizes, we reduce the number of convolutional layers accordingly for MNIST and CIFAR-10, and add a fully-connected layer of 1024 units for MNIST, as done in (Chen et al., 2016).

Original: SGD with a momentum of 0 .
Revised: SGD with a momentum of 0 .

Original: 9 is used to train all models.
Revised: 9 is used to train all models.

Original: The training process of PGA is stable in general, given the non-adversarial losses.
Revised: The training process of PGA is stable in general, given the non-adversarial losses.

Original: However, stability issues can occur when batch normalization (Ioffe & Szegedy, 2015) is introduced, since both the encoder and decoder are fed with multiple batches drawn from different distributions.
Revised: However, stability issues can occur when batch normalization (Ioffe & Szegedy, 2015) is introduced, since both the encoder and decoder are fed with multiple batches drawn from different distributions.

Original: In our experiments, we only use batch normalization when it does not cause stability issues, in which case it is observed to substantially accelerate convergence.
Revised: In our experiments, we only use batch normalization when it does not cause stability issues, in which case it is observed to substantially accelerate convergence.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1,for both LPGA and VPGA, the visual quality of the generated samples are competitive with that of GANs.
Revised: 1, the visual quality of the PGA-generated samples is signiﬁcantly improved over that of VAE.

Original: Moreover, we observe that VPGA tends to generate slightly blurred samples compare to LPGA; it is, nevertheless, signiﬁcantly improved over VAE.
Revised: In particular, VPGA generates much sharper samples on CIFAR-10 and CelebA compared to vanilla VAE.

Original: For CelebA, we further showlinear interpolations in the latent space  in Fig.
Revised: For CelebA, we further show latent space interpolations in Fig.

Original: In addition, we use the Fr ´ echet Inception Distance (FID) (Heusel et al., 2017) to evaluate LPGA, VPGA, and VAE.
Revised: In addition, we use the Fr ´ echet Inception Distance (FID) (Heusel et al., 2017) to evaluate LPGA, VPGA, and VAE.

Original: For each model and each dataset, we take 5,000 generated samples to compute the FID score.
Revised: For each model and each dataset, we take 5,000 generated samples to compute the FID score.

Original: The results are summarized in Table.
Revised: The results are summarized in Table.

Revised: Compared to other non-adversarial generative models (Tolstikhin et al., 2018; Kolouri et al., 2019; Ghosh et al., 2019), where similar but larger architectures are used, we obtain substantially better FID scores on CIFAR-10 and CelebA.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a framework, PGA, for training autoencoder-based generative models, with nonadversarial losses and unrestricted neural network architectures.
Revised: We proposed a framework, PGA, for training autoencoder-based generative models, with nonadversarial losses and unrestricted neural network architectures.

Original: By matching target distributions in the latent space, PGA trained with maximum likelihood generalizes the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities.
Revised: By matching target distributions in the latent space, PGA trained with maximum likelihood generalizes the idea of reversible generative models to unrestricted neural network architectures and arbitrary latent dimensionalities.

Original: In addition, it improves the performance of VAE when combined together.
Revised: In addition, it improves the performance of VAE when combined together.

Original: In principle, PGA can be combined with any method that can train the perceptual generative model.
Revised: In principle, PGA can be combined with any method that can train the perceptual generative model.

Original: While we have only considered two non-adversarial approaches, an interesting future work would be to combine PGA with an adversarial discriminator trained on latent representations.
Revised: While we have only considered two non-adversarial approaches, an interesting future work would be to combine PGA with an adversarial discriminator trained on latent representations.

Original: Moreover, the compatibility issue with batch normalization deserves further investigation
Revised: Moreover, the compatibility issue with batch normalization deserves further investigation.

Original: [Figure]
Revised: [Figure]