G ENERATIVE L ATENT F LOW
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work we propose Generative Latent Flow (GLF), an algorithm for generative modeling of data distributions.
GLF uses an auto-encoder to learn latent representations of the data, and a normalizing ﬂow to map the distribution of the latent variables to that of a standard Gaussian. GLF can be seen as a variational auto-encoder, with normalizing ﬂow prior, and a vanishing limit of the pixel-wise variance of the data.
We carefully study this relationship and the pros and cons of using an auto-encoder vs. a variational auto-encoder. In contrast to a number of other auto-encoder based generative models, which use various regularizers to encourage the encoded latent distribution to match the prior distribution, our model explicitly constructs a mapping between these two distributions, leading to better density matching while avoiding over regularizing the latent variables.
We compare our model with several related techniques, and show that under standard quantitative evaluations, our method achieves state-of-the-art sample quality and diversity among AE based models on commonly used datasets and is competitive with GANs’ benchmarks.
I NTRODUCTION
Generative models have attracted much attention in the literature on deep learning.
These models are used to formulate the distribution of complex data as a function of random noise passed through a network, so that rendering samples from the distribution is particularly easy.
The most dominant generative models are Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), as they have exhibited impressive performance in generating high quality images (Radford et al., 2015; Brock et al., 2018) and in other vision tasks (Zhu et al., 2017; Ledig et al., 2017).
Despite their success, training GANs can be challenging, partly because they are trained by solving a saddle point optimization problem formulated as an adversarial game.
It is well known that training GANs is unstable and sensitive to hyper-parameter settings (Salimans et al., 2016; Arora et al., 2017), and sometimes training leads to mode collapse (Goodfellow, 2016).
Although there have been multiple efforts to overcome the difﬁculties in training GANs (Arjovsky et al., 2017; Metz Luke & SohlDickstein, 2017; Srivastava et al., 2017; Miyato et al., 2018), researchers are also actively studying non-adversarial methods that are known to be less affected by these issues.
Some models explicitly deﬁne p ( x ) , the distribution of the data, and training is guided by maximizing the data likelihood.
One approach is to express the data distribution in an auto-regressive pattern (Papamakarios et al., 2017; Oord et al., 2016); another is to express it as an invertible transformation of a simple distribution using the change of variable formula, where the invertible transformation is deﬁned using a normalizing ﬂow network (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018).
While being mathematically clear and well deﬁned, normalizing ﬂows keep the dimensionality of the original data in order to maintain bijectivity. Consequently, they cannot provide low-dimensional representations of the data and training is computationally expensive.
Considering the prohibitively long training time and advanced hardware requirements in training large scale ﬂow models such as (Kingma & Dhariwal, 2018), we believe that it is worth exploring the application of ﬂows in the low dimensional representation spaces rather than for the original data.
Another class of generative models employs an encoder-decoder structure and low dimensional latent variables to represent and generate the data. An encoder is used to produce estimates of the latent variables corresponding to a particular data point, and samples from a predeﬁned prior distribution on the latent space are passed through a decoder to produce new samples from the data distribution.
We call these auto-encoder (AE) based models, of which variational auto-encoders (VAEs) are perhaps the most inﬂuential (Kingma & Welling, 2013; Rezende et al., 2014).
VAEs use the encoder to produce approximations to the posterior distribution of the latent variable given the data, and the training objective is to maximize a variational lower bound of the data log likelihood.
VAEs are easy to train, but their generation quality still lies far below that of GANs, as they tend to generate blurry images (Dosovitskiy & Brox, 2016). Whereas the original VAE uses a standard Gaussian prior, it can be extended by introducing a learnable parameterized prior distribution.
There have been a number of studies in this direction (see section 2), some of which use a normalizing ﬂow parameterization, where the prior is modeled as a trainable continuous bijective transformation of the standard Gaussian. We carefully study this method, and make the surprising novel observation that in order to produce high quality samples, it is necessary to signiﬁcantly increase the weight on the reconstruction loss. This corresponds to decreasing the variance of the observational noise of the generative model at each pixel, where we are assuming the data distribution is factorial Gaussian conditioned on the output of the decoder, which yields the MSE as the reconstruction loss.
It is important to note that increasing this weight alone without access to a trainable prior does not consistently improve generation quality.
We show that as this weight increases, we approach a vanishing noise limit that corresponds to a deterministic auto-encoder.
This leads to a new algorithm we call Generative Latent Flow (GLF), which combines a deterministic auto-encoder that learns a mapping to and from a latent space, and a normalizing ﬂow that matches the standard Gaussian to the distribution of latent variables of the training data produced by the encoder.
Our contributions are summarized as follows: i) we carefully study the effects of equipping VAEs with a normalizing ﬂow prior on image generation quality as the weight of the reconstruction loss is increased.
ii)
Based on this ﬁnding, introduce Generative Latent Flow, which uses auto-encoders instead of VAEs.
iii)
Through standard evaluations, we show that our proposed model achieves state-of-the-art sample quality among competing AE based models, and has the additional advantage of faster convergence.  R ELATED W ORK
In general, in order for an AE based model with encoder-decoder structure to generate samples resembling the training data distribution, two criteria need to be ensured: (a) the decoder is able to produce a good reconstruction of a training image given its encoded latent variable z ; and (b) the empirical latent distribution q ( z ) of z ’s returned by the encoder is close to the prior p ( z ) .
In
VAEs, the empirical latent distribution is often called aggregated or marginal posterior: q ( z ) =
E x ∼ p data [ q ( z | x )] .
While (a) is mainly driven by the reconstruction loss, satisfying criterion (b) is more complicated.
Intuitively, criterion (b) can possibly be achieved by designing mechanisms that either modify the empirical latent distribution q ( z ) , or conversely modify the prior p ( z ) .
There is plenty of previous work in both directions. Modifying the empirical latent distribution q ( z ) :
In the classic VAE model, D KL ( q ( z | x ) (cid:107) p ( z )) in the ELBO loss can be decomposed as D KL ( q ( z ) (cid:107) p ( z )) plus a mutual information term as shown in (Hoffman & Johnson, 2016).
Therefore, VAEs modify q ( z ) indirectly through regularizing the posterior distribution q ( z | x ) .
Several modiﬁcations to VAE’s loss (Chen et al., 2018; Kim & Mnih, 2018), which are designed for the task of unsupervised disentanglement, put a stronger penalty speciﬁcally on the mismatch between q ( z ) and p ( z ) .
There are also attempts to incorporate normalizing ﬂows into the encoder to provide more ﬂexible approximate posteriors (Rezende & Mohamed, 2015; Kingma et al., 2016; Berg et al., 2018).
However, empirical evaluation shows that VAEs with ﬂow posteriors do not reduce the mismatch between q ( z ) and p ( z ) (Rosca et al., 2018).
Furthermore, as of yet, all these modiﬁcations to VAEs have not been shown to improve generation quality.
Adversarial auto-encoders (AAEs) (Makhzani et al., 2015) and Wasserstein auto-encoders (WAEs) (Tolstikhin et al., 2017) use an adversarial regularizer or MMD regularizer (Gretton et al., 2012) to force q ( z ) to be close to p ( z ) .
WAEs are shown to improve generation quality, as they generate sharper images than VAEs do.
Modifying the prior distribution p ( z ) : An alternative to modifying the approximate posterior is using a trainable prior.
(Tomczak & Welling, 2017; Klushyn et al., 2019; Bauer & Mnih, 2018) propose different ways to approximate q ( z ) using a sampled mixture of posteriors during training, and then use the approximated q ( z ) as the prior in the VAE.
This is a natural way to let the prior match q ( z ) , however, these methods have not been shown to improve generation quality.
Two-stage VAE (Dai & Wipf, 2019) introduces another VAE on the latent space deﬁned by the ﬁrst VAE to learn the distribution of its latent variables.
VQ-VAE (Oord et al., 2017)
ﬁrst trains an auto-encoder with discrete latent variables, and then ﬁts an auto-regressive prior on the latent space.
GLANN (Hoshen et al., 2019) learns a latent representation by GLO (Bojanowski et al., 2017) and matches the densities of the latent variables with an implicit maximum likelihood estimator (Li & Malik, 2018).
RAE+GMM (Ghosh et al., 2019) trains a regularized auto-encoder (Alain & Bengio, 2014) and ﬁts a mixture of Gaussian distribution on the latent space.
Note that all these methods involve two stage-training, which means that the prior distribution is ﬁtted after training the variational or deterministic auto-encoder.
They have been shown to improve the quality of the generated images. VAEs with a normalizing ﬂow as a learnable prior (Chen et al., 2016b; Huang et al., 2017) also fall into this category. Since these are the main focus of this paper, we discuss them in detail in Section 3.2.
We note that modiﬁcations of VAEs with a normalizing ﬂow posterior have been extensively studied.
In contrast, VAEs with ﬂow prior have attracted much less attention. (Huang et al., 2017) brieﬂy discusses this model to solve the distribution mismatch in the latent space, and recently (Xu et al., 2019) shows the advantages of learning a ﬂow prior over learning a ﬂow posterior. However, these papers only focus on improvements of the data likelihood.
Here we study the model from the perspective of the effects of the normalizing ﬂow prior on sample generation quality, leading to some important and novel observations.
[Equation]
Encoder z
Decoder ˆ x
Recon
Loss
Flow (cid:15)
Likelihood loss (a) yy
C yy C yy
P y (b)
C OMBINING N ORMALIZING F LOW WITH AE B ASED M ODELS
In this section, we discuss the combination of normalizing ﬂow priors with AE based models in detail.
We ﬁrst review normalizing ﬂows in section 3.1, then in section 3.2 we introduce VAEs with normalizing ﬂow prior and present some novel observations with respect to this model.
Finally in section 3.3 we propose Generative Latent Flow (GLF) to further simplify the model and improve performance.
3. R EVIEW : N ORMALIZING F LOWS
Normalizing ﬂows are carefully-designed invertible networks that map the training data to a simple distribution.
Let z ∈
Z be an observation from an unknown target distribution z ∼ p ( z ) and p (cid:15) be the unit Gaussian prior distribution on E .
Given a bijection f θ : Z → E , we deﬁne a probability model p θ ( z ) with parameters θ on Z . The negative log likelihood (NLL) of z is computed by the change of variable formula:
[Equation]
where ∂f θ ( z ) ∂z is the Jacobian matrix of f θ .
In order to learn the ﬂow f θ , the NLL objective of z is minimized, which is equivalent to maximizing the likelihood of z .
Since the mapping is a bijection, sampling from the trained model p θ ( z ) is trivial: simply sample (cid:15) ∼ p (cid:15) and compute z = f − 1 θ ( (cid:15) ) .
The key to designing a normalizing ﬂow model is deﬁning the transformation f θ so that the inverse transformation and the determinant of the Jacobian matrix can be efﬁciently computed.
Based on (Dinh et al., 2016), we adopt the following layers to form the ﬂows used in our model.
Afﬁne coupling layer:
Given D dimensional input data z and d < D , we partition the input into two vectors z 1 =
z 1: d and z 2 =
z d +1: D .
The output of one afﬁne coupling layer is given by y 1 =
z 1 , y 2 =
z 2 (cid:12) exp( s ( z 1 ))
+ t ( z 1 ) where s and t are functions from R d → R D − d and (cid:12) is the element-wise product.
The inverse of the transformation is explicitly given by z 1 =
y 1 , z 2 = ( y 2 − t ( y 1 ))
(cid:12) exp( − s ( y 1 )) .
The determinant of the Jacobian matrix of this transformation is det ∂ y ∂
z = (cid:81) dj=
(exp[s(z 1 ) j ]) .
Since computing both the inverse and the Jacobian of an afﬁne coupling layer does not require computing the inverse and Jacobian of s and t , both functions can be arbitrarily complex.
Combining coupling layers with random permutation: Afﬁne coupling layers leave some components of the input data unchanged.
In order to transform all the components, two coupling layers are combined in an alternating pattern to form a coupling block, so that the unchanged components in the ﬁrst layer can be transformed in the second layer.
In particular, we add a ﬁxed random permutation of the coordinates of the input data at the end of each coupling block.
See Figure 1b for an illustration of a coupling block used in our model.
3. VAE S WITH N ORMALIZING F LOW P RIOR
We begin by introducing the training loss of the model. Consider the ELBO loss of standard VAEs with Gaussian prior and posterior ( η, φ denote the parameters of encoder and decoder, respectively):
[Equation]
The ﬁrst term is related to the reconstruction loss, while the last two terms can be combined as
D KL (
[Equation]
( z | x ) (cid:107) p ( z )) . β > 0 is a hyper-parameter that controls the relative weight of the reconstruction loss and the KL divergence loss.
In the standard formulation of VAEs, p φ assumes an independent Gaussian distribution with variance 1 at each pixel.
The parameter β allows us to adjust this variance as 1 /β .
If we introduce a normalizing ﬂow f θ for the prior distribution, then the prior p θ becomes p θ ( z ) =
p
[Equation]
, where p (cid:15) is the standard Gaussian density.
Substituting this prior into equation 2, we obtain ELBO ( η, φ, θ ) for VAEs with ﬂow prior:
[Equation]
The second and third terms together are the log-likelihood of z under the prior distribution modeled by the ﬂow (corresponding to the negative of equation 1). The last term corresponds to the entropy of the posterior distribution returned by the encoder. Both the VAE and the normalizing ﬂow are trained by minimizing − ELBO ( η, φ, θ ) .
Previous work on VAEs with a ﬂow prior did not consider tuning β (which means the reconstruction loss and the KL loss are weighted equally) as they focused on comparing the obtained log likelihoods with those from plain VAEs.
However, we observe that when β = 1 , VAEs with a ﬂow prior do not signiﬁcantly improve the generation quality (see section 4.2 and Table 1).
The reason might be that although p ( z ) is matched with q ( z ) due to the ﬂow transformation, the decoder is not good enough to reconstruct sharp images (i.e, criterion (a) is not ensured).
In contrast, we ﬁnd that increasing β in the objective produces samples with signiﬁcantly higher quality (see Figure 3).
Intuitively, larger weight on the reconstruction loss forces the decoder to produce sharper reconstructed images, while the normalizing ﬂow prior is ﬂexible enough to match the latent distribution.
To the best of our knowledge, we are the ﬁrst to observe such a relation between the weight of the reconstruction loss and the generation quality of VAEs with ﬂow prior.
As β increases, two things occur as demonstrated empirically in Section 4.2.1.
First the estimated variances from the encoder decrease, and second the generation quality consistently improves. In the limit, as the posterior variance goes to zero, we obtain a deterministic encoder, leading to a deterministic auto-encoder and a normalizing ﬂow that is used to match the distribution of the latent variables obtained from the data.
This is described in detail in the next section. 3. G ENERATIVE L ATENT F LOW (GLF) In an auto-encoder, z = E η ( x ) is deterministic so that q ( z | x ) in equation 3 becomes a delta distribution and the entropy term in equation 3 can be removed. The overall training loss is then
[Equation] where L recon (cid:0) x i , G φ ( E η ( x i )) (cid:1) corresponds to − log p φ ( x i | z i ) and L NLL (cid:0) f θ ( E η ( x i )) to − log p θ ( z ) in the negative of equation 3. As noted in section 3.2, larger β ’s yield better results, in which case the parameters of the auto-encoder are affected almost exclusively by L recon , while L NLL only affects the parameters θ of the normalizing ﬂow.
Therefore, optimizing (4) with extremely large β is approximately equivalent to optimizing [Equation] where sg [ · ] is the stop gradient operation. The weight parameter β is no longer needed because the two loss terms affect independent sets of parameters.
We name the model trained by equationas Generative Latent Flow (GLF), to highlight that our model applies normalizing ﬂows on latent variables. See Figure 1a for an illustration of the GLF model. We call the model trained by equation 4, without stopped gradient, regularized GLF , since the ﬂow acts as a regularizer on the encoder. Note that when stopping the gradients, GLF can also be trained in two stages, namely an auto-encoder is trained ﬁrst, and then the ﬂow is trained to map the distribution of estimated latent variables to the standard Gaussian.
Empirically, we ﬁnd that the two-stage training strategy leads to similar performance, so we only focus on one-stage training as it follows our derivation more naturally. 3.3. N ECESSITY OF S TOPPING T HE G RADIENTS
The stop gradient operation is necessary when using deterministic auto-encoders. In VAEs with ﬂow prior, the entropy term, which encourages the posterior to have large variance, prevents the degeneracy of the z ’s. However, when using a deterministic encoder, if we let gradients of L NLL back propagate into the latent variables, training can lead to degenerate z ’s produced by the encoder E η . This is because f θ has to transform the z ’s to unit Gaussian noise, so the smaller the scale of the z ’s, the larger the magnitude of the log-determinant of the Jacobian. Since there is no constraint on the scale of the output of E η , the Jacobian term can dominate the entire objective.
While the latent variables cannot become exactly 0 because of the presence of the reconstruction loss, the extremely small scale of z may cause numerical issues that lead to severe ﬂuctuations. In summary, we stop the gradient of L NLL at the latent variables, preventing it from modifying the values of z and affecting the parameters of the encoder. We demonstrate the issues with regularized GLF in Section 4.2.1.
E XPERIMENTS
To demonstrate the performance of our method, we present both quantitative and qualitative evaluations on four commonly used datasets for generative models: MNIST (Lecun, 2010), Fashion
MNIST (Xiao et al., 2017), CIFAR-10 (Krizhevsky et al., 2009) and CelebA (Liu et al., 2015).
Throughout the experiments, we use 20-dimensional latent variables for MNIST and Fashion MNIST, and 64-dimensional latent variables for CIFAR-10 and CelebA.
(Lucic et al., 2018) adopted a common network architecture based on InfoGAN (Chen et al., 2016a) to evaluate GANs.
In order to make fair comparisons without designing arbitrarily large networks to achieve better performance, we use the generator architecture of InfoGAN as our decoder’s architecture, and the encoder is set symmetric to the decoder.
For details of the AE network structures, see Appendix A. For the ﬂow applied on the latent variables, we use 4 afﬁne coupling blocks as shown in Figure 1b, where each block contains 3 fully connected layers each with k hidden units.
For MNIST and Fashion MNIST, k = 64 , while for CIFAR-10 and CelebA, k = 256 .
Note that the ﬂow only adds a small parameter overhead on the auto-encoder (less than 3% ).
4. M ETRICS
Estimated test data log likelihood is a popular metric to evaluate models based on VAEs.
It is not trivial to estimate the log likelihood obtained from GLF, as it uses an deterministic auto-encoder.
More importantly, as shown in (Grover et al., 2018; Theis et al., 2015), likelihood is not well correlated with sample quality.
We use the Fr´echet Inception Distance (FID) (Heusel et al., 2017) as a metric for image generation quality.
FID is computed by ﬁrst extracting features of a set of real images x and a set of generated images g from an intermediate layer of the Inception network (Szegedy et al., 2015).
Each set of features is ﬁtted with a Gaussian distribution, yielding means µ x , µ g and co-variances matrices Σ x , Σ g .
The FID score is deﬁned to be the Fr´echet distance between these two Gaussians:
[Equation]
It is claimed that the FID score is sensitive to mode collapse and correlates well with human perception of generator quality (Lucic et al., 2018).
Recently, (Sajjadi et al., 2018) proposed using Precision and
Recall for Distributions (PRD) which can assess both the quality and diversity of generated samples.
We also include PRD in our studies.
[Equation]
b
[Equation]
Fashion MNIST
[-VAE+ﬂow posterior 26 . 7 ± 
 1 ± 0 .143 . 6 ± 0 .67 .-][Equation]
c) CIFAR
[-6 ± 0 .13 . 0 ± 
 5 ± 0 .46 . 3 ± 0 . GLF+perceptual loss (ours) 5.8 ± 0.-][Figure]
4. R ESULTS
Table 1 summarizes the main results of this work.
We compare the FID scores obtained by GLF with the scores of the VAE baseline and several existing AE based models that are claimed to produce high quality samples.
Instead of directly citing their reported results, we re-ran the experiments because we want to evaluate them under standardized settings so that all models adopt the same AE architectures, latent dimensions and image pre-processing.
We report the results of VAE+ﬂow prior/posterior with [Table] β = 1 .
For other methods, we largely follow their proposed experimental settings.
Details of each experiment are presented in Appendix B.
Note that the authors of WAE propose two variants, namely WAE-GAN and WAE-MMD.
We only report the results of WAE-GAN, as we found it consistently outperforms WAE-MMD.
Note also that,
GLANN (Hoshen et al., 2019) obtains impressive FID scores, but it uses perceptual loss (Johnson et al., 2016) as the reconstruction loss, while other models use MSE loss.
The perceptual loss is obtained by feeding both training images and reconstructed images into a pre-trained network such as VGG (Simonyan & Zisserman, 2014), and computing the L 1 distance between some of the intermediate layers’ activation.
We also train our method with perceptual loss and compare with GLANN in the last two rows of Table 1.
As shown in Table 1, our method obtains signiﬁcantly lower FID scores than competing AE based models across all four datasets.
In particular, GLF greatly outperforms VAE+ﬂow prior with the default setting of β = 1 .
A more detailed analysis and comparison between the two methods will be done in Section 4.2.1.
We also conﬁrm that VAE+ﬂow posterior cannot improve generation quality.
Perhaps the competing model with the closest performances to ours is RAE+GMM, which shares some similarity with GLF in that both methods ﬁt the density of the latent variables of an AE explicitly.
To compare our method with GANs, we also include the results from (Lucic et al., 2018) in Appendix D. In (Lucic et al., 2018), the authors conduct standardized and comprehensive evaluations of representative GAN models with large-scale hyper-parameter searches, and therefore, their results can serve as a strong baseline.
The results indicate that our method’s generation quality is competitive with that of carefully tuned GANs.
In Table 3, Appendix C, we present the Precision and Recall scores of our method and several competing methods.
As shown in the table, GLF obtains state-of-the-art Precision and Recall scores across all datasets, indicating that our method outperforms competing methods in terms of both sample quality and diversity.
Some qualitative results are shown in Figure 2.
Besides samples of the datasets used for quantitative evaluation, samples of CelebA-HQ (Karras et al., 2017) with the larger size of 256 × 256 are also included in Figure 2e to show our method’s ability to scale up to images with higher resolution.
Qualitative results show that our model can generate sharp and diverse samples in each dataset.
In Figure 2f, we show CelebA images generated by linearly interpolating two sampled random noise vectors.
The smooth and natural transition shows that our model can generate samples that have not been seen during training.
To provide further evidence that our model does not overﬁt or ‘memorize’ the training set, we show nearest neighbors in the training set for some generated samples in Appendix
G. For more qualitative results, including samples from models trained with perceptual loss, see
Appendix H. We observe that samples from models trained with perceptual loss have higher quality.
4.2. C OMPARISONS : GLF VS . R EGULARIZED GLF AND VAE+ FLOW P RIOR .
As discussed in section 1 and section 3.2, we underline the novel ﬁnding regarding the relation between the weight on the reconstruction loss and the sample quality of VAEs with ﬂow prior.
In this section, we present detailed experiments on this relation. We train VAEs+ﬂow prior on CIFAR-10 for different choices of β , plus one with a learnable β
(Dai & Wipf, 2019).
We record the progression of FID scores of these models in Figure 3a.
In Figure 3b, we plot the entropy term, which is the last term in equation 3, the objective of VAE+ﬂow prior.
The entropy is expressed as − (cid:80) dj
=1 log( σ j ) / 2 , where σ j is the standard deviation of the approximate posterior on the j th latent variable.
Higher entropy means that the latent variables have lower variances.
In Figure 3c, we plot the NLL loss.
We omit the results for β = 1 because the obtained FID scores are too high to ﬁt the scale of the plot.
Settings for the experiments in this subsection can be found in Appendix B.6.
[Figure]
From Figure 3a, we clearly observe the trend that the generation quality measured by FID scores improves as β increases. We also observe that as β increases, the performance gap between VAE+ﬂow prior and GLF closes, indicating that GLF captures the limiting behavior of VAE+ﬂow prior.
We also ﬁnd that learnable β is not effective, probably due to the relatively small values of β at the early stages of training.
When β is large, as indicated by Figure 3b, the posterior variances of VAEs become very small, so that effectively we are training an AE.
For example, as shown in Figure 3b, when β = 400 , the corresponding average posterior variance is around 10 − 4 .
This motivates us to use a deterministic auto-encoder in GLF, which as we have said above can be seen as the vanishing observational variance limit of VAE+ﬂow prior. It is important to note that the relation between β and generation quality only exists for VAEs with a trainable prior (such as normalizing ﬂow), as we verify empirically that increasing β on plain VAEs leads to worse FID scores. As discussed in section 3.3.1, training regularized GLF is unstable because of the degeneracy of the latent variables driven by the NLL loss. We empirically study the effect of latent regularization as a function of β and present results in Figure 4.
For low values of β = 1 and 10 , the NLL loss completely dominates the learning signal and the reconstruction loss quickly diverges, therefore we omit them in the plot.
For larger values of β = 50 , 100 , 400 we observe that the NLL loss decreases to a negative value of very large magnitude, and although overall performance is reasonable, it oscillates quite strongly as training proceeds.
In contrast, for GLF, where the ﬂow does not modify z , the NLL loss does not degenerate, resulting in stable improvements of FID scores as training progress.
In contrast to regularized GLF, which uses a deterministic encoder, no degeneracy in the latent variables is observed for VAE+ﬂow prior, thanks to the noise introduced in the stochastic encoder and the corresponding entropy term.
Indeed, Figure 3c shows that the training of VAE+ﬂow prior does not over-ﬁt the NLL loss, as opposed to regularized GLF where severe over-ﬁtting to NLL loss occurs as shown in Figure 4c.
Comparing Figure 3a and 4a, we observe that unlike regularized GLF, VAE+ﬂow prior does not suffer from divergence or ﬂuctuations in FID scores, even with relatively small β .
In summary, the results of FID scores show that regularized GLF is unstable, while as β increases, the performance of VAE+ﬂow prior converges to that of GLF. Note that although GLF only slightly outperforms VAE+ﬂow prior even when β is very large, it has the advantage that there is no need to tune β . [Figure]
4. T RAINING TIME
Besides better performance, our method also has the advantage of faster convergence among competing methods such as GLANN and Two-stage VAE.
In Table 5, Appendix F, we compare the number of training epochs to obtain the FID scores in Table 1.
We also compare the per epoch training clock time in Table 6,
Appendix F.
The combined results indicate that GLF requires much less training time while generating samples with higher quality.
C ONCLUSION
In this paper, we introduce Generative Latent Flow, a novel generative model which uses an autoencoder to learn a latent space from training data and a normalizing ﬂow to match the distribution of the latent variables with the prior.
Under standardized evaluations, our model achieves state-of-the-art results in image generation quality and diversity among several recently proposed auto-encoder based models.
While we are not claiming that our GLF model is superior to GANs, we do believe that it opens the door to realizing the potential of AE based models to produce high quality samples just as GANs do.
Our proposed model is motivated by our novel ﬁnding on the relation between large reconstruction weight and generation quality of VAEs with normalizing ﬂow prior.
The ﬁnding itself is important, as it can potentially motivate future work to study the trade-off between reconstruction and density matching in the objective of VAEs with learnable priors.
A N ETWORK A RCHITECTURES
In this section we provide Table 2 that summarizes the auto-encoder network structure.
The network structure is adopted from InfoGAN(Chen et al., 2016a), and the difference between the networks we used for each dataset is the size of the fully connected layers, which depends on the size of the image.
All convolution and deconvolution layers have stride = 2 and padding = 1 to ensure the spatial dimension decreases/increases by a factor of 2.
M is simply the size of an input image divided by 4 .
Speciﬁcally, for MNIST and Fashion MNIST, M = 7 ; for CIFAR-10, M = 8 ; for CelebA, M = 16 .
BN stands for batch normalization.
For VAEs, the ﬁnal FC layer of the encoder will have doubled output size to return both the mean and standard deviation of latent variables.
B E XPERIMENT S ETTINGS
In this section, we present the details of our experimental settings for results in Table 1.
Since the settings for MNIST and Fashion MNIST are the same, we only mention MNIST for simplicity.
For
GLANN, we directly cite the results from (Hoshen et al., 2019), as their experimental settings is very similar to ours.
We use the original images in the training sets for MNIST, Fashion MNIST and CIFAR-10.
For
CelebA, we follow the same pre-processing as in (Lucic et al., 2018): center crop to 160 × 160 and then resize to 64 × 64 .
We normalize the pixel values to [0 , 1] , without adding noise to pixels (i.e, no de-quantization).
B. S ETTINGS FOR TRAINING GLF
For all datasets (except CelebA-HQ), we use batch size 256 and Adam (Kingma & Ba, 2014) optimizer with initial learning rate 10 − 3 for the parameters of both the AE and the ﬂow.
We add a weight decay 2 × 10 − 5 to the optimizer for the ﬂow.
For MNIST, we train our model for 100 epochs, with learning rate decaying by a factor of 2 after 50 epochs.
For CIFAR-10, we train our model for 200 epochs, with the learning rate decaying by a factor of 2 every 50 epochs.
For CelebA, we train our model for 40 epochs with no learning rate decay.
For GLF with perceptual loss, we compute the perceptual loss as suggested in (Hoshen & Wolf, 2018).
See https://github.com/facebookresearch/NAM/blob/master/code/ perceptual_loss.py for their implementation.
Other settings are the same.
For CelebA-HQ dataset, we adopt our AE network structure based on DCGAN (Radford et al., 2015).
Note that this is a relatively simple network for high resolution imgaes.
We use batch size 64, with initial learning rate 10 − 3 for both the AE and the ﬂow.
We train our model for 60 epochs, with learning rate decaying by a factor of 2 after 40 epochs.
B. S ETTINGS FOR TRAINING VAE S AND VAE VARIANTS
We adopt common settings for our reported results of VAE, VAE+ﬂow prior and VAE+ﬂow posterior.
We use β = 1 for all three VAE variants.
We still use batch size 256, and Adam optimizer with initial learning rate 10 − 3 for both the VAE and the ﬂow, if applicable.
We ﬁnd VAEs need longer time to converge, so we double the training epochs.
We train MNIST for 200 epochs, with learning rate decaying by a factor of 2 after 100 epochs.
We train CIFAR-10 for 400 epochs, with the learning rate decaying by a factor of 2 every 100 epochs.
We train CelebA for 80 epochs with learning rate decaying by a factor of 2 after 40 epochs.
B. S ETTINGS FOR TRAINING WAE-GAN
We follow the settings introduced in the original
WAE paper(Tolstikhin et al., 2017).
The adversary in WAE-GAN has the following architecture:
[Equation]
where d is the dimension of the latent variables.
WAE has two major hyper-parameters: λ that controls the weight coefﬁcient of the adversarial regularizer, and σ 2 which is the variance of the prior.
Batch size is 100 for all datasets.
For MNIST, λ
= 10 and σ
= 1 , and the model is trained for 100 epochs.
The initial learning rate is 10 − 3 for the
AE and 5 × 10 − 4 for the adversary.
After 30 epochs both learning rates decreased both by factor of 2, and after ﬁrst 50 epochs further by factor of 5.
For CIFAR, λ = 10 and σ 2 = 1 and the model is trained for 200 epochs.
The initial learning rates are the same as training MNIST, and the learning rate decays by a factor of 2 after ﬁrst 60 epochs, and further by a factor of 5 after 120 epochs.
For
CelebA, λ = 1 and σ 2 = 2 .
The model is trained for 55 epochs.
The initial learning rate is 3 × 10 −for the AE and 10 − 3 for the adversary.
Both learning rates decays by factor of 2 after 30 epochs, further by factor of 5 after
ﬁrst epochs.
B. S ETTINGS FOR TRAINING T WO STAGE VAE
We adopt the settings in the original paper (Dai & Wipf, 2019).
For all datasets, the batch size is set to be 64, and the initial learning rate for both the ﬁrst and the second is 10 − 4 .
For MNIST is trained
400 epochs, with learning rate halved every 150 epochs second VAE is trained for 800 epochs with learning rate halved every 300 epochs
For CIFAR-10, 1000 and 2000 epochs are trained for the two VAEs respectively the learning rates are halved every 600 epochs the two stages
For CelebA 120 and 300 epochs are trained
Explaining the discrepancy between our reported results and the results in the original paper:
The original Two stage VAE paper adopts similar settings with our experiments, but we observe large discrepancies on the results of CIFAR-10 and CelebA. After carefully reviewing their published codes, we ﬁnd that there is an issue in their FID score computation particularly for CIFAR-10 dataset.
Speciﬁcally, the true images used for computing the FID on CIFAR-10 is obtained from saving the original data ﬁle in .jpg
format and reading them back, and the saving will cause some errors in pixel values.
After ﬁxing this issue, we re-ran their published codes and obtained similar results as we reported.
We also run through their original FID computation protocol using samples from our models, and we obtain scores around 65.
For CelebA, one particular detail worth noting is that, (Dai & Wipf, 2019) applies 128 × 128 center-crop before re-sizing on CelebA, while 160 ×center-crop is used in our evaluations.
With smaller center-crops the human faces occupy a larger portion of the image with less background, making the generative modeling easier.
B. S ETTINGS FOR TRAINING RAE+GMM
The settings of batch size, learning rate scheduling and number of epochs for training RAE are the same as those of GLF.
The objective of the RAE is reconstruction loss plus a penalty on the norm of the latent variable.
Since the author does not report their choices for the penalty coefﬁcient γ , we search over γ ∈ 0 .
5 , 1 , 2 , and we ﬁnd
that β = 0 .
5 leads to the best overall performances, and therefore we let γ = 0 .
After training the RAE, we ﬁt a 10-component Gaussian mixture distribution on the latent variables.
B. S ETTINGS FOR E XPERIMENTS IN S ECTION 4.2.
For all experiments in Section 4.2.1, we use batch size 256 and initial learning rate 10 − 3 for both AE and ﬂow.
We train all models for 500 epochs with learning rates decaying by a factor of 2 everyepochs.
C P RECISION AND R ECALL
In this section
[Table]
we report the precision and recall (PRD) evaluation of samples on each dataset in
Table 3.
We include WAE-GAN, Two-stage VAE, RAE+GMM and GLANN (with perceptual loss) for comparisons.
As the case of FID scores, for GLANN, we directly cite their reported results
[Table]
and we compute the results ofr other models.
We report the PRD of models trained under the settings introduced in Appendix B.
The two numbers in each entry are F 8 , F 18 that capture recall and precision, respectively.
See (Sajjadi et al., 2018) for more details.
Higher numbers are better.
[Table]
D C OMPARISON WITH GAN S
In Table 4 we combine our reported results of AE based models and the FID scores of GANs cited from (Lucic et al., 2018)
[Table] [Table]
E I SSUES WITH LATENT REGULARIZATION
In this appendix section, we present the plots of FID scores, reconstruction loss and NLL loss of regularized GLF.
Related results are discussed in Section 4.2.1.
F T RAINING TIME COMPARISONS
In Table [Table] we report the number of training epochs of our method, two-stage VAE and GLANN.
In
Table 6, we report the clock training time per epoch of these methods.
Note that for methods using perceptual loss, the per epoch training time is longer because VGG activations need to be computed.
These two tables show that GLF needs much shorter training time than the two competing methods.
In GLF, training the ﬂow does not add much computational time due to the low dimensionality.
[Table] [Table]
Quantitative measurements of sample quality, such as FID score and Precision/Recall can be minimized by letting the generative model memorize the training set. The smooth transition of noise interpolation shown in Figure 2 provides evidence that our model can generalize, i
[Table]
e., it generate samples that have not be seen during training. Here we provide additional evidence showing that our generative model generalizes well.
We randomly generate some samples from the models trained on MNIST and CelebA datasets. Then we present the 5 nearest neighbors of each generated samples in the training set. The nearest neighbor is deﬁned in terms of L 2 distance.
Results are shown in Figure 5.
By inspecting the ﬁgure, we ﬁnd that we can easily differentiate each generated sample from the closest training data. This indicates that our model generalizes well.
H M ORE QUALITATIVE RESULTS
In Figure 6, we show more samples of each dataset generated by GLF, using either MSE or perceptual loss as reconstruction loss.
In Figure 7, we show samples of CelebA-HQ datasets from GLF trained with perceptual loss.
In Figure 8, we show examples of interpolations between two randomly sampled noises on CelebA from GLF trained with perceptual loss.
[Figure]
[Figure]
[Figure]