F ROM V ARIATIONAL TO D ETERMINISTIC A UTOENCODERS
A
BSTRACT
Variational Autoencoders (VAEs) provide a theoretically-backed and popular framework for deep generative models.
However, learning a VAE from data poses still unanswered theoretical questions and considerable practical challenges.
In this work, we propose an alternative framework for generative modeling that is simpler, easier to train, and deterministic, yet has many of the advantages of VAEs.
We observe that sampling a stochastic encoder in a Gaussian VAE can be interpreted as simply injecting noise into the input of a deterministic decoder.
We investigate how substituting this kind of stochasticity, with other explicit and implicit regularization schemes, can lead to an equally smooth and meaningful latent space without forcing it to conform to an arbitrarily chosen prior.
To retrieve a generative mechanism to sample new data, we introduce an ex-post density estimation step that can be readily applied also to existing VAEs, improving their sample quality.
We show, in a rigorous empirical study, that the proposed regularized deterministic autoencoders are able to generate samples that are comparable to, or better than, those of VAEs and more powerful alternatives when applied to images as well as to structured data such as molecules.
I NTRODUCTION
Generative models lie at the core of machine learning.
By capturing the mechanisms behind the data generation process, one can reason about data probabilistically, access and traverse the lowdimensional manifold the data is assumed to live on, and ultimately generate new data .
It is therefore not surprising that generative models have gained momentum in applications such as computer vision (Sohn et al., 2015; Brock et al., 2019), NLP (Bowman et al., 2016; Severyn et al., 2017), and chemistry (Kusner et al., 2017; Jin et al., 2018; G´omez-Bombarelli et al., 2018).
Variational Autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) cast learning representations for high-dimensional distributions as a variational inference problem.
Learning a VAE amounts to the optimization of an objective balancing the quality of samples that are autoencoded through a stochastic encoder–decoder pair while encouraging the latent space to follow a ﬁxed prior distribution.
Since their introduction, VAEs have become one of the frameworks of choice among the different generative models.
VAEs promise theoretically well-founded and more stable training than Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) and more efﬁcient sampling mechanisms than autoregressive models (Larochelle & Murray, 2011; Germain et al., 2015).
However, the VAE framework is still far from delivering the promised generative mechanism, as there are several practical and theoretical challenges yet to be solved.
A major weakness of VAEs is the tendency to strike an unsatisfying compromise between sample quality and reconstruction quality.
In practice, this has been attributed to overly simplistic prior distributions (Tomczak & Welling, 2018; Dai & Wipf, 2019) or alternatively, to the inherent over-regularization induced by the KL divergence term in the VAE objective (Tolstikhin et al., 2017).
Most importantly, the VAE objective itself poses several challenges as it admits trivial solutions that decouple the latent space from the input (Chen et al., 2017; Zhao et al., 2017), leading to the posterior collapse phenomenon in conjunction with powerful decoders (van den Oord et al., 2017).
Furthermore, due to its variational formulation, training a VAE requires approximating expectations through sampling at the cost of increased variance in gradients (Burda et al., 2015; Tucker et al., 2017), making initialization, validation, and annealing of hyperparameters essential in practice (Bowman et al., 2016; Higgins et al., 2017; Bauer & Mnih, 2019).
Lastly, even after a satisfactory convergence of the objective, the learned aggregated posterior distribution rarely matches the assumed latent prior in practice (Kingma et al., 2016; Bauer & Mnih, 2019; Dai & Wipf, 2019), ultimately hurting the quality of generated samples.
All in all, much of the attention around VAEs is still directed towards “ﬁxing” the aforementioned drawbacks associated with them.
In this work, we take a different route: we question whether the variational framework adopted by
VAEs is necessary for generative modeling and, in particular, to obtain a smooth latent space.
We propose to adopt a simpler, deterministic version of VAEs that scales better, is simpler to optimize, and, most importantly, still produces a meaningful latent space and equivalently good or better samples than VAEs or stronger alternatives, e.g., Wasserstein Autoencoders (WAEs) (Tolstikhin et al., 2017).
We do so by observing that, under commonly used distributional assumptions, training a stochastic encoder–decoder pair in VAEs does not differ from training a deterministic architecture where noise is added to the decoder’s input.
We investigate how to substitute this noise injection mechanism with other regularization schemes in the proposed deterministic Regularized Autoencoders (RAEs), and we thoroughly analyze how this affects performance.
Finally, we equip RAEs with a generative mechanism via a simple ex-post density estimation step on the learned latent space.
In summary, our contributions are as follows: i) we introduce the RAE framework for generative modeling as a drop-in replacement for many common VAE architectures; ii) we propose an ex-post density estimation scheme which greatly improves sample quality for VAEs, WAEs and RAEs without the need to retrain the models; iii) we conduct a rigorous empirical evaluation to compare RAEs with VAEs and several baselines on standard image datasets and on more challenging structured domains such as molecule generation (Kusner et al., 2017; G ´ omez-Bombarelli et al., 2018).
V ARIATIONAL A UTOENCODERS
For a general discussion, we consider a collection of high-dimensional i.i.d.
samples X = { x i } Ni = drawn from the true data distribution p data ( x ) over a random variable
X taking values in the input space.
The aim of generative modeling is to learn from X a mechanism to draw new samples x new ∼ p data .
Variational Autoencoders provide a powerful latent variable framework to infer such a mechanism.
The generative process of the VAE is deﬁned as
[Equation]
where p ( Z ) is a ﬁxed prior distribution over a low-dimensional latent space Z .
A stochastic decoder
[Equation]
links the latent space to the input space through the likelihood distribution p θ , where g θ is an expressive non-linear function parameterized by θ .
As a result, a VAE estimates p data ( x ) as the inﬁnite mixture model p θ ( x ) =
[Equation]
p θ
[Equation]
x | z ) p
[Equation]
z ) d z .
At the same time, the input space is mapped to the latent space via a stochastic encoder
[Equation]
where q φ ( z | x ) is the posterior distribution given by a second function f φ parameterized by φ .
Computing the marginal log-likelihood log p θ ( x ) is generally intractable.
One therefore follows a variational approach, maximizing the evidence lower bound (ELBO) for a sample x :
[Equation]
Maximizing Eq.
4 over data X w.r.t.
model parameters φ , θ corresponds to minimizing the loss
[Equation]
5) where L REC and L KL are deﬁned for a sample x as follows:
[Equation]
Intuitively, the reconstruction loss L REC takes into account the quality of autoencoded samples x through D θ ( E φ ( x )) , while the KL-divergence term L KL encourages q φ ( z | x ) to match the prior p ( z ) for each z which acts as a regularizer during training (Hoffman & Johnson, 2016).
2. P RACTICE AND SHORTCOMINGS OF VAE S
To ﬁt a VAE to data through Eq.
5 one has to specify the parametric forms for p ( z ) , q φ ( z | x ) , p θ ( x | z ) , and hence the deterministic mappings f φ and g θ .
In practice, the choice for the above distributions is guided by trading off computational complexity with model expressiveness.
In the most commonly adopted formulation of the VAE, q φ ( z | x ) and p θ ( x | z ) are assumed to be Gaussian:
[Equation]
with means µ φ , µ θ and covariance parameters σ φ , σ θ given by f φ and g θ .
In practice, the covariance of the decoder is set to the identity matrix for all z , i.e., σ θ ( z ) =
(Dai & Wipf, 2019).
The expectation of L REC in Eq.
6 must be approximated via k Monte Carlo point estimates.
It is expected that the quality of the Monte Carlo estimate, and hence convergence during learning and sample quality increases for larger k
(Burda et al., 2015).
However, only a 1-sample approximation is generally carried out (Kingma & Welling, 2014) since memory and time requirements are prohibitive for large k .
With the 1-sample approximation, L REC can be computed as the mean squared error between input samples and their mean reconstructions µ θ by a decoder that is deterministic in practice:
[Equation]
Gradients w.r.t.
the encoder parameters φ are computed through the expectation of L REC in Eq.
6 via the reparametrization trick (Kingma & Welling, 2014) where the stochasticity of E φ is relegated to an auxiliary random variable (cid:15) which does not depend on φ :
[Equation]
where (cid:12) denotes the Hadamard product.
An additional simplifying assumption involves ﬁxing the prior p ( z ) to be a d -dimensional isotropic Gaussian N ( Z | 0 , I ) .
For this choice, the KL-divergence for a sample x is given in closed form: 2 L KL
[Equation]
While the above assumptions make VAEs easy to implement, the stochasticity in the encoder and decoder are still problematic in practice (Makhzani et al., 2016;
Tolstikhin et al., 2017; Dai & Wipf, 2019).
In particular, one has to carefully balance the trade-off between the L KL term and L REC during optimization (Dai & Wipf, 2019; Bauer & Mnih, 2019).
A too-large weight on the L KL term can dominate L ELBO , having the effect of over-regularization .
As this would smooth the latent space, it can directly affect sample quality in a negative way.
Heuristics to avoid this include manually ﬁnetuning or gradually annealing the importance of L KL during training (Bowman et al., 2016; Bauer &
Mnih, 2019).
We also observe this trade-off in a practical experiment in Appendix A.
Even after employing the full array of approximations and “tricks” to reach convergence of Eq.for a satisfactory set of parameters, there is no guarantee that the learned latent space is distributed according to the assumed prior distribution.
In other words, the aggregated posterior distribution q φ ( z ) =
E x ∼ p data q ( z | x ) has been shown not to conform well to p ( z ) after training (Tolstikhin et al., 2017; Bauer & Mnih, 2019; Dai & Wipf, 2019).
This critical issue severely hinders the generative mechanism of VAEs (cf. Eq. 1) since latent codes sampled from p ( z ) (instead of q ( z ) ) might lead to regions of the latent space that are previously unseen to D θ during training.
This results in generating out-of-distribution samples.
We refer the reader to Appendix H for a visual demonstration of this phenomenon on the latent space of VAEs.
We analyze solutions to this problem in Section 4.
2. C ONSTANT -V ARIANCE E NCODERS
Before introducing our fully-deterministic take on VAEs, it is worth investigating intermediate ﬂavors of VAEs with reduced stochasticity.
Analogous to what is commonly done for decoders as discussed in the previous section, one can ﬁx the variance of q φ ( z | x ) to be constant for all x .
This simpliﬁes the computation of E φ from Eq.
9 to
[Equation]
where σ is a ﬁxed scalar.
Then, the KL loss term in a Gaussian VAE simpliﬁes (up to a constant) to L CVKL =
|| µ φ ( x ) ||
We name this variant Constant-Variance VAEs (CV-VAEs).
While CV-VAEs have been adopted in some applications such as variational image compression (Ball´e et al., 2017) and adversarial robustness (Ghosh et al., 2019), to the best of our knowledge, there is no systematic study of them in the literature.
We will ﬁll this gap in our experiments in Section 6.
Lastly, note that now σ in Eq.10 is not learned along the encoder as in Eq. 9. Nevertheless, it can still be ﬁtted as an hyperparameter, e.g., by cross-validation, to maximise the model likelihood. This highlights the possibility to estimate a better parametric form for the latent space distribution after training, or in a outer-loop including training.
We address this provide a more complex and ﬂexible solution to deal with the prior structure over Z via ex-post density estimation in Section 4.
D ETERMINISTIC R EGULARIZED A UTOENCODERS
Autoencoding in VAEs is deﬁned in a probabilistic fashion: E φ and D θ map data points not to a single point, but rather to parameterized distributions (cf.
Eq. 7).
However, common implementations of VAEs as discussed in Section 2 admit a simpler, deterministic view for this probabilistic mechanism.
A glance at the autoencoding mechanism of the VAE is revealing.
The encoder deterministically maps a data point x to mean µ φ ( x ) and variance σ φ ( x ) in the latent space.
The input to D θ is then simply the mean µ φ ( x ) augmented with Gaussian noise scaled by σ φ ( x ) via the reparametrization trick (cf.
Eq. 9).
In the CV-VAE, this relationship is even more obvious, as the magnitude of the noise is ﬁxed for all data points (cf.
Eq. 10).
In this light, a VAE can be seen as a deterministic autoencoder where (Gaussian) noise is added to the decoder’s input.
We argue that this noise injection mechanism is a key factor in having a regularized decoder.
Using random noise injection to regularize neural networks is a well-known technique that dates back several decades (Sietsma & Dow, 1991; An, 1996).
It implicitly helps to smooth the function learned by the network at the price of increased variance in the gradients during training.
In turn, decoder regularization is a key component in generalization for VAEs, as it improves random sample quality and achieves a smoother latent space.
Indeed, from a generative perspective, regularization is motivated by the goal to learn a smooth latent space where similar data points x are mapped to similar latent codes z , and small variations in Z lead to reconstructions by D θ that vary only slightly.
We propose to substitute noise injection with an explicit regularization scheme for the decoder .
This entails the substitution of the variational framework in VAEs, which enforces regularization on the encoder posterior through L KL , with a deterministic framework that applies other ﬂavors of decoder regularization.
By removing noise injection from a CV-VAE, we are effectively left with a deterministic autoencoder (AE).
Coupled with explicit regularization for the decoder, we obtain a
Regularized Autoencoder (RAE).
Training a RAE thus involves minimizing the simpliﬁed loss
[Equation]
where L REG represents the explicit regularizer for D θ (discussed in Section 3.1) and L RAE Z = 1 / 2 ||
z
|| 22 (resulting from simplifying L CVKL ) is equivalent to constraining the size of the learned latent space, which is still needed to prevent unbounded optimization.
Finally, β and λ are two hyper parameters that balance the different loss terms.
Note that for RAEs, no Monte Carlo approximation is required to compute L REC .
This relieves the need for more samples from q φ ( z | x ) to achieve better image quality (cf.
Appendix A).
Moreover, by abandoning the variational framework and the L KL term, there is no need in RAEs for a ﬁxed prior distribution over Z .
Doing so however loses a clear generative mechanism for RAEs to sample from Z .
We propose a method to regain random sampling ability in Section 4 by performing density estimation on Z ex-post, a step that is otherwise still needed for VAEs to alleviate the posterior mismatch issue.
3. R EGULARIZATION S CHEMES FOR RAE S
Among possible choices for L REG , a ﬁrst obvious candidate is Tikhonov regularization (Tikhonov & Arsenin, 1977) since is known to be related to the addition of low-magnitude input noise (Bishop, 2006).
Training a RAE within this framework thus amounts to adopting L REG =
L L 2 =
||
θ || 22 which effectively applies weight decay on the decoder parameters θ .
Another option comes from the recent GAN literature where regularization is a hot topic (Kurach et al., 2018) and where injecting noise to the input of the adversarial discriminator has led to improved performance in a technique called instance noise (Sønderby et al., 2017).
To enforce Lipschitz continuity on adversarial discriminators, weight clipping has been proposed (Arjovsky et al., 2017), which is however known to signiﬁcantly slow down training.
More successfully, a gradient penalty on the discriminator can be used similar to Gulrajani et al.
(2017); Mescheder et al.
, yielding the objective L REG =
L GP =
||∇ D
θ ( E φ ( x )) ||
which bounds the gradient norm of the decoder w.r.t.
its input.
Additionally, spectral normalization (SN) has been successfully proposed as an alternative way to bound the Lipschitz norm of an adversarial discriminator (Miyato et al., 2018).
SN normalizes each weight matrix θ (cid:96) in the decoder by an estimate of its largest singular value: θ SN (cid:96) =
θ (cid:96) /
s ( θ (cid:96) ) where s ( θ (cid:96) ) is the current estimate obtained through the power method.
In light of the recent successes of deep networks without explicit regularization (Zagoruyko & Komodakis, 2016; Zhang et al., 2017), it is intriguing to question the need for explicit regularization of the decoder in order to obtain a meaningful latent space.
The assumption here is that techniques such as dropout (Srivastava et al., 2014), batch normalization (Ioffe & Szegedy, 2015), adding noise during training (An, 1996) implicitly regularize the networks enough.
Therefore, as a natural baseline to the L RAE objectives introduced above, we also consider the RAE framework without L REG and L RAE Z , i.e., a standard deterministic autoencoder optimizing L REC only.
To complete our “autopsy” of the VAE loss, we additionally investigate deterministic autoencoders with decoder regularization, but without the L RAE Z term, as well as possible combinations of different regularizers in our RAE framework (cf.
Table 3 in Appendix I).
Lastly, it is worth questioning if it is possible to formally derive our RAE framework from ﬁrst principles.
We answer this afﬁrmatively, and show how to augment the ELBO optimization problem of a VAE with an explicit constraint, while not ﬁxing a parametric form for q φ ( z | x ) .
This indeed leads to a special case of the RAE loss in Eq.
Speciﬁcally, we derive a regularizer like L GP for a deterministic version of the CV-VAE.
Note that this derivation legitimates bounding the decoder’s gradients and as such it justiﬁes the spectral norm regularizer as well since the latter enforces the decoder’s Lipschitzness.
We accommodate the full derivation in Appendix B.
E X -P OST D ENSITY E STIMATION
By removing stochasticity and ultimately, the KL divergence term L KL from RAEs, we have simpliﬁed the original VAE objective at the cost of detaching the encoder from the prior p ( z ) over the latent space.
This implies that i) we cannot ensure that the latent space Z is distributed according to a simple distribution (e.g., isotropic Gaussian) anymore and consequently, ii) we lose the simple mechanism provided by p ( z ) to sample from Z as in Eq. 1.
As discussed in Section 2.1, issue i) is compromising the VAE framework in any case, as reported in several works (Hoffman & Johnson, 2016; Rosca et al., 2018; Dai & Wipf, 2019).
To ﬁx this, some works extend the VAE objective by encouraging the aggregated posterior to match p ( z ) (Tolstikhin et al., 2017) or by utilizing more complex priors (Kingma et al., 2016; Tomczak & Welling, 2018;
Bauer & Mnih, 2019).
To overcome both i) and ii), we instead propose to employ ex-post density estimation over Z .
We ﬁt a density estimator denoted as q δ ( z ) to { z = E φ ( x ) | x ∈ X} .
This simple approach not only ﬁts our
RAE framework well, but it can also be readily adopted for any VAE or variants thereof such as the
WAE as a practical remedy to the aggregated posterior mismatch without adding any computational overhead to the costly training phase.
The choice of q δ ( z ) needs to trade-off expressiveness – to provide a good ﬁt of an arbitrary space for Z – with simplicity , to improve generalization.
For example, placing a Dirac distribution on each latent point z would allow the decoder to output only training sample reconstructions which have a high quality, but do not generalize.
Striving for simplicity, we employ and compare a full covariance multivariate Gaussian with a 10-component Gaussian mixture model (GMM) in our experiments.
R ELATED WORKS
Many works have focused on diagnosing the VAE framework, the terms in its objective (Hoffman & Johnson, 2016;
Zhao et al., 2017; Alemi et al., 2018), and ultimately augmenting it to solve optimization issues (Rezende & Viola, 2018; Dai & Wipf, 2019).
With RAE, we argue that a simpler deterministic framework can be competitive for generative modeling.
Deterministic denoising (Vincent et al., 2008) and contractive autoencoders (CAEs) (Rifai et al., 2011) have received attention in the past for their ability to capture a smooth data manifold.
Heuristic attempts to equip them with a generative mechanism include MCMC schemes (Rifai et al., 2012;
Bengio et al., 2013).
However, they are hard to diagnose for convergence, require a considerable effort in tuning (Cowles & Carlin, 1996), and have not scaled beyond MNIST, leading to them being superseded by VAEs.
While computing the Jacobian for CAEs (Rifai et al., 2011) is close in spirit to L GP for RAEs, the latter is much more computationally efﬁcient.
Approaches to cope with the aggregated posterior mismatch involve ﬁxing a more expressive form for p ( z ) (Kingma et al., 2016; Bauer & Mnih, 2019) therefore altering the VAE objective and requiring considerable additional computational efforts.
Estimating the latent space of a VAE with a second VAE (Dai & Wipf, 2019) reintroduces many of the optimization shortcomings discussed for
VAEs and is much more expensive in practice compared to ﬁtting a simple q δ ( z ) after training.
Adversarial Autoencoders (AAE) (Makhzani et al., 2016) add a discriminator to a deterministic encoder–decoder pair, leading to sharper samples at the expense of higher computational overhead and the introduction of instabilities caused by the adversarial nature of the training process. [Figure] RAE provides slightly sharper samples and reconstructions while interpolating smoothly in the latent space.
Corresponding qualitative overviews for MNIST and CIFAR-10 are provided in Appendix F.
Wasserstein Autoencoders (WAE) (Tolstikhin et al., 2017) have been introduced as a generalization of AAEs by casting autoencoding as an optimal transport (OT) problem.
Both stochastic and deterministic models can be trained by minimizing a relaxed OT cost function employing either an adversarial loss term or the maximum mean discrepancy score between p ( z ) and q φ ( z ) as a reg- ularizer in place of L KL .
Within the RAE framework, we look at this problem from a different perspective: instead of explicitly imposing a simple structure on Z that might impair the ability to ﬁt high-dimensional data during training, we propose to model the latent space by an ex-post density estimation step.
The most successful VAE architectures for images and audio so far are variations of the VQVAE (van den Oord et al., 2017; Razavi et al., 2019).
Despite the name, VQ-VAEs are neither stochastic, nor variational, but they are deterministic autoencoders.
VQ-VAEs are similar to RAEs in that they adopt ex-post density estimation.
However, VQ-VAEs necessitates complex discrete autoregressive density estimators and a training loss that is non-differentiable due to quantizing Z .
Lastly, RAEs share some similarities with GLO ( ? ).
However, differently from RAEs, GLO can be interpreted as a deterministic AE without and encoder, and when the latent space is built “ondemand” by optimization.
On the other hand, RAEs augment deterministic decoders as in GANs with deterministic encoders.
E XPERIMENTS
Our experiments are designed to answer the following questions: Q1:
Are sample quality and latent space structure in RAEs comparable to VAEs?
Q2:
How do different regularizations impact RAE performance?
Q3:
What is the effect of ex-post density estimation on VAEs and its variants?
[Table]
sian to q δ ( z ) (for the remaining models); GMM : random samples generated by ﬁtting a mixture of 10 Gaussians in the latent space; Interp .
: mid-point interpolation between random pairs of test reconstructions.
The RAE models are competitive with or outperform previous models throughout the evaluation.
Interestingly, interpolations do not suffer from the lack of explicit priors on the latent space in our models.
6. RAE S FOR IMAGE MODELING
We evaluate all regularization schemes from Section 3.1: RAE-GP, RAE-L2, and RAE-SN.
For a thorough ablation study, we also consider only adding the latent code regularizer L RAE Z to L REC (RAE), and an autoencoder without any explicit regularization (AE).
We check the effect of applying one regularization scheme while not including the L RAE Z term in the AE-L2 model.
As baselines, we employ the regular VAE, constant-variance VAE (CV-VAE), Wasserstein Autoencoder (WAE) with the MMD loss as a state-of-the-art method, and the recent 2-stage VAE (2sVAE) (Dai & Wipf, 2019) which performs a form of ex-post density estimation via another VAE.
For a fair comparison, we use the same network architecture for all models.
Further details about the architecture and training are given in Appendix C.
We measure the following quantities: held-out sample reconstruction quality, random sample quality, and interpolation quality.
While reconstructions give us a lower bound on the best quality achievable by the generative model, random sample quality indicates how well the model generalizes.
Finally, interpolation quality sheds light on the structure of the learned latent space.
The evaluation of generative models is a nontrivial research question (Theis et al., 2016; Sajjadi et al., 2017; Lucic et al., 2018).
We report here the ubiquitous Fr´echet Inception Distance (FID) (Heusel et al., 2017) and we provide precision and recall scores (PRD)
(Sajjadi et al., 2018) in Appendix E.
Table 1 summarizes our main results.
All of the proposed RAE variants are competitive with the
VAE, WAE and 2sVAE w.r.t.
generated image quality in all settings.
Sampling RAEs achieve the best FIDs across all datasets when a modest 10-component GMM is employed for ex-post density estimation.
Furthermore, even when N is considered as q δ ( z ) , RAEs rank ﬁrst with the exception of
MNIST, where it competes for the second position with a VAE.
Our best RAE FIDs are lower than the best results reported for VAEs in the large scale comparison of (Lucic et al., 2018), challenging even the best scores reported for GANs.
While we are employing a slightly different architecture than theirs, our models underwent only modest ﬁnetuning instead of an extensive hyperparameter search.
A comparison of the different regularization schemes for RAEs (Q2) yields no clear winner across all settings as all perform equally well.
Striving for a simpler implementation, one may prefer
RAE-L2 over the GP and SN variants.
For completeness, we investigate applying multiple regularization schemes to our RAE models.
We report the results of all possible combinations in Table 3, Appendix I. There, no signiﬁcant boost of performance can be spotted when comparing to singly regularized RAEs.
Surprisingly, the implicitly regularized RAE and AE models are shown to be able to score impressive
FIDs when q δ ( z ) is ﬁt through GMMs.
FIDs for AEs decrease from 58.73 to 10.66 on MNIST and from 127.85 to 45.10 on CelebA – a value close to the state of the art.
This is a remarkable result that follows a long series of recent conﬁrmations that neural networks are surprisingly smooth by design (Neyshabur et al., 2017).
It is also surprising that the lack of an explicitly ﬁxed structure on the latent space of the RAE does not impede interpolation quality.
This is further conﬁrmed by the qualitative evaluation on CelebA as reported in Fig.
2 and for the other datasets in Appendix F, where RAE interpolated samples seem sharper than competitors and transitions smoother.
Our results further conﬁrm and quantify the effect of the aggregated posterior mismatch.
In Table 1, ex-post density estimation consistently improves sample quality across all settings and models.
A 10-component GMM halves FID scores from ∼ 20 to ∼ 10 for WAE and RAE models on MNIST and from 116 to 46 on CelebA.
This is especially striking since this additional step is much cheaper and simpler than training a second-stage VAE as in 2sVAE (Q3).
In summary, the results strongly support the conjecture that the simple deterministic RAE framework can challenge VAEs and stronger alternatives (Q1).
6. G RAMMAR RAE: MODELING STRUCTURED INPUTS
We now evaluate RAEs for generating complex structured objects such as molecules and arithmetic expressions.
We do this with a twofold aim: i) to investigate the latent space learned by RAE for more challenging input spaces that abide to some structural constraints, and ii) to quantify the gain of replacing the VAE in a state-of-the-art generative model with a RAE.
To this end, we adopt the exact architectures and experimental settings of the GrammarVAE (GVAE) (Kusner et al., 2017), which has been shown to outperform other generative alternatives such as the CharacterVAE (CVAE) (G´omez-Bombarelli et al., 2018).
As in Kusner et al.
(2017), we are interested in traversing the latent space learned by our models to generate samples (molecules or expressions) that best ﬁt some downstream metric.
This is done by Bayesian optimization (BO) by considering the log(1 + MSE ) (lower is better) for the generated expressions w.r.t.
some ground truth points, and the water-octanol partition coefﬁcient ( log P ) (Pyzer-Knapp et al., 2015) (higher is better) in the case of molecules.
A well-behaved latent space will not only generate molecules or expressions with better scores during the BO step, but it will also contain syntactically valid ones, i.e., , samples abide to a grammar of rules describing the problem.
Figure 3 summarizes our results over 5 trials of BO.
Our GRAEs (Grammar RAE) achieve better average scores than CVAEs and GVAEs in generating expressions and molecules.
This is visible also for the three best samples and their scores for all models, with the exception of the ﬁrst best expression of GVAE.
We include in the comparison also the GCVVAE, the equivalent of a CV-VAE [Table] Cl NH Cl O Cl F
O
Cl O NH Cl NH Cl O Cl F
O
Cl
S CORE 3.
GVAE
Cl N
O
F
O
Cl N
O
F
O
Cl N
O
F
O
S CORE 3.
CVAE
O
O SS
NH 2 NH
S NN
S N NNN NN
O
O S S
NH 2 NH
S N N
S N N N N
N N O
O SS
NH 2 NH
S NN
S N NNN NN
S CORE 2. for structured objects, as an additional baseline.
We can observe that while the GCVVAE delivers better average scores for the simpler task of generating equations (even though the single three best equations are on par with GRAE), when generating molecules GRAEs deliver samples associated to much higher scores.
More interestingly, while GRAEs are almost equivalent to GVAEs for the easier task of generating expressions, the proportion of syntactically valid molecules for GRAEs greatly improves over
GVAEs (from 28% to 72%).
C ONCLUSION
While the theoretical derivation of the VAE has helped popularize the framework for generative modeling, recent works have started to expose some discrepancies between theory and practice.
We have shown that viewing sampling in VAEs as noise injection to enforce smoothness can enable one to distill a deterministic autoencoding framework that is compatible with several regularization techniques to learn a meaningful latent space.
We have demonstrated that such an autoencoding framework can generate comparable or better samples than VAEs while getting around the practical drawbacks tied to a stochastic framework.
Furthermore, we have shown that our solution of ﬁtting a simple density estimator on the learned latent space consistently improves sample quality both for the proposed RAE framework as well as for VAEs, WAEs, and 2sVAEs which solves the mismatch between the prior and the aggregated posterior in VAEs. A CKNOWLEDGEMENTS We would like to thank Anant Raj, Matthias Bauer, Paul Rubenstein and Soubhik Sanyal for fruitful discussions. variatiodelectat!
A PPENDIX A R ECONSTRUCTION AND REGULARIZATION TRADE - OFF
We train a VAE on MNIST while monitoring the test set reconstruction quality by FID.
Figure(left) clearly shows the impact of more expensive k > 1 Monte Carlo approximations of Eq.
7 on sample quality during training.
The commonly used 1-sample approximation is a clear limitation for VAE training.
Figure 4 (right) depicts the inherent trade-off between reconstruction and random sample quality in VAEs.
Enforcing structure and smoothness in the latent space of a VAE affects random sample quality in a negative way.
In practice, a compromise needs to be made, ultimately leading to subpar performance.
[Figure]
B A P ROBABILISTIC D ERIVATION OF R EGULARIZATION
In this section, we propose an alternative view on enforcing smoothness on the output of D θ by augmenting the ELBO optimization problem for VAEs with an explicit constraint.
While we keep the Gaussianity assumptions over a stochastic D θ and p ( z ) for convenience, we however are not ﬁxing a parametric form for q φ ( z | x ) yet.
We discuss next how some parametric restrictions over q φ ( z | x ) lead to a variation of the RAE framework in Eq.
11, speciﬁcally the introduction of L GP as a regularizer of a deterministic version of the CV-VAE.
To start, we augment Eq.
5 as:
[Equation]
[Equation]
where D θ ( z )
= µ θ ( E φ ( x )) and the constraint on the decoder encodes that the output has to vary, in the sense of an L p norm, only by a small amount (cid:15) for any two possible draws from the encoding of x . Let D θ ( z ) :
R dim( z ) → R dim( x ) be given by a set of dim( x ) given by { d i ( z ) :
R dim( z ) → R 1 } .
Now we can upper bound the quantity || D
θ ( z 1 ) − D θ ( z 2 ) || p by dim( x ) ∗ sup i {|| d i ( z 1 ) − d i ( z 2 ) || p } . Using mean value theorem ||
d i ( z 1 )
− d i ( z 2 ) ||
p ≤ ||∇ t d
i ((1 − t ) z 1 + t z 2 ) || p ∗ ||
z 1 − z
2 || p .
Hence sup i {|| d i ( z 1 ) − d i ( z 2 ) || p } ≤ sup i {||∇ t d i ((1 − t ) z
1 + t z 2 )
|| p ∗|| z 1 −
z
||
p } .
Now if we choose the domain of q φ ( z | x ) to be isotopic the contribution of || z 2 − z 1 || p to the afore mentioned quantity becomes a constant factor.
Loosely speaking it is the radios of the bounding ball of domain of q φ ( z | x ) .
Hence the above term simpliﬁes to sup i {||∇ t d
i ((1 − t ) z 1 + t z 2 ) || p } . Recognizing that here z 1 and z 2 is arbitrary lets us simplify this further to sup i {||∇ z d i ( z ) || p } From this form of the smoothness constraint, it is apparent why the choice of a parametric form for q φ ( z | x ) can be impactful during training.
For a compactly supported isotropic PDF q φ ( z | x ) , the extension of the support sup {|| z 1 − z 2 || p } would depend on its entropy H ( q φ ( z | x )) .
through some functional r .
For instance, a uniform posterior over a hypersphere in z would ascertain r ( H ( q φ ( z | x ))) ∼ = e H ( q φ ( z | x ))
/n where n is the dimensionality of the latent space.
Intuitively, one would look for parametric distributions that do not favor overﬁtting, e.g., degenerating in Dirac-deltas (minimal entropy and support) along any dimensions.
To this end, an isotropic nature of q φ ( z | x ) would favor such a robustness against decoder over-ﬁtting.
We can now rewrite the constraint as
[Equation]
The L KL term can be expressed in terms of H ( q φ ( z | x )) , by decomposing it as L KL =
L CE − L H , where L H = H ( q φ ( z | x )) and L CE
= H ( q φ ( z | x ) , p ( z )) represents a cross-entropy term.
Therefore, the constrained problem in Eq.
12 can be written in a Lagrangian formulation by including Eq. 13:
[Equation]
(14) where L LANG =
r ( H ( q φ ( z | x ))) ∗ ||∇ D
θ ( z ) || p .
We argue that a reasonable simplifying assumption for q φ ( z | x ) is to ﬁx H ( q φ ( z | x )) to a single constant for all samples x .
Intuitively, this can be understood as ﬁxing the variance in q φ ( z | x ) as we did for the CV-VAE in Section 2.2.
With this simpliﬁcation, Eq.
14 further reduces to
[Equation]
We can see that ||∇ D θ ( z ) || p results to be the gradient penalty L GP and L CE =
||
z || 22 corresponds to
L RAEKL , thus recovering our RAE framework as presented in Eq.
C N ETWORK ARCHITECTURE , T RAINING D ETAILS AND E VALUATION
We follow the models adopted by Tolstikhin et al.
(2017) with the difference that we consistently apply batch normalization (Ioffe & Szegedy, 2015).
The latent space dimension is 16 for MNIST (LeCun et al., 1998), 128 for CIFAR-10 (Krizhevsky & Hinton, 2009) and 64 for CelebA (Liu et al., 2015).
For all experiments, we use the Adam optimizer with a starting learning rate of 10 − 3 which is cut in half every time the validation loss plateaus.
All models are trained for a maximum of 100 epochs on MNIST and CIFAR and 70 epochs on CelebA. We use a mini-batch size of 100 and pad MNIST digits with zeros to make the size 32 × 32 .
We use the ofﬁcial train, validation and test splits of CelebA. For MNIST and CIFAR, we set aside 10k train samples for validation.
For random sample evaluation, we draw samples from N (0 , I ) for VAE and WAE-MMD and for all remaining models, samples are drawn from a multivariate
Gaussian whose parameters (mean and covariance) are estimated using training set embeddings.
For the GMM density estimation, we also utilize the training set embeddings for ﬁtting and validation set embeddings to verify that GMM models are not over ﬁtting to training embeddings.
However, due to the very low number of mixture components (10), we did not encounter overﬁtting at this step.
The GMM parameters are estimated by running EM for at most 100 iterations.
[Table]
[-x ∈ R 32 × 32 → C ONV 128 → BN → R E LU → C ONV 256 → BN → R E LU → C ONV 
-]
[-→ BN → R E LU → C ONV 1024 → BN → R E LU → F LATTEN → FC 
-]
[-× M x ∈ R 32 × 32 → C ONV 128 → BN → R E LU → C ONV 
-]
[-→ BN → R E LU → C ONV 
-]
[Table]
[Table]
Conv n represents a convolutional layer with n ﬁlters.
All convolutions Conv n and transposed convolutions ConvT n have a ﬁlter size of 4 × 4 for MNIST and CIFAR-10 and 5 × 5 for CELEBA.
They all have a stride of size 2 except for the last convolutional layer in the decoder.
Finally, M =for all models except for the VAE which has M = 2 as the encoder has to produce both mean and variance for each input.
D E VALUATION S ETUP
We compute the FID of the reconstructions of random validation samples against the test set to evaluate reconstruction quality.
For evaluating generative modeling capabilities, we compute the
FID between the test data and randomly drawn samples from a single Gaussian that is either the isotropic p ( z ) ﬁxed for VAEs and WAEs, a learned second stage VAE for 2sVAEs, or a single
Gaussian ﬁt to q δ ( z ) for CV-VAEs and RAEs.
For all models, we also evaluate random samples from a 10-component Gaussian Mixture model (GMM) ﬁt to q δ ( z ) .
Using only 10 components prevents us from overﬁtting (which would indeed give good FIDs when compared with the test set) 3 .
For interpolations, we report the FID for the furthest interpolation points resulted by applying spherical interpolation to randomly selected validation reconstruction pairs.
We use 10k samples for all FID and PRD evaluations.
Scores for random samples are evaluated against the test set.
Reconstruction scores are computed from validation set reconstructions against the respective test set.
Interpolation scores are computed by interpolating latent codes of a pair of randomly chosen validation embeddings vs test set samples.
The visualized interpolation samples are interpolations between two randomly chosen test set images.
E E VALUATION BY P RECISION AND R ECALL
[Table]
It is notable that the proposed ex-post density estimation improves not only precision, but also recall throughout the experiment. For example, WAE seems to have a comparably low recall of only 0 . 88 on MNIST which is raised considerably to 0 . 95 by ﬁtting a GMM.
In all cases, GMM gives the best results. Another interesting point is the low precision but high recall of all models on CIFAR-10 – this is also visible upon inspection of the samples in Fig. [Figure] [Figure]
[Figure] [Figure] [Figure]
M
[Figure]
UALITATIVE
[Figure] [Figure]
G I NVESTIGATING O VERFITTING
[Figure]
seems that have generalized well only
H V ISUALIZING E X -P OST D ENSITY E STIMATION
To visualize that ex-post density estimation does indeed help reduce the mismatch between the aggregated posterior and the prior we train a VAE on the MNIST dataset whose latent space isdimensional.
The unique advantage of this setting is that one can simply visualize the density of test sample in the latent space by plotting them as a scatterplot.
As it can be seen from ﬁgure 12, an expressive density estimator effectively ﬁxes the miss-match and this as reported earlier results in better sample quality.
[Figure]
Here in ﬁgure 13 we perform the same visualization on with all the models trained on the MNIST dataset as employed on our large evaluation in Table 1.
Clearly every model depicts rather large mismatch between aggregate posterior and prior.
Once again the advantage of ex-post density estimate is clearly visible.
[Figure]
[Table]
I C OMBINING MULTIPLE REGULARIZATION TERMS
The rather intriguing facts that AE without explicit decoder regularization performs reasonably well as seen from table 1, indicates that convolutional neural networks when combined with gradient based optimizers inherit some implicit regularization.
This motivates us to investigate a few different combinations of regularizations e.g. we regularize the decoder of an auto-encoder while drop the regularization in the z space.
The results of this experiment is reported in the row marked AE-L2 in table 3.
Further more a recent GAN literature ? report that often a combination of regularizations boost performance of neural networks.
Following this, we combine multiple regularization techniques in out framework.
However note that this rather drastically increases the hyper parameters and the models become harder to train and goes against the core theme of this work, which strives for simplicity.
Hence we perform simplistic effort to tune all the hyper parameters to see if this can provide boost in the performance, which seem not to be the case.
These experiments are summarized in the second half of the table