V ARIANCE R EDUCTION IN H IERARCHICAL V ARIATIONAL A UTOENCODERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Variational autoencoders with deep hierarchies of stochastic layers have been known to suffer from the problem of posterior collapse, where the top layers fall back to the prior and become independent of input.
We suggest that the hierarchical
VAE objective explicitly includes the variance of the function parameterizing the mean and variance of the latent Gaussian distribution which itself is often a high variance function.
Building on this we generalize VAE neural networks by incorporating a smoothing parameter motivated by Gaussian analysis to reduce higher frequency components and consequently the variance in parameterizing functions and show that this can help to solve the problem of posterior collapse.
We further show that under such smoothing the VAE loss exhibits a phase transition, where the top layer KL divergence sharply drops to zero at a critical value of the smoothing parameter that is similar for the same model across datasets.
We validate the phenomenon across model conﬁgurations and datasets
I NTRODUCTION
Variational autoencoders (VAE) [10] are a popular latent variable model for unsupervised learning that simpliﬁes learning by the introduction of a learned approximate posterior.
Given data x and latent variables z , we specify the conditional distribution p ( x | z ) by parameterizing the distribution parameters by a neural network.
Since it is difﬁcult to learn such a model directly, another conditional distribution q ( z | x ) is introduced to approximate the posterior distribution.
During learning the goal is to maximize the evidence lower bound (ELBO), which lower bounds the log likelihood, log p ( x ) ≥ E q ( z | x ) (cid:2) log p ( x | z )+log p ( z ) − log q ( z | x ) (cid:3) .
In their simplest form, the generative model p ( x | z ) and the approximate posterior q ( z | x ) are Gaussian distributions optimized in unison.
A natural way to increase the modeling capacity of VAE is to incorporate a hierarchy of stochastic variables.
Such models, however, turn out to be difﬁcult to train and higher levels in the hierarchy tend to remain independent of input data – a problem termed posterior collapse .
Posterior collapse in
VAEs manifests itself by the latent distribution tending to fall back to the prior.
With hierarchical
VAEs the effect is found to be more pronounced in the top layers farther from the output.
For the purpose of the paper and for clarity of exposition, we focus on the simplest extension of hierarchical variational autoencoders where stochastic layers are stacked serially on top of each other [2, 21] , p ( x, z ) =
p ( x | z 1 ) p ( z L ) (cid:81)
L − 1 i =1 p
z i | z i +1 ) and q ( z | x ) =
q ( z 1 | x ) (cid:81)
L − 1 i =1 q
( z i +1 |
z i ) .
The intermediate distributions in this model are commonly taken to be Gaussian distributions parameterized by neural network functions, so that p ( z i | z i +1 )
=
N ( z i | µ ( z i +1 ) , σ ( z i +1 )) , where µ ( z ) , σ ( z ) are neural networks computing the mean and variance of the Gaussian distribution.
We refer to them as vanilla hierarchical variational autoencoders .
For each stochastic layer in this model there is a corresponding
KL divergence term in the objective given by
[Equation]
As described later, expression 1 can be easily decomposed to show an explicit dependence on the variance of the parameterizing functions µ ( z i ) , σ ( z i ) of the intermediate Gaussian distribution.
We further show the KL divergence term to be closely related to the harmonics of the parameterizing function.
For complex parameterizing functions the KL divergence term has large high frequency components (and thus high variance) which leads to unstable training causing posterior collapse.
Building on this, we suggest a method for training the simplest hierarchical extension of VAE that avoids the problem of posterior collapse without introducing further architectural complexity [13, 21].
Given a hierarchical variational autoencoder, our training method incorporates a smoothing parameter (we denote this by ρ ) in the neural network functions used to parameterize the intermediate latent distributions.
The smoothing is done such that expected values are preserved, the higher frequencies are attenuated and the variance is reduced.
Next, the gradients computed with the smooth functions are used to train the original hierarchical variational autoencoder.
For the construction of the smoothing transformations for VAEs with Gaussian latent spaces we make use of ideas from the analysis of Gaussian spaces.
We analyze the stochastic functions in vanilla hierarchical VAEs as Hermite expansions on Gaussian spaces [9].
The Ornstein-Uhlenbeck (OU) semigroup from Gaussian analysis is a set of operators that we show to smoothly interpolate between a random variable and its expectation.
The OU semigroup provides the appropriate set of smoothing operators which enable us to control variance and avoid posterior collapse.
We further show that by smoothing the intermediate parameterizing functions µ ( z ) , σ ( z ) in the proposed manner, the KL divergence of the top layer sees a sudden sharp drop toward zero as the amount of smoothing is decreased.
This behaviour is retained when we evaluate the KL divergence on the original unsmoothed variational autoencoder model.
This behaviour is reminiscent of phase transitions from statistical mechanics and we adopt the same terminology to describe the phenomenon.
Our experiments suggest that the phenomenon is general across datasets and commonly used architectures.
Furthermore, the critical value of the smoothing parameter ρ at which the transition occurs is ﬁxed for a given model conﬁguration and varies with stochastic depth and width.
We make the following contributions.
First, we establish a connection between higher harmonics, variance, posterior collapse and phase transitions in hierarchical VAEs.
Second, we show that by using the Ornstein-Uhlenbeck semigroup of operators on the generative stochastic functions in
VAEs we reduce higher frequencies and consequently variance to mitigate posterior collpase.
We corroborate our ﬁndings experimentally and further obtain in CIFAR-10 likelihoods competitive with more complex architectural solutions alongside a reduction in model size.
We refer to the proposed family of models as Hermite variational autoencoders (HVAE).
H ERMITE V ARIATIONAL A UTOENCODERS 2. A NALYSIS ON G AUSSIAN SPACES
The analysis of Gaussian spaces studies functions of Gaussian random variables.
These are realvalued functions deﬁned on R n endowed with the Gaussian measure.
Many functions employed in machine learning are instances of such functions: decoders for variational autoencoders, as is the case in this work, and generators for generative adversarial networks being two examples.
By way of summary, the main facts we use from this ﬁeld are that a function on a Gaussian space can be expanded in an orthonormal basis, where the basis functions are the Hermite polynomials.
This orthonormal expansion is akin to a Fourier transform in this space.
The second fact is that the coefﬁcients of such an expansion can be modiﬁed in a way to reduce the variance of the expanded function by applying an operator from the Ornstein-Uhlenbeck semigroup of operators.
Next, we give a brief introduction.
For further details on Gaussian analysis we refer to [9].
Gaussian Spaces: Let L 2 ( R n , γ ) be the space of square integrable functions, f : R n → R , with the
Gaussian measure γ ( z ) =
(cid:81)
i N ( z i | 0 , 1) .
Given functions f, g in this space, the inner product is given by (cid:104) f, g (cid:105) =
E γ ( z ) [ f ( z ) g
( z )] .
Basis functions for L 2 ( R , γ ) :
Taking the space of univariate functions L 2 ( R , γ ) , it is known that the polynomial functions φ i ( z ) =
z i are a basis for this space.
By a process of orthonormalization we obtain the normalized Hermite polynomial basis for this space.
The ﬁrst few Hermite polynomials are the following:
[Equation]
Basis functions for L 2 ( R n , γ ) : Letting α ∈ N n be a multi-index, the basis functions for L 2 ( R n , γ ) are obtained by multiplying the univariate basis functions across dimension, h α ( z )
=
(cid:81)
i h
α
i ( z i ) .
Hermite expansion:
A function in L 2 ( R n , γ ) can be expressed as f = (cid:80) α ∈ N n ˆ f ( α ) h α , where ˆ f ( α ) are the Hermite coefﬁcients of f and are computed as ˆ f ( α )
= (cid:104) f, h α (cid:105) =
E γ ( z ) [ f ( z ) h
α ( z )] .
Plancherel’s theorem is the following relation between the norm of f and ˆ f which follows from orthnormality of the basis functions.
[Equation]
Ornstein-Uhlenbeck (OU) Semigroup:
Given a parameter ρ ∈ [0 , 1] and a Gaussian variable z , we construct a correlated variable z (cid:48) as z (cid:48)
=
ρz
+ (cid:112) 1 − ρ 2 z ω , where z ω ∼ N (0 , 1) is a random standard Gaussian sample.
The OU semigroup is a set of operators, denoted U ρ and parameterized by ρ ∈ [0 , 1] .
The action of U ρ on f at z is to average the function values on correlated z (cid:48)
s around z ,
[Equation]
The action of the U ρ operators on the Hermite expansion of function f ( z ) is to decay Hermite coefﬁcients according to their degree, U
ρ f ( z ) = (cid:80) α ∈ N n ρ |
[Equation]
| ˆ f ( α ) h α .
where | α | = (cid:80) i α i .
If z is reparameterized as z = σ(cid:15) 1 + µ , the correlated OU sample is given by z (cid:48)
=
σ ( ρ(cid:15)
+ (cid:112) 1 − ρ 2 (cid:15) 2 ) + µ , where (cid:15) 1 , (cid:15) 2 are standard Gaussian variables.
This can also be expressed in terms of z as
[Equation]
2. H ERMITE EXPANSIONS FOR VAE S
Our proposed method is a new training procedure for the vanilla hierarchical variational autoencoder that builds upon Hermite expansions of Gaussian functions and properties of the OU semigroup.
In the context of hierarchical variational autoencoders, the Gaussian function f is the generative model µ i ( z i +1 ) and σ i ( z i +1 ) that receives as inputs the latent variable
z
i +1 to return the Gaussian latent variable of the next layer, z i ∼ N ( µ i ( z i +1 ) , σ i ( z i +1 )) .
We make use of the following properties of the OU semigroup to construct Gaussian functions of lower variance.
The ﬁrst property we employ is that the OU semigroup of operators interpolates between a random variable ( ρ = 1 ) and its expectation ( ρ = 0 ), where the parameter ρ controls the extent of the interpolation.
Proposition 1 The operators U ρ retain the expected value of the operated function, E [ f ] =
E [ U ρ f ] .
Proposition 2 The operators U ρ interpolate between a random variable and its expectation.
In particular, as ρ → 1 , U ρ f = f .
and as ρ → 0 , U ρ f = E
[ f ]
The second property we exploit is that the new random variable U ρ f ( z ) has lower variance compared with original variable f ( z ) and is in general a smoother function than f ( z ) .
The smoothing properties of the operator U ρ can be understood by examining the Hermite expansion of U ρ f .
First we note that we can express the expectation and variance of a function f in terms of its Hermite coefﬁcients, speciﬁcally E [ f ] =
ˆ f (0) and Var( f )
=
E [( f − E [ f ]) 2 ] =
E [( f − ˆ f (0)) 2 ] = (cid:80) α : | α | >
ˆ f ( α ) 2 , which follows from Plancherel’s theorem (equation 2).
Replacing f with U ρ f and using the Hermite expansion of U
ρ f from equation 3, the mean remains the same, E
[ U ρ f ] = ρ
ˆ f (0) =
ˆ f (0) , and variance reduces like
[Equation]
The last equation indicates that the contribution to the variance by ˆ f ( α ) decays by an amount ρ 2 | α | when ρ ∈ (0 , 1) .
This, in turn, leads to a decrease in variance.
Algorithm.
In essence, Hermite variational autoencoders are similar to variational autoencoders, save for applying the OU semigroup to the latent distributions p ( z i
| z i +1 )
that comprise the generator to compute gradients during training only.
Speciﬁcally, we apply these operators to the functions parameterizing the mean and variance of the latent Gaussian distributions.
For each distribution p ( z i | z i +1 )
we substitute N ( z i | µ i ( z i +1 ) , σ i ( z i +1 )) with N ( z i | U ρ µ
i
z i +1 ) ,
U ρ σ i ( z i +1 )) .
The new functions result in latent distributions with parameters that have lower variance but the same expected value relative to the conditional input latent distribution.
In an alternative parameterization we apply the OU semigroup to the ratio of the mean and variance functions: U ρ µ i σ i ( z i +1 )
(see next section for a justiﬁcation of this).
The OU semigroup operators can also be applied on approximate posterior functions, but we observe little beneﬁt.
In practice, we compute U ρ µ
i
z i +1 )
and U ρ σ i ( z i +1 ) by Monte Carlo averaging.
As for a function f , U ρ f = E z (cid:48) | z [ f ( z (cid:48) )] , where z (cid:48) are the correlated samples, we estimate the expectation by Monte Carlo averaging over z (cid:48) .
Experiments show that 5 to 10 samples sufﬁce.
It is important to emphasize that the substitution of the lower variance functions for parameterizing the distributions is only done when computing gradients during training .
All evaluations, training or test, are still done on the original hierarchical variational autoencoder model.
Thus, the new training procedure has an additional computational cost only for the intermediate distributions in the generator, proportional to the number of correlated samples during training.
KL D IVERGENCE A NALYSIS
In this section we justify our approach as affecting a bias-variance trade-off in the KL divergence terms of the hierarchical VAE objective.
The bias-variance trade-off arises from the fact
that the
KL divergence term can be written so that it contains the variance of the functions µ ( z ) , σ ( z ) parameterizing the intermediate distributions.
This variance is related to spectral complexity and reducing spectral complexity leads to a reduction in variance.
Consider the following expression from the ELBO as part of the KL divergence term.
[Equation]
where p ( z 1 | z 2 ) =
N ( z 1 | µ p ( z 2 ) , σ p ) and q ( z 1 | x ) =
N ( z 1 | µ q ( x ) , σ q ( x )) .
For the purpose of analysis we assume that the standard deviation for p is ﬁxed and independent of z 2 , σ ( z 2 ) =
σ p .
Furthermore for ease of analysis and without loss of generality we think of z 1 as a scalar.
The general result for multivariate z 1 follows from summing for all dimensions.
For Gaussian p we write equation 6 as E q ( z 1 ,z 2 | x ) [ − log (cid:113)
πσ 2 p − 12 σ 2 p ( z 1 − µ p ( z 2 )) 2 ] .
From the inner term E q ( z 1 ,z 2 | x ) [( z 1 − µ p ( z 2 )) 2 ] we focus on the quadratic µ p ( z 2 ) 2 which is expanded as
[Equation]
By Plancherel’s theorem (equation 2) we have
[Equation]
This shows that for σ p independent of z 2 the KL divergence term in the ELBO contains the variance of the parameterizing function µ p ( z 2 ) .
In our proposal we replace µ p ( z 2 ) by U ρ [ µ p ( z 2 )] and the right side of equation 8 becomes
[Equation]
since E [ U ρ f ] = E [ f ] .
The new variance is of order O ( ρ 2 ) .
Comparing this objective with the original
VAE objective, for the second term we get a bias proportional to the difference of the variance
[Equation]
Comparing equations 8 and 9 we see the bias to be O (1 − ρ 2 ) .
The analysis above assumed σ p independent of z 2 .
For σ p dependent on z 2 we can make a similar argument for the variance of ( z 1 − µ ( z 2 )) /σ
( z 2 ) or for the ratio µ ( z 2 ) /σ
( z 2 ) by expanding the square and considering the terms separately.
R ELATED W ORK
To increase the stochastic depth of VAEs [10, 19], [21] proposes the Ladder VAEs.
With an architecture that shares a top-down dependency between the encoder and the decoder, Ladder VAEs allow for interactions between the bottom-up and top-down signals and enable training with several layers deep VAEs.
Extending Ladder VAEs, [13] proposed the bidirectional-inference VAEs, adding skip connections in the generative model and a bidirectional stochastic path in the inference model.
[1] observed that the latent distribution collapses to the prior in deep stochastic hierarchies – a phenomenon now called posterior collapse .
Posterior collapse appears in different contexts including images or text, and is strongly associated with the presence of powerful decoders, be it LSTMs [1] for text or strong autoregressive models for images [16], where although the model may produce good reconstructions, it does not learn a meaningful generative distribution.
A prevalent hypothesis behind posterior collapse is that when the decoder is strong enough to generate very low cross entropy losses, the optimization may ﬁnd it easier to simply set the KL divergence term to 0 to minimize the ELBO [1].
Making an association with probabilistic PCA, [12] hypothesize that posterior collapse is caused by local optima in the optimization landscape due to high variance, even without powerful decoders.
High variance was identiﬁed as a potential culprit also by [21] for posterior collapse.
Given the breadth of the problem, many tried to address posterior collapse.
[1, 8, 21, 14] anneal the KL divergence between the approximate posterior to the prior from 0 to 1 .
Unfortunately, this solution does not optimize the original ELBO formulation and is shown [27, 3] to cause instabilities, especially with large datasets and complex decoders.
[11] introduce the concept of free bits ignoring the gradient if not signiﬁcant enough.
[17] proposed δ -VAEs, which constrain the latent distribution to have a minimum distance to the prior.
[7] monitor the mutual information between the latent and the observed variable to aggressively optimize the inference model before every model update.
[2] suggest that using a tighter multi-sample ELBO can help alleviate collapse to some extent.
While many [1, 21] suggested a connection between posterior collapse and variance reduction, no real solutions using variance reduction has been proposed.
One reason may be the low variance the reparameterization trick [10, 19] already offers.
Empirically, while the reparametrization is successful with producing low variance forward and backward propagations in shallow models, it has not been enough for deeper and wider ones.
Another reason suggested by the approach of this paper is that variance appears as a side effect of spectral complexity.
To reduce the variance in reparameterization gradients, [20] suggest removing a mean zero score function term from the total derivative while [15] build a control variate using a linear approximation for variance reduction.
[2] propose importance weighted gradients and [24] extend [15] to multiple samples to obtain an estimator with improved signal-to-noise ratio.
Other approaches to increase the power of VAE models include normalizing ﬂows [18], better posterior or prior distributions [22], adding autoregressive components [6] or a combination of both [11].
Here, we theoretically argue and empirically validate that damping higher frequency components, thus lowering variance, allows for training deeper latent hierarchies while addressing posterior collapse.
We rely on tools from the ﬁeld of analysis on Gaussian spaces, amenable to the analysis of stochastic processes [9].
E XPERIMENTS
In the following experiments we ﬁrst test our method’s ability to prevent posterior collapse and compare with other methods designed for the same end.
We then present our observation that the top layer KL divergence undergoes a phase transition as we decrease the amount of smoothing.
Finally, we compare performance on commonly used benchmarks against other methods from the literature.
[Figure]
We validate Hermite variational autoencoders on binary MNIST, OMNIGLOT and CIFAR-10 with various ResNet and MLP architectures.
Validation ELBOs are evaluated using importance-weighted samples [2] with L 100 and L 5000 denoting evaluation with 100 and 5000 samples.
5. I NVESTIGATING POSTERIOR COLLAPSE
We test Hermite VAEs for their ability to prevent posterior collapse on basic MLP network architectures.
We compare on static and dynamically binarized MNIST against various standard methods for mitigating posterior collapse.
For dynamic MNIST we choose two models: the ﬁrst is astochastic layer model with 64,32,16,8 latent variables.
The deterministic layers have two layers of 512,256,128,64 units respectively, going from bottom to top.
The second model has 4 stochastic layers VAE with 40 units per stochastic layer and 2 layers of 200 units per stochastic layer .
For static MNIST we only use the second model described above.
All models have a simple stacked architecture with no skip connections.
First we compare with a standard VAE on static MNIST.
We trained the VAE with the standard training method and our method with ρ ∈ { 0 . 9 , 0 . 8 } .
We show the validation curves and the KL divergence of the top stochastic layer KL ( q ( z 4 | z 3 ) ||N (0 , 1) in ﬁgure 1.
The standard training collapses the posterior immediately while our method avoids posterior collapse and yields better validation ELBO.
Next we compare against other methods designed to mitigate posterior collapse including KL annealing, free bits [11] and importance weighted objectives [2].
For KL annealing the annealing coefﬁcient is set to 0 for the ﬁrst 10,000 steps and is linearly annealed to 1 over the next 500, steps.
For free bits, we apply the same free bits value to each stochastic layer.
The free bits values are chosen from { 0 . 5 , 1 . 0 , 2 . 0 , 3 . 0 } .
We ﬁnd that training slows down considerably when using free bits.
Values of free bits of 4.0 or more caused training to become unstable.
For this comparison we use the 40-40-40-40 4-layer architecture used for all methods for static
MNIST and both the 40-40-40-40 and 64-32-16-8 architectures for dynamic MNIST.
We show the results in tables 2 and 1, where we include the total KL divergence, top layer KL divergence as well as the number of active units in each of the 4 layers.
Compared to the baseline methods, our method is able to maintain signiﬁcant activity across the layers and a considerably higher KL divergence in the top layer.
For dynamic MNIST the 64,32,16,8 model has successively smaller networks higher in the hierarchy and the other methods show better activity in the latent units than the 40-40-40-40 model.
This suggests that training dynamics and the architecture affect the extent of posterior collapse with standard methods and have less of an effect with our proposal.
Here we also ﬁnd that employing other posterior collapse mitigation techniques can sometimes help with HVAE as well.
As shown in table 2, KL annealing with the 4 layer HVAE allows it to reach higher validation ELBO.
However, we did not ﬁnd KL annealing to be advantageous when used with
HVAE on more complex architectures and in particular we do not see improvement in CIFARperformance.
[Table]
[Figure]
[Table] [Table]
[Figure]
The plots show the KL divergence values for the original unsmoothed model after training on the smoothed version.
The left ﬁgure shows this for same 4 stochastic (40 units per layer) layer MLP model on MNIST and
OMNIGLOT.
It can be seen that the critical value where the KL divergence drops toward 0 is the same for both datasets on this model.
The next two ﬁgures show phase transition for a 3 stochastic layer (200 units per layer)
MLP model on OMNIGLOT (middle) and a convolutional model (8x7x latent dimension) on MNIST (right) showing that the phase transition becomes especially pronounced with the width of the latent layer.
5.1. P HASE T RANSITIONS
[Table]
We have given some evidence to show that attenuating the higher frequency components of parameterizing functions by smoothing, thus also reducing variance, is a justiﬁable mitigation against posterior collapse.
Here we show that as the amount of smoothing is reduced (as ρ approaches 1) to recover the original
VAE gradients, the top level KL divergence shows a sudden decline at a critical value of the smoothing parameter.
The sharpness of this decline depends on the model conﬁguration (stochastic layers, latent dimension).
On the other hand, our experiments suggest that the sharpness of the decline is independent of dataset.
An example of this phenomenon can be seen in ﬁgure 2 and the appendix.
Here we show the top layer KL divergence after training for 100k steps with varying ρ for 3 different architectures.
We see that the phase transition becomes more pronounced with greater stochastic dimension.
Figure 3 further shows how HVAE can prevent collapse across a spectrum of varying stochastic depth and width while maintaining good validation performance.
5. B ENCHMARK C OMPARISONS 5.2. MNIST & OMNIGLOT
MLPs.
For static MNIST we report test ELBO for MLPs with 4 stochastic layers with 40 units each.
Before each stochastic layer we have two deterministic layers of 200 tanh units.
We show the results in table 4 comparing with others that improve VAEs: IWAE [2], VAE with normalizing ﬂow [18] and VampPrior [23].
The dynamic MNIST results are in table 5.
The 5-layer HVAE model in this instance has latent dimensions 64, 32, 16, 8, 4 with two layers of 512, 256, 128, 64, 32 units in the respective stochastic layers.
[Table] [Table]
Residual ConvNets.
Next, we experiment with a more complex ResNet architecture on MNIST and OMNIGLOT with up to 4 stochastic layers and up to
ResNet blocks between stochastic layers ( 14 × 14 features maps).
The networks have the same hierarchical structure as the MLP VAEs with convolutional latent layers.
We do not employ stochastic skip connections between blocks.
We improve the MLP scores reaching -96.08 validation
ELBO on OMNIGLOT, compared to -97.65 and -97.
for VAE(L=2) and VampPrior (L=2).
The detailed results are in the appendix.
In both experiments we obtain competitive scores despite the simple architecture.
5.2. CIFAR-
We experiment with ResNets of up to 6 stochastic layers, interwined with deterministic layers comprising 6 ResNet blocks.
We used 100 feature maps for all deterministic layers.
The stochastic layers have 8 feature maps of width 16 × 16 .
We also experiment with skip connections between stochastic layers.
We report results in table 3.
We obtain an ELBO of 3.5 bpd without skip connections.
After adding skip connections, we improve to 3.
bpd on the 3-layer architecture.
This corresponds to an ELBO of 3.39 when evaluated withimportance samples.
This result is on par with the 15-layer Ladder VAE (LVAE) [13], and comparable to the 15-layer LVAE+ [13] architecture that adds skip connections to LVAE.
Note also that the improvement comes at a signiﬁcation reduction of parameters.
Compared to Ladder
VAE the 3 layer Hermite VAE uses about 7 times fewer parameters.
C ONCLUSION
Training variational autoencoders with large hierarchies of stochastic layers has proven to be difﬁcult.
An often shared observation when moving to more complex variational autoencoders is posterior collapse where a subset of the latent variables falls back to the prior distribution.
For a solution we turn to the ﬁeld of analysis of Gaussian functions and analyze intermediate VAE functions as Hermite expansions.
We argue that the Ornstein-Uhlenbeck semigroup can be used to reduce variance and empirically show that its parameterizing variable can be used to precisely control phase transitions.
We validate the analysis and solution on three datasets, MNIST, OMNIGLOT,
CIFAR-10, where we are able to avoid posterior collapse and obtain performance that is competitive with more complex architectural solutions.