Langevin Autoencoders for Learning Deep Latent Variable Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Markov chain Monte Carlo (MCMC), such as Langevin dynamics, is valid forapproximating intractable distributions.
However, its usage is limited in the contextof deep latent variable models owing to costly datapoint-wise sampling iterationsand slow convergence.
This paper proposes the amortized Langevin dynamics(ALD), wherein datapoint-wise MCMC iterations are entirely replaced with updatesof an encoder that maps observations into latent variables.
This amortizationenables efﬁcient posterior sampling without datapoint-wise iterations.
Despiteits efﬁciency, we prove that ALD is valid as a MCMC algorithm, whose Markovchain has the target posterior as a stationary distribution under mild assumptions.
Based on the ALD, we also present a new deep latent variable model namedthe Langevin autoencoder (LAE).
Interestingly, the LAE can be implemented byslightly modifying the traditional autoencoder.
Using multiple synthetic datasets,we ﬁrst validate that ALD can properly obtain samples from target posteriors.
Wealso evaluate the LAE on the image generation task, and show that our LAE canoutperform existing methods based on variational inference, such as the variationalautoencoder, and other MCMC-based methods in terms of the test likelihood.
Introduction
Variational inference (VI) and Markov chain Monte Carlo (MCMC) are two practical tools toapproximate intractable distributions.
Recently, VI has been dominantly used in deep latent variablemodels (DLVMs) to approximate the posterior distribution over the latent variable z given theobservation x , i.e., p ( z | x ) .
At the core of the success of VI is the invention of amortized variationalinference (AVI)
[Kingma and Welling, 2013, Rezende et al., 2014], which replaces optimization ofdatapoint-wise variational parameters with an encoder that predicts latent variables from observations.
The framework of learning DLVMs based on AVI is called the variational autoencoder (VAE), whichis widely used in applications [Kingma et al., 2014, An and Cho, 2015, Zhang et al., 2016, Eslamiet al., 2018, Kumar et al., 2018].
An important advantage of AVI over traditional VI is that theoptimized encoder can be used to infer a latent representation even for new data in test time, expectingits generalization ability.
On the other hand, the approximation power of AVI (or VI itself) is limitedbecause it relies on tractable distributions for complex posterior approximations of DLVMs as shownin Figure 1 (a).
Although there have been attempts to improve their ﬂexibility (e.g., normalizingﬂows [Rezende and Mohamed, 2015, Kingma et al., 2016, Van Den Berg et al., 2018, Huang et al.,2018]), such methods typically have architectural constraints (e.g., invertibility in normalizing ﬂows).
Compared to VI, MCMC (e.g., Langevin dynamics) can approximate complex distributions byrepeating sampling from a Markov chain that has the target distribution as its stationary distribution[Liu and Liu, 2001, Robert et al., 2004, Gilks et al., 1995, Geyer, 1992].
However, despite itshigh approximation ability, MCMC has received relatively little attention in learning DLVMs.
It isbecause MCMC methods take a long time to converge, making it difﬁcult to be used in the training of
[Figure]
DLVMs.
When learning DLVMs with MCMC, we need to run time-consuming MCMC iterations forsampling from each posterior per data point, i.e., p (cid:0) z | x ( i ) (cid:1) ( i = 1 , . .
. , n ), where n is the numberof mini-batch data, as shown in Figure 1 (b).
Furthermore, we need to re-run the sampling procedurewhen we obtain new data at test time.
As in VI, there have been some attempts to introduce the concept of amortized inference to MCMC.
For example, Hoffman [2017] initializes datapoint-wise MCMC using an encoder that predicts latentvariables from observations as shown in Figure 1 (c).
However, as they use encoders only for theinitialization of MCMC, these methods still rely on datapoint-wise sampling iterations.
Not only is ittime-consuming, but implementations of such partially amortized methods also tend to be complicatedcompared to the simplicity of AVI.
To make MCMC more suitable for the inference of DLVMs, amore sophisticated framework of amortization is needed.
This paper proposes amortized Langevin dynamics (ALD), which entirely replaces datapoint-wise
MCMC iterations with updates of an encoder that maps the observation into the latent variable asshown in Figure 1 (d).
Since the latent variable depends on the encoder, the updates of the encodercan be regarded as implicit updates of latent variables.
By replacing MCMC on the latent spacewith one on the encoder’s parameter space, we can beneﬁt from the encoder’s generalization ability.
For example, after running a Markov chain of the encoder for some mini-batch data, it is expectedthat the encoder can map data into high density area in the latent space; hence we can acceleratethe convergence of MCMC for other data by initializing it with the encoder.
Moreover, despite thesimplicity, we can theoretically guarantee that ALD has the true posterior as a stationary distributionunder mild assumptions, which is a critical requirement for valid MCMC algorithms.
Using our ALD for sampling from the latent posterior, we derive a novel framework of learning
DLVMs, which we refer to as the Langevin autoencoder (LAE).
Interestingly, the learning algorithmof LAEs can be regarded as a small modiﬁcation of traditional autoencoders [Hinton and Salakhutdinov, 2006].
In our experiments, we ﬁrst show that ALD can properly obtain samples from targetdistributions using toy datasets.
Subsequently, we perform numerical experiments of the imagegeneration task using the MNIST, SVHN, CIFAR-10, and CelebA datasets.
We demonstrate that our
LAE can outperform existing learning methods based on variational inference, such as the VAE, andexisting MCMC-based methods in terms of test likelihood.
Preliminaries
2.1 Problem Deﬁnition
Consider a probabilistic model with the observation x , the continuous latent variable z , and the modelparameter θ as follows:
[Equation]
To learn this latent variable model (LVM) via maximum likelihood in a gradient-based manner, weneed to calculate the expectation over the posterior distribution p ( z | x ; θ ) as follows:
[Equation]
where ˆ p data is the empirical distribution deﬁne by the training set, and x (1) , . .
, x ( n ) are mini-batchdata uniformly drawn from ˆ p data .
However, this expectation cannot be calculated in a closed-formbecause the posterior is intractable.
In this paper, we consider to use Monte Carlo approximation byobtaining samples from the posterior per data point.
2.2 Langevin Dynamics
Langevin dynamics (LD)
[Neal, 2011] is a sampling algorithm based on the following Langevinequation:
[Equation] [Equation]
where U is a Lipschitz continuous potential function that satisﬁes an appropriate growth condition, βis an inverse temperature parameter, and B is a Brownian motion.
This stochastic differential equation(SDE) has p β
( z | x ; θ ) ∝ exp ( − βU ( x , z ; θ )) as its stationary distribution.
We set β = 1 anddeﬁne the potential as follows to obtain the target posterior p ( z | x ; θ ) as its stationary distribution:
[Equation]
We can obtain samples from the posterior by simulating the SDE of Eq.
(3) using the Euler–Maruyamamethod [Kloeden and Platen, 2013] as follows:
[Equation]
where η is a step size for discretization.
The initial value z 0 is typically sampled from the prior p ( z ) .
When the step size is sufﬁciently small, the samples asymptotically move to the target posteriorby repeating this sampling iteration.
To remove the discretization error, an additional MetropolisHastings (MH) rejection step is often used.
In a MH step, we ﬁrst calculate the acceptance rate α t asfollows:
[Equation]
The sample z t +1 is accepted with probability α t , and rejected with probability 1 − α t .
If the sampleis rejected, we set z t +1 =
z t .
LD can be applied to any posterior inference problems for continuouslatent variables, provided the potential energy is differentiable on the latent space.
To obtain theposterior samples for all mini-batch data x (1) , . .
x ( n ) , we should perform iterations of Eq.
(5) perdata point, as shown in Figure 1 (b).
After obtaining the samples, the gradient in Eq.
(2) is approximated using the samples as follows:
[Equation]
The time averaged gradient in Eq.
(8) is sometimes substituted for the one calculated with the ﬁnalsample as follows:
[Equation]
However, in these traditional approaches, we need to run MCMC iterations from random initializationevery time after updating the model parameter θ .
In addition, we also need to run it from stratch toperform inference for new data in test time.
This inefﬁciency hinders the practical use of MCMC forlearning DLVMs.
A naive approach to mitigate this problem is to initialize MCMC with a persistentsample [Tieleman, 2008, Han et al., 2017] that is the ﬁnal value of the previous Markov chain.
However, this approach is also inefﬁcient especially when the training set is large, because we needto store persistent samples for all training examples.
[Equation]
[Table]
all n datapoints
[Table]
repeat
[Equation]
[Table]
[Equation]
[Table]
[Equation] Add samples. until convergence of parameters
[Table] [Table]
To alleviate the inefﬁciency of LD for LVMs, Hoffman [2017] has proposed to use a stochasticencoder q ( z | x ) to initialize the datapoint-wise MCMC as shown in Figure 1 (c).
The encoder istypically deﬁned as a Gaussian distribution as in the VAE:
[Equation]
10) where µ and σ 2 are mappings from the observation space into the latent space.
In the training, LDiterations of Eq.
(5) are initialized using a sample from the distribution of Eq.
(10), and the modelparameter is updated using the stochastic gradient in Eq.
Along with it, the encoder is trained viamaximization of the evidence lower bound as in VAEs:
[Equation]
Although initializing LD with the encoder can speed up the convergence, this method still relieson datapoint-wise MCMC iterations.
Moreover, the encoder has to be trained using the differentobjective, which also makes its implementation complicated.
In Section 3, we demonstrate a methodthat entirely remove the datapoint-wise iterations by amortization.
Method
3.1 Amortized Langevin Dynamics
As an alternative to the direct simulation of latent dynamics of Eq.
(3), we deﬁne a deterministicencoder f
z | x , which maps the observation into the latent variable, and consider an SDE over itsparameter φ as follows: [Equation]
[Equation]
Because the function f z | x outputs the latent variable, the stochastic dynamics on the parameterspace induces dynamics on the latent space.
The main idea of our amortized Langevin dynamics(ALD) is to regard the transition on this induced dynamics as a sampling procedure for the posteriordistributions, as shown in Figure 1 (d).
We can use the Euler–Maruyama method to simulate Eq.
(12)with discretization:
[Equation]
[Equation] [Equation]
As in the traditional LD, the discretization error can be removed by adding a MH rejection step,calculating the acceptance rate as follows:
[Equation]
Through the iterations, the posterior sampling is implicitly performed by collecting outputs of theencoder for each data point as described in Algorithm 1.
Note that Z ( i ) denotes a set of samples
ofthe posterior for the i -th data (i.e., p (cid:0) z | x ( i ) (cid:1) ) obtained using ALD.
If this implicit update iteration has the posterior as its stationary distribution, this sampling procedureis valid as a MCMC algorithm.
To obtain the stationary distribution over the latent variables, we ﬁrstderive the stationary distribution over the parameter φ of Eq.
(12), then transform it into the latentspace by considering the change of random variable by the encoder f
z | x .
Based on this strategy, wederive the following theorem:
Theorem 1.
Let q ( Z | X ) :
= (cid:81) ni
=1 q (cid:0)
z ( i ) | x ( i ) (cid:1) be a stationary distribution over the latentvariables that is induced by Eq.
When the mapping f z | x meets the following conditions, thestationary distribution satisﬁes q ( Z | X ) ∝ exp (cid:0) − (cid:80) ni
=1 U (cid:0) x ( i ) , z ( i ) (cid:1)(cid:1) .1.
The mapping has the form of f z | x ( x ; Φ ) =
Φ g ( x ) , where Φ is a d z × d matrix, g isa mapping from R d x to R d , and d x , d z and d are the dimensionalities of x , z and g ( x )respectively.2.
The rank of G is n , where G is a matrix with g (cid:0)
x
( i ) (cid:1) in row
G i, : .
See the appendix for the solid proof.
Theorem 1 suggests that samples obtained by ALD asymptotically converge to the true posterior when we construct the encoder f
z | x with an appropriateform.
Practically, we can implement such a function using a neural network whose parameters areﬁxed except for the last linear layer.
In this implementation, the last linear layer takes a role of theparameter Φ , and the preceding feature extractor takes a role of the function g .
To meet the second condition, the dimensionality of the last linear layer should be larger than thebatch size n .
It is relatively easy to meet this condition because the batch size is about 1,000 at mostin practice.
3.2 Remarks
Remark 1: ALD completely removes datapoint-wise iterations .
Because ALD treats the encoder’s outputs themselves as posterior samples, we no longer need to runtime-consuming iterations of datapoint-wise MCMC, whereas existing methods, such as Hoffman[2017] only use the encoder for initialization of datapoint-wise MCMC.
Remark 2: ALD is valid as a MCMC algorithm .
Although the sampling procedure of ALD is quite simple, ALD has the true posterior as its stationarydistribution under mild assumptions, which guarantee that ALD is valid as an MCMC algorithm.
Basically, we can meet the assumptions by using a sufﬁciently wide neural network for the encoder.
In addition, it is worth mentioning that the traditional LD can be seen as a special case of ALDwhere g (cid:0) x
( i ) (cid:1) =
one-hot ( i ) .
In that case, z ( i ) = f z | x (cid:0) x ( i ) ; Φ (cid:1) corresponds to the i -th column ofΦ , which is equivalent to running MCMC on the latent space independently for each data point. Remark 3: The encoder may accelerate the convergence of MCMC .
After running ALD iterations of the encoder for some mini-batch data, it is expected that the encodercan map data into high density area in the latent space.
Therefore, the encoder can accelerate theconvergence of MCMC for other data by initializing it with the encoder.
This characteristics is usefulnot only in the training of DLVMs to efﬁciently estimate the gradient of the model parameters butalso at test time to infer the latent representation for new test data.
3.3 Langevin Autoencoder
Based on ALD, we propose a novel framework of learning deep latent variable models called the
Langevin autoencoder (LAE).
Algorithm 2 is a summary of the LAE’s training procedure.
First,we prepare an encoder deﬁned by f z | x ( x ; Φ , ψ ) :
= Φ g ( x ; ψ ) .
The feature extractor g is typicallyimplemented using a deep neural network.
Before each update of the model parameter θ , theencoder’s ﬁnal linear layer Φ is updated using ALD for T times, and the gradient of θ in Eq.
(8) iscalculated using the time average of V t .
Along with the update of the model parameter, the encoder’s
1: θ , Φ , ψ ← Initialize parameters 2: repeat 3: x (1) , . . .
, x ( n ) ∼ ˆ p ( x ) (cid:46) Sample a minibatch of n examples from the training data.
4: for t = 0 , . . .
, T − 1 do (cid:46)
Run ALD iterations.
V t = − (cid:80) ni
=1 log p (cid:0) x
( i ) , z ( i ) =
Φ g (cid:0) x ( i [Equation] ; ψ (cid:1) ; θ (cid:1) 6: Φ (cid:48) ∼ q ( Φ (cid:48) | Φ ) :
= N ( Φ (cid:48) ; Φ − η ∇ Φ V t , 2 η I ) 7: V (cid:48) t = − (cid:80) ni =1 log p (cid:0) x
( i ) , z ( i ) = Φ (cid:48) g (cid:0) x ( i ) ; ψ (cid:1) ; θ (cid:1) 8: Φ ← Φ (cid:48) with probability min (cid:26) 1 , exp ( − V (cid:48) t ) q ( Φ | Φ (cid:48) ) exp( − V t ) q
( Φ (cid:48) | Φ ) (cid:27) .
(cid:46) MH rejection step.
9: end for 10:
V T = − (cid:80) ni
=1 log p (cid:0) x
( i ) , z ( i ) =
Φ g (cid:0) x ( i ) ; ψ (cid:1) ; θ (cid:1) 11: θ ← θ − η ∇ θ 1 T (cid:80) Tt =1 V t (cid:46) Update the decoder.
12: ψ ← ψ − η ∇ ψ 1 T (cid:80) Tt =1 V t (cid:46) Update the encoder.
13: until convergence of parameters 14: return θ , Φ , ψ feature extractor ψ is also updated using the gradient so that the encoder can map data into highdensity area of the posteriors. Although we describe the parameter update using a simple stochasticgradient ascent, it can be substituted for more sophisticated optimization methods, such as Adam [DPand Ba, 2015].
It can be seen that the algorithm is very similar to the one for traditional deterministic autoencoders[Hinton and Salakhutdinov, 2006].
In particular, when we skip the ALD iterations in lines 4 to 9and the encoder’s ﬁnal linear layer Φ is also trained via gradient ascent, the algorithm is identical tothe training of autoencoders regularized by the latent prior p ( z ) .
In that case, the encoder tends toshrink to the maximum a posteriori (MAP) estimate rather than the posterior; hence the gradient ofthe model parameter θ would be a strongly biased estimate of the marginal log-likelihood gradient.
Therefore, the additional ALD iterations can be interpreted as the modiﬁcation to reduce the biasof the traditional autoencoder by updating the encoder’s parameters along with the noise-injectedgradient.
Related Works
Amortized inference is well-investigated in the context of variational inference.
It is often referredto as amortized variational inference (AVI)
[Rezende and Mohamed, 2015, Shu et al., 2018].
Thebasic idea of AVI is to replace the optimization of the datapoint-wise variational parameters withthe optimization of shared parameters across all datapoints by introducing an encoder that predictslatent variables from observations.
The AVI is commonly used in generative models [Kingma and
Welling, 2013], semi-supervised learning [Kingma et al., 2014], anomaly detection [An and Cho,2015], machine translation [Zhang et al., 2016], and neural rendering [Eslami et al., 2018, Kumaret al., 2018].
However, in the MCMC literature, there are few works on such amortization.
Hanet al.
[2017] use traditional LD to obtain samples from posteriors for the training of deep latentvariable models.
Such Langevin-based algorithms for deep latent variable models are known asalternating back-propagation (ABP) and are widely applied in several ﬁelds [Xie et al., 2019, Zhanget al., 2020, Xing et al., 2018, Zhu et al., 2019].
However, ABP requires datapoint-wise Langeviniterations, causing slow convergence.
Moreover, when we perform inference for new data in testtime, ABP requires MCMC iterations from randomly initialized samples again.
Li et al.
[2017] and
Hoffman [2017] propose to use a VAE-like encoder to initialize MCMC, and Salimans et al.
[2015]also proposes to combine VAE-based inference and MCMC by interpreting each MCMC step as anauxiliary variable.
However, they only amortize the initialization cost in MCMC by using an encoder;hence, they still rely on datapoint-wise MCMC iterations.
Autoencoders (AEs)
[Hinton and Salakhutdinov, 2006] are a special case of LAEs, wherein the ALDiterations are omitted and a uniform distribution is used as the latent prior p ( z ) .
When a different [Equation] distribution is used as the latent prior as regularization, it is known as sparse autoencoders (SAEs)[Ng et al., 2011].
As described in the previous section, the encoder of the traditional AE tends toconverge to MAP estimates of the latent posterior.
Therefore, the gradient of the decoder’s parameteris biased as the gradient estimate of the marginal log-likelihood.
Our LAE can modify this bias by
Variational Autoencoders (VAEs) are based on AVI, wherein an encoder is deﬁned as a variationaldistribution q ( z | x ; φ ) using a neural network.
Its parameter φ is optimized by maximizing theevidence lower bound (ELBO), i.e., E [Equation] ( z [Equation]
x ; [Equation] (cid:105) .
Interestingly, there is a contrastbetween VAEs and LAEs relative to when stochastic noise is used in posterior inference.
In VAEs,noise is used to sample from the variational distribution in calculating the potential U , i.e., in theforward calculation.
However, in LAEs, noise is used for the parameter update along with the gradientcalculation ∇ φ U , i.e., in the backward calculation.
This contrast characterizes their two differentapproaches to approximate posteriors: the optimization-based approach of VAEs and the samplingbased approach of LAEs.
The advantage of LAEs over VAEs is that LAEs can ﬂexibly approximatecomplex posteriors by obtaining samples, whereas VAEs’ approximation ability is limited by choiceof variational distribution q ( z | x ; φ ) because it requires a tractable density.
Although there areseveral considerations in the improvement of the approximation ﬂexibility, these methods typicallyhave architectural constraints (e.g., invertibility and ease of Jacobian calculation in normalizing ﬂows[Rezende and Mohamed, 2015, Kingma et al., 2016, Van Den Berg et al., 2018, Huang et al., 2018,
Titsias and Ruiz, 2019]), or they incur more computational costs (e.g., MCMC sampling for thereverse conditional distribution in unbiased implicit variational inference [Titsias and Ruiz, 2019]).
Experiment
In our experiment, we ﬁrst test our ALD algorithm on toy examples to investigate its behavior, thenwe show the results of its application to the training of deep generative models for image datasets.
5.1 Toy Examples
We perform numerical simulation using toy examples to demonstrate that our ALD can properlyobtain samples from target distributions.
First, we use a LVM where the posterior density can bederived in a closed-form.
We initially generate three synthetic data x 1 , x 2 , x 3 , where each x i issampled from a bivariate Gaussian distribution as follows:
[Equation]
In this case, we can calculate the exact posterior as follows:
[Equation] [Equation]
In this experiment, we set µ z = (cid:20)
(cid:21) , Σ z = (cid:20) 1 0 0 1 (cid:21) , and Σ x = (cid:20) 0 .
8 (cid:21) .
We simulateour ALD algorithm for this setting to obtain samples from the posterior.
We use a neural network [Figure] of three fully connected layers of 128 units with ReLU activation for the encoder f
z | x ; setting thestep size to 4 × 10 − 4 , and update the parameters for 3,000 steps.
We omit the ﬁrst 1,000 samplesas burn-in steps and use the remaining 2,000 samples for qualitative evaluation.
As we described in
Section 3.1, we need to make the last linear layer of the encoder have sufﬁciently large dimensions toguarantee the convergence to the true posterior.
To empirically demonstrate this theoretical ﬁnding,we change the dimensionality of the last linear layer of the encoder from 2 ( < n ) to 128 ( (cid:29) n ) .
The results are summarized in Figure 2.
It can be observed that the sample quality is good whenthe dimensionality of the last linear layer is equal to or greater than the number of data points (i.e.,d ≥ n ).
When the dimensionality is smaller than the number of data points, the samples for somedata points shrink to a small area, while good samples are obtained for the remaining data points.
In addition to the simple conjugate Gaussian example, we experiment with a complex posterior,wherein the likelihood is deﬁned with a randomly initialized neural network.
For comparison, wealso implement the variational inference (VI), in which the posterior is approximated with a Gaussiandistribution.
Figure 3 shows a typical example, which characterizes the difference between VI and
ALD.
The mean-ﬁled VI and the full VI use Gaussians with diagonal and full covariance matricesfor variational distributions, respectively.
The advantage of our ALD over VI is the ﬂexibility ofposterior approximation.
VI methods typically approximate posteriors using variational distributions,which have tractable density functions.
Hence, their approximation power is limited by the choiceof variational distribution family, and they often fail to approximate such complex posteriors.
Inparticular, the mean-ﬁeld VI, which is widely used for learning DLVMs, cannot capture the correlationbetween dimensions due to the constraint of the variational distribution.
The full VI mitigate theinﬂexibility, but it still cannot capture the multimodality of the true posterior.
Moreover, the full VIrequires computational costs proportional to the square of the dimension of the latent variable.
Onthe other hand, ALD can capture such posteriors well.
The results in other examples are summarizedin the appendix.
5.2 Image Generation
To demonstrate the applicability of our LAE to the generative model training, we experiment on imagegeneration tasks using MNIST, SVHN, CIFAR10, and CelebA datasets.
Note that our goal here is notto provide the state-of-the-art results on image generation benchmarks but to verify the effectivenessof our ALD as a method of approximate inference in deep latent variable models.
For this aim, wecompare our LAE with some baseline methods, as shown in Table 1.
The VAE [Kingma and Welling,2013] is one of the most popular deep latent variable models, in which the posterior distribution isapproximated using the VI.
The VAE-ﬂow is an extension of VAE, in which the ﬂexibility of VI isimproved using normalizing ﬂows [Rezende and Mohamed, 2015].
In addition to VI-based methods,we use Hoffman [2017] as a baseline method based on Langevin dynamics (LD).
As described in
Section 2.2, Hoffman [2017] uses a VAE-like encoder to initialize LD, and the encoder is trained bymaximizing the evidence lower bound.
We use the same fully connected deep neural networks for themodel construction of all methods.
We set the number of ALD iterations of the LAE to 2 , i.e, T = 2in Algorithm 2.
Please refer to the appendix for more implementation details.
For evaluation, since the marginal log-likelihood for test data is not tractable, we substitute itsevidence lower bound (ELBO) for it using a proposal distribution q as follows:
[Equation]
[Table]
For the baseline methods, their stochastic encoders are used for the proposal distribution, i.e.,q ( z ) :
= q ( z | x ; φ ) .
For our LAE, we use a Gaussian distribution whose mean parameter is deﬁnedby its encoder, i.e., q ( z ) :
= N (cid:0) z ; Φ g ( x ; ψ ) , σ 2 I (cid:1) .
We set σ = 0 .
05 in the experiment.
The results are summarized in Table 1.
Note thatthe negative ELBO is shown in the table, so lowervalues indicate better results.
It can be observed thatthe LAE consistently outperforms baseline methods,demonstrating that accurate posterior approximationby ALD leads to better results in the training of
DLVMs.
On training speed, we observe that our LAE is 2.24 and 1.88 times slower than VAE and
VAE-ﬂow respectively. This is natural because VAEand VAE-ﬂow do not require MCMC steps. Hoffman[2017] and our LAE are almost identical in terms oftraining speed. We also investigate the effect of the MH rejection step and the number of ALD iterations, i.e., Tin Algorithm 2, using MNIST dataset.
Figure 4 shows a comparison of the learning curves of thenegative ELBO for MNIST’s test set.
It can be seen that the number of ALD iterations has little effecton performance as long as the number of steps is at least 2.
In addition, we observe that the MHrejection step is important to stabilize the training.
Conclusion
This paper proposed amortized Langevin dynamics (ALD), an efﬁcient MCMC method for deeplatent variable models (DLVMs).
ALD amortizes the cost of datapoint-wise iterations by using anencoder that predict the latent variable from the observation.
We showed that our ALD algorithmcould accurately approximate posteriors with both theoretical and empirical studies.
Using ALD,we derived a novel scheme of deep generative models called the Langevin autoencoder (LAE).
We demonstrated that our LAE performs better than VI-based methods, such as the variationalautoencoder, and existing LD-based methods in terms of the marginal log-likelihood for test sets.
This study will be the ﬁrst step to further work on encoder-based MCMC methods for latent variablemodels.
For instance, developing algorithms based on more sophisticated Hamiltonian Monte Carlomethods is an exciting direction of future work.
One of the limitations of MCMC-based learning algorithms is the difﬁculty in choosing the numberof MCMC iterations.
To reduce the bias of the gradient estimate, we need to run the iterations formany times, but there are few clues as to how many MCMC iterations are sufﬁcient in advance.
Recently, a method to remove the bias of MCMC with couplings is proposed by Jacob et al.
[2020],and it may help to overcome this limitation of MCMC-based learning algorithm in the future work.
Another limitation of our LAE is that there is a constraint on the structure of the encoder as describedin Theorem 1.
Although the constraint is relatively minor, it may be problematic when applying ourmethod to modern DLVMs that have a hierarchical structure in the latent variables (e.g., Vahdat and
Kautz [2020] and Child [2020]).
From a broader perspective, developing deep generative models that can synthesize realistic imagescould cause a negative impact, such as abuse of Deepfake technology.
We must consider the negativeaspects and take measures for them.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See the abstract and Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 6. (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Theorem
(b) Did you include complete proofs of all theoretical results?
[Yes] See the appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
[Yes] See theappendix.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See the appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 5.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the appendix.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]