M AXIMUM L IKELIHOOD T RAINING OF P ARAMETRIZED D IFFUSION M ODEL
Anonymous authors
Paper under double-blind review
A BSTRACT
Whereas diverse variations of diffusion models exist in image synthesis, the previous variations have not innovated on a topic of their diffusing mechanism that was always kept linear.
Meanwhile, it is intuitive that there would be more promising diffusion patterns to result in a distribution closer to the approximating data distribution.
This paper introduces such adaptive and nonlinear diffusion method for score-based diffusion models.
Unlike static and linear Variance Exploring (VE) or
Variance Preserving (VP) Stochastic Differential Equations (SDE) of the previous diffusion models, our Parametrized Diffusion Model (PDM) learns the optimal diffusion process by combining a normalizing ﬂow and a diffusion process.
Specifically, PDM utilizes the ﬂow to nonlinearly transform a data variable into a latent variable, and PDM applies the diffusion process to the transformed latent variable with the linear diffusing mechanism.
Subsequently, PDM enjoys a nonlinear and learned diffusion from the perspective of the data variable.
We train PDM with a variational proxy of the log-likelihood, and we prove that variational gap between the variational bound and the log-likelihood becomes tight when the normalizing ﬂow is optimal.
I NTRODUCTION
Diffusion models have recently achieved successes on a task of sample generations, and some researches claim the state-of-the-art performance over Generative Adversarial Networks (GAN) (Karras et al., 2019).
This success is highlighted particularly in the community of likelihood-based models, including normalizing ﬂows (Grci´c et al., 2021), autoregressive models (Parmar et al., 2018), and variational autoencoders (VAE) (Vahdat and Kautz, 2020).
Moreover, this success is noteworthy because it is achieved merely using linear diffusing mechanisms, such as VESDEs (Song and Ermon, 2020) and VPSDEs (Ho et al., 2020).
This paper expands these linear diffusing mechanisms of VE/VPSDE to a data-adaptive trainable nonlinear diffusion.
To motivate the expansion, though there are structural similarities between diffusion models and VAEs, the forward diffusion in a diffusion model has not been trained in existing literature while its counterpart, which is the encoder of VAE, is trainable.
Rather, the current diffusion models assume the linear diffusing mechanism to be ﬁxed throughout training procedure.
Because of this static nature of the diffusing mechanism, variational gap (Cremer et al., 2018) between the log-likelihood and the Evidence Lower BOund (ELBO) remains to be strictly positive unless the score perfectly estimates the data score.
This variational gap prevents the score training from being Maximum Likelihood Estimation (MLE).
This gap is a fundamental motivation to develop a trainable encoder of the diffusion model, so that the gap can be tight.
As we tighten variational gap by training a nonlinear forward diffusion, the biggest challenge comes from an intractable optimization loss.
The denoising diffusion loss requires a transition probability to be closed-form in order to achieve fast optimization, but a nonlinear diffusing mechanism, in general, has no closed-form perturbation probability.
Hence, our innovation becomes designing a diffusion model with a tractable loss while the diffusion is learnable and ﬂexible, which we achieve by merging previous diffusion models and the tractable variable transformation.
Our innovation concentrates on the theoretic and the practical aspects of a new diffusion, which is tractable, learnable, and ﬂexible.
Theoretically, we observe that a nonlinear diffusing mechanism can be transformed to a linear diffusion under an invertible transformation, and vice versa.
In practice, we implement PDM by merging a normalizing ﬂow model and a diffusion model, which constructs a tractable, learnable, and ﬂexible nonlinear diffusing mechanism.
From this construction, we prove that variational gap can be further tightened by training the forward diffusion, nonlinearly.
Also, we demonstrate the state-of-the-art performance in CelebA on Fréchet Inception Distance (FID) (Heusel et al., 2017).
We summarize our key contributions, as below.
• PDM expands the scope of a forward diffusion process from linear and ﬁxed dynamics to nonlinear and trainable dynamics.
• PDM minimizes variational gap between the log-likelihood and ELBO, lower than the gap of linear diffusion models because PDM learns a data-adaptive nonlinear diffusion.
• PDM explicitly proves in Theorem 2 that the optimal generative distribution equals to the real distribution, which was not the case for a linear diffusion model.
P RELIMINARY
A diffusion model is constructed with bidirectional stochastic processes.
The forward direction diffuses an input variable to a noise variable, and the reverse direction denoises the random noise to construct a realistic input data instance.
The diffusion model learns this reverse direction (generative process) by estimating data score.
We provide a brief summary of diffusion models, as below.
Forward Diffusion
At the beginning of model build-up, a diffusion process requires a diffusing mechanism on an input variable.
This paper assumes that the diffusion process is governed by a SDE of d x t
= f
( x t , t ) d t + G ( x t , t ) d ω t with x 0 ∼ p r , where p r is the distribution from a real-world dataset; and x t is the solution of the SDE.
Reverse Diffusion
The theory of stochastic calculus guarantees that we could create an identical diffusion process, { x t } Tt =0 , by solving an associated reverse SDE backwards in time (Anderson, 1982).
The associated reverse SDE is
[Equation]
where ω t and ¯ ω t are standard Wiener processes with time ﬂows forward and backward, respectively; and p t is a probability law of x t .
Generative Diffusion A diffusion model approximates the above SDE 1 to eventually yield an estimation on the data distribution of p r .
In SDE 1, previous literature setup that drift and diffusion terms, f and G , are determined a-priori in the forward diffusion.
However, data score, ∇ x t log p t ( x t ) , is intractable to compute, so we estimate this data score with a score network of s θ ( x t , t ) in order to mimic the reverse diffusion with our generative process.
This score network approximates the reverse diffusion by plugging the estimated score in place of data score with the below generative process:
[Equation]
The generative process starts from a prior distribution ( π ), and it constructs time-continuous random variables x θ t by solving a SDE 2 backwards in time.
The generated stochastic process is denoted by { x θ t } Tt =0 , and we omit θ in the superscript if no confusion arises.
With this generative process, we deﬁne a generative distribution, x θ 0 ∼ p θ , as the probability density of the generated random variable at time t = 0 .
Score Estimation We train the score network by a variational bound of the log-likelihood, given by
[Equation]
where L t (cid:0) { x t } Tt =0 ; θ (cid:1) =
E x 0 , x t (cid:2) (cid:107) s θ ( x t , t ) − ∇ x t log p 0 t ( x t | x 0 ) (cid:107) 22 (cid:3) up to a constant, where p 0 t ( x t | x 0 ) is a transition probability from x 0 to x t .
Here, λ is a weighting function that determines the level of contribution for each diffusion time on the overall diffusion loss, L ( { x t } Tt =0 , λ ; θ ) (Song et al., 2020).
Variational bound holds when the weighting function is the likelihood weighting ( g ) (Song et al., 2021), where G is a scalar-valued g function.
M OTIVATION OF N ONLINEAR D IFFUSING M ECHANISM
Though it has long been theoretically and empirically grounded to train the encoder part in VAE, such solid ground is not accomplished in diffusion models.
This section analyzes structural similarities between a VAE model and a diffusion model, which brings the foundation of a data-adaptive nonlinear diffusing mechanism.
3. V ARIATIONAL G AP OF VAE
Given Negative ELBO (NELBO) of Negative Log-Likelihood (NLL) in VAE as
[Equation]
we have a pair of interpretations on this NELBO.
First, when we focus on Eq. 4, NELBO 1) aids data reconstruction by optimizing E q φ ( z | x ) (cid:2) − log p θ ( x | z ) (cid:3) ; and NELBO 2) regularizes the inference distribution of the encoder to a prior distribution by D KL (cid:0) q φ ( z | x ) (cid:107) p ( z ) (cid:1) .
On the other hand, if we concentrate on Eq. 5, NELBO bounds NLL by approximating an intractable decoder posterior of p θ ( z | x ) with a tractable encoder posterior of q φ ( z | x ) .
A vanilla VAE (Kingma and Welling, 2013) assumes this approximate posterior to be a Gaussian distribution with mean and diagonal covariance estimated by amortized inference: q φ ( z | x ) = N (cid:0) z ; µ φ ( x ) , σ 2 φ ( x ) I (cid:1) .
By expanding the ﬂexibility of this approximate posterior into a variational family of general distributions, for instance, Rezende and Mohamed (2015) resulted in tighter NELBO, which leads the optimization of VAE closer to MLE, and their choice of a generalizable model was normalizing ﬂow.
To connect this NELBO to a diffusion loss, we restate NELBO in the language of a stochastic process.
Having that VAE attains a stochastic process of bivariate random variables, { x , z } , NELBO is reformulated to a KL divergence between two joint distributions modeled in bidirectional ways:
[Equation]
The inequality 6, i.e., D KL ( p r (cid:107) p θ ) ≤ D KL ( q φ ( x , z ) (cid:107) p θ ( x , z )) , is well-known by itself in the ﬁeld of information theory (Duchi, 2016) by the name of data processing inequality .
This restated bound interprets that VAE is essentially a bimodeling approach of a joint distribution on the bivariate stochastic process, { x , z } .
On the forward direction ( x → z ), a latent variable is conditioned on a data variable, and the joint distribution is modeled by q φ ( x , z )
= p r ( x ) q φ ( z | x ) .
On the reverse direction ( z → x ), a generative data variable is conditioned on the latent variable, and the joint distribution is modeled by p θ ( x , z )
= p ( z ) p θ ( x | z ) .
Under this equivalent framework, we present an analytic tool to measure the closeness of NELBO and NLL (see Table 5) by
[Equation]
and we denote this quantity as variational gap (Cremer et al., 2018).
Though it is one of central tasks in VAE to minimize this variational gap (Kingma et al., 2016), only limited works (Burda et al., 2015; Neal, 2001) have estimated variational gap due to its computational burden.
We introduce the tractable computation of variational gap in the next section on diffusion models.
3. V ARIATIONAL G AP OF D IFFUSION M ODELS
This section constructs an analogy of a diffusion model to VAE.
To begin with, we observe that the classical data processing inequality in VAE holds on a stochastic process with bivariate random variables on a ﬁnite-dimensional Euclidean space.
On the other hand, the time horizon of a diffusion
[Figure]
model is continuous, so it contains an inﬁnite number of random variables in its stochastic process as { x t } Tt =0 .
Fortunately, the ﬁnite-dimensional classical data processing inequality on the bivariate random variables is directly extendable to a stochastic process with a continuous index.
Formally, an inﬁnite-variate joint distribution is called a path measure (Oksendal, 2013), which is deﬁned as a probability law on a space of continuous functions induced by { x t } T t =0 .
Therefore, the path measure is an alternative concept of a joint distribution for a stochastic process with a continuous index.
The path measure for the forward diffusion in a diffusion model corresponds to a joint distribution for the encoder part in VAE, and similarly, the path measure for a generative diffusion corresponds to the decoder part in VAE.
If µ is the forward (inference) path measure, and if ν θ is the generative path measure; then the data processing inequality becomes
[Equation]
The real distribution of p r is a marginal distribution of µ at t = 0 , and the generative distribution of p θ becomes a marginal distribution of ν θ at t = 0 .
Under this concept, the data processing inequality means that a distributional divergence of D KL ( µ (cid:107) ν θ ) between two path measures lower bounded by a divergence of D KL ( p r (cid:107) p θ ) between two sliced path measures at t = 0 .
Previous works (Song et al., 2021; Huang et al., 2021) regard this variational bound being equivalent to a diffusion loss,
[Equation]
where G ( x t , t ) = g ( t ) is a scaler function on t , and λ ( t ) is a square of the diffusion term g 2 ( t ) .
Since the previous diffusion models optimize the diffusion loss, the KL divergence of D KL ( µ (cid:107) ν θ ) is an equivalent target of optimization on the diffusion loss.
Subsequently, we can credit variational gap as a training performance to tighten NELBO to NLL by
[Equation]
This variational gap satisﬁes (see Appendix B.2)
[Equation]
so variational gap is minimized if an inference posterior, µ ( { x t }| x 0 ) , estimates the intractable generative posterior, ν θ ( { x t }| x 0 ) .
However, unlike VAE, the inference posterior is not trainable, which restricts variational gap to be strictly positive in previous diffusion models.
We note that a diffusion model provides tractable NLL using the Instantaneous Change of Variable (Song et al., 2020), and this tractable variational gap becomes useful in analyzing a diffusion model quality.
P ARAMETRIZED D IFFUSION M ODEL 4. N O C LOSED - FORM T RANSITION P ROBABILITY ON N ONLINEAR D IFFUSION
As far as we know, all previously proposed SDEs, including VESDE and VPSDE, are categorized as linear SDEs because they have linear drift term of f ( x t , t ) .
This drift term determines the dynamics of the diffusion, so a data-adaptive diffusion could be attained simply by parametrizing drift and diffusion terms.
However, a transition probability with nonlinear coefﬁcients for a SDE, p 0 t ( x t | x 0 ) , is intractable in general, and a diffusion loss becomes intractable to optimize.
In fact, it is worth noting that a closed-form of the transition probability exists only on a certain type of SDEs, like
VESDE and VPSDE.
[Figure]
4. P ARAMETRIZED F ORWARD D IFFUSION WITH N ORMALIZING F LOW
To construct a tractable transition probability while keeping nonlinearity, we observe that a nonlinear diffusion can be decomposed into a linear diffusion and a nonlinear invertible transformation, where the transformation provides the ﬂexible nonlinearity on a diffusion process.
Figure 1 and 2 provide a schematic view of our model.
Normalizing ﬂow transforms a data input to a latent variable, and a diffusion model linearly permeates the latent variable to a noise distribution.
Concretely, let us deﬁne z 0 to be a transformed latent variable, z
=
h φ ( x 0 ) , where φ is ﬂow model parameters.
Then, a canonical diffusion with linear dynamics of
[Equation]
describes a diffusing mechanism of a latent variable.
Since the transformation is invertible, if we deﬁne an inverse random variable by x φ t = h − 1 φ ( z t ) , the dynamics of a stochastic process of { x φ t } Tt = is governed by a SDE of
[Equation]
where drift and diffusion terms are derived by
[Equation]
from the Ito’s lemma (Oksendal, 2013).
See full derivation and explanation in Appendix B.1.
This diffusion process is induced diffusion on the data space that is deﬁned indirectly by transforming back from the latent space to the data space.
The nonlinearity of this induced diffusion fully depends on the invertible transformation as illustrated in Figure 2.
It is not straightforward to directly construct a generative reverse process on the data space, x φ , θ t .
Figures 1 and 2 build the generative process by red arrows using the transformation.
Starting from a prior distribution of π , we denoise a random variable from the prior to z θ
according to the generative process of the linear diffusion on the latent space, and we transform the fully denoised latent of z θback to the data space x φ , θ
[Equation]
4. T RACTABLE V ARIATIONAL B OUND OF P ARAMETRIZED D IFFUSION M ODEL
The joint modeling of normalizing ﬂow and diffusion enables a tractable training of nonlinear dynamics.
The log-likelihood on a data space reduces to the log-likelihood on a latent space by adding the log determinant of Jacobian to the latent log-likelihood.
This calculation of log-likelihoods provides Theorem 1 that bounds the log-likelihood with a tractable ELBO.
Theorem 1.
Suppose that p φ , θ ( x 0 ) is the log-likelihood of a generative random variable x φ , θ 0 .
Then, the log-likelihood is bounded by
[Equation]
where
[Equation]
with L
[Equation]
, up to a constant.
Here, p 0 t ( z t | z 0 ) is a transition probability of the linear diffusion process in the latent space.
PDM ELBO in Eq.
7 is different from the vanilla diffusion ELBO in Eq.
3 from three perspectives.
First, the normalizing ﬂow loss, i.e., the log determinant of Jacobian, is added to the loss of PDM
ELBO.
This normalizing ﬂow loss represents a transformation of a data space into a latent space.
Second, the remaining terms, related to the diffusion model, is a linear diffusion on the latent space with linear dynamics, which provides tractable optimization, rather than an intractable data diffusion of nonlinear dynamics.
Third, the optimization loss in Eq.
7 estimates the score on the latent space by a score network of s θ ( z t , t ) , while the diffusion loss in Eq.
3 estimates the score on the data space by a score network of s θ ( x t , t ) .
4. O PTIMIZATION L OSSES
Previous works (Ho et al., 2020; Song et al., 2021; Vahdat et al., 2021) have empirically demonstrated that training a diffusion model with a likelihood weighting, λ ( t ) =
g 2 ( t ) , could bring worse performances than applying other weighting functions.
For instance, Ho et al.
; Song and
Ermon (2020) introduce a variance weighting function, λ ( t ) =
σ 2 ( t ) , and Ho et al.
(2020) show that this weighting outperforms a likelihood weighting in terms of the sample generation.
Within our framework, we also observe that a variance weighting function is advantageous over a likelihood weighting.
Henceforth, we optimize a ﬂow model and a diffusion model with different losses as follows:
[Equation]
Note that the ﬁrst term of the log Jacobian and the third term of the cross entropy in Eq.
7 are independent to score parameters, θ .
Therefore, diffusion parameters are not inﬂuenced by these terms, in which only the second term remains to update the diffusion parameters.
4. T IGHT V ARIATIONAL G AP OF P ARAMETRIZED F ORWARD D IFFUSION
This section theoretically analyzes variational gap of PDM.
To construct variational gap of PDM, we enumerate few nomenclatures for path measures (see Table 6).
On a data space, µ d φ is a path measure for an inference process; and ν d φ , θ is a path measure for a generative process.
On a latent space, µ l φ is a path measure for an inference process; and ν l θ is a path measure for a generative process.
Here, ν l θ does not depend on φ because the generative diffusion on the latent space starts from the φ -independent noise distribution ( π ); and because the diffusion proceeds a denoising process by following a score estimation of s θ ( z t , t ) that depends only on θ .
Variational gap on the data space becomes
[Equation]
(9) where the subscript in Gap d represents the data space.
Recall that training the encoder in VAE guarantees tighter variational gap because the encoder distribution of q φ ( x , z ) approaches toward the decoder distribution of p θ ( x , z ) by minimizing NELBO of D KL (cid:0) q φ ( x , z ) (cid:107) p θ ( x , z ) (cid:1) .
However, this analogy is not valid in variational gap in Eq. 8 because both inference and generative path measures depend on φ .
This means that the generative path measure could arbitrarily deviate from the inference path measure when we optimize φ .
Using the change of variable, however, variational gap is equivalent to
[Equation]
With this derivation, optimizing an invertible transformation guarantees tighter variational gap because the generative measure of ν l θ is now ﬁxed on φ .
Also, we note that this variational gap reduces to a KL divergence between the inference posterior and the generative posterior by
[Equation]
which provides another analogy to VAE.
We enumerate variational gaps of the explained models in Table 5.
In fact, we show in Theorem 2 that the optimum of PDM guarantees that the generative distribution p φ ∗ , θ equals to the data distribution p r , and variational gap reduces to zero in the optimum.
See Appendix A for proofs.
Theorem 2.
Suppose a ﬂow network is ﬂexible enough to transform p r to arbitrary continuous distribution.
For any θ , if ∇ · s θ ( · , t ) is symmetric for any t ∈ [0 , T ] and ∇ · log π ( · ) ≡ s θ ( · , T ) , then there exists φ ∗ such that Gap d ( µ d φ ∗ , ν d φ ∗ , θ ) = 0 .
Furthermore, we have
[Equation]
φ , θ ) .
To the best of our knowledge, Theorem 2 is the ﬁrst analysis in the community of diffusion models that the theoretic optimum of a model completely recovers the data distribution.
To proceed this argument, let us consider a linear diffusion model in a data space with no ﬂow model.
Then, a marginal distribution of the forward process is p T at t = T , which differs from a prior distribution of π .
In addition, suppose that the diffusion model reaches to the optimum point, i.e., its score network estimates the exact data score by s θ ∗ ( x t , t ) =
∇ x t log p t ( x t ) .
Then, a generative transition probability equals to the groundtruth (reverse) transition probability of p T
( x 0 | x T ) .
Therefore, the generative distribution, p θ ∗ ( x 0 ) =
(cid:82) π ( x T ) p T 0 ( x 0 | x T )
d x T , naturally diverges from the real distribution, p r ( x 0 )
= (cid:82) p T ( x T ) p T 0 ( x 0 | x T ) d x T , because of the mismatch of p T and π .
This implies that the original linear diffusion model cannot estimate the real distribution, p r ( x 0 ) .
In contrast to the original diffusion model, applying normalizing ﬂow to a diffusion model enables PDM to recover the real distribution by Theorem 2.
See Figures 8 and 9 for further illustration.
R ELATED W ORK
[Figure]
Recently, a diffusion model has been jointly modeled in conjunction with other deep generative models.
There are two major previous works on this line of research.
First, Vahdat et al.
(2021) (LSGM) in Figure 3 applies a diffusion model to the latent space of VAE.
Since VAE does not attain an invertible structure, LSGM is not suggesting the expansion of diffusing mechanisms.
In other words, LSGM cannot provide diffused data learning at the middle of the process because LSGM does not induce the diffusion process on the data space due to the lack of VAE’s invertibility.
Consequently, the analyzing tool for LSGM should be variational gap of VAE, rather than that of the diffusion model.
[Figure]
Second, Song et al.
(2021) (ScoreFlow) in Figure 4 simply concatenate normalizing ﬂow on top of a diffusion model to provide a dequantized data distribution only at the initial diffusion time.
Despite of the structural similarity of ScoreFlow and PDM, a major difference of ScoreFlow and PDM lies on their density model constructions: p θ in Scoreﬂow and p φ , θ in PDM.
ScoreFlow constructs p θ only with a diffusion model, so the ﬂow and the diffusion networks are not jointly combined in the loss function of (cid:82) p r ( x ) (cid:82) [0 , [Equation]
d q φ ( u | x ) log p [Equation] x + u ) q φ ( u | x ) d u d x , where u is a dequantization variable that lies in a unit cube, [0 , 1) d .
In other words, ScoreFlow adopts the ﬂow outside of a model density, so the ﬂow is independent to the model density.
In fact, ScoreFlow can be best understood as a dequantization technique (Ho et al., 2019), whose goal is estimating the optimal dequantizing region
[Table] [Table]
on [0 , 1) d to minimize dequantization gap (Hoogeboom et al., 2020).
Therefore, variational gap of ScoreFlow remains to be strictly positive because training the ﬂow does not optimize the forward diffusing mechanism.
On the other hand, PDM constructs ﬂow-embedded model density of p φ , θ , and PDM optimizes the loss of (cid:82) p r ( x ) log (cid:82) [0 , 1) d p φ , θ ( x + u ) d u d x that puts normalizing ﬂow inside of the model density.
Although we train PDM with a uniform dequantization (Theis et al., 2015) by default, we could apply the variational dequantization to PDM as well.
E XPERIMENTS
This section quantitatively and qualitatively analyzes suggested PDM on CIFAR-10 (Krizhevsky et al., 2009) and CelebA (Liu et al., 2015) 64 × 64 .
We compute NLL for density estimations in bits per dimension scale by solving probability ﬂow (Song et al., 2020) backwards in time, and we compare baseline performances of Song et al.
(2020; 2021) computed on a uniform dequantization.
For sample generations, we use the Predictor-Corrector algorithm (Song et al., 2020) for PDM on VESDE, and we use the numerical ODE sampler of the probability ﬂow (Song et al., 2020) with the RK45 method on VPSDE.
Throughout experimental results, we use NCSN++ (VE) and DDPM++ (VP) (Song et al., 2020) as backbones of diffusion models, and a pair of ResNet-based ﬂow models (Chen et al., 2019; Ma et al., 2020) as backbones of ﬂow models.
We give a full description on models for our experiments in Appendix C.
We name PDM (FID) and PDM (NLL) for experiments that use weighting functions as σ 2 and g 2 , respectively.
6. Q UANTITATIVE R ESULTS
We compare PDM to baselines in Table 1 on VESDE and VPSDE as linear diffusing mechanisms.
Although minor performance degradation on the sample performance, Table 1 empirically demonstrates reduced variational gap.
In all experimental settings, variational gaps signiﬁcantly drop less than 0.1.
On the other hand, variational gaps of the baseline diffusions on VPSDE for the FID model and the NLL model are 0.16 and 0.19, respectively.
We conclude that PDM effectively decreases variational gap by training a parametrized inference path measure.
Table 2 compares experiments on CelebA to the current state-of-the-art baseline models in various disciplines: the diffusion model of UDM (Kim et al., 2021) and DDGM (Nachmani et al., 2021), the VAE model of CR-NVAE (Sinha and Dieng, 2021), the ﬂow model of DenseFlow (Grci´c et al., 2021), and the GAN model of Styleformer (Park and Kim, 2021).
We observe a pair of implications.
First, training a diffusing mechanism with PDM largely reduces variational gap.
In particular, PDM reduces this inference gap from 1.57 to 0.09 on the NCSN++ backbone with VESDE and from 0.40 to 0.05 on the DDPM++ backbone with VPSDE.
Also, we emphasize that NELBO of PDM largely outperforms NELBO of the baseline models.
Second, our PDM reports the state-of-the-art performance with 2. FID score in the CelebA sample generation.
This opposite trend on FID (compared to the results in CIFAR-10) indicates that a nonlinear diffusion is more effective on datasets of higher dimensions.
In fact, this is a quite intuitive result because as the data dimension increases, the family of feasible diffusion expands exponentially, and the data-optimal diffusing mechanism is more likely to deviate from linear diffusions.
[Figure]
[Figure]
[Figure] [Figure]
L( # params)
NLL
NELBO
Gap
PDM 4(1.2M) 3.
ResFlow 5.
PDM 8(2.3M) 3.
ResFlow 4.
PDM 32(10.6M) 3.
[Figure]
[Figure] [Figure]
ResFlow 3.
6. Q UALITATIVE R ESULTS
Test NELBO To ﬁgure out an effect of a nonlinear diffusing mechanism on the training procedure,
Figure 5 (a) shows test NELBO by training iterations on a diffusion model with 35M parameters.
Without any modiﬁcation on hyperparameters, we train PDM with a shallow ResFlow (Chen et al., 2019) of 3M parameters, and Figure 5 (a) shows that the training on PDM surprisingly stabilizes test NELBO.
Figure 5 (b) illustrates that such training stability can be attributed to nearly identical score scales.
As described in Figure 5 (b), the score scale on a data space is unbounded as the diffusion time goes to zero, and this extreme scale variation eventually leads to training instability on the baseline diffusion models.
On the other hand, adopting ﬂow transformation on a data space mitigates such extreme scale variations that destabilize training.
Figure 6 shows test NELBO on a diffusion model with 107.6M number of parameters.
To stabilize the diffusion model, we apply the AdamW optimizer (Loshchilov and Hutter, 2017) with 0.01 weight decay, and Exponential Moving Average (Song et al., 2020) of 0.9999.
Using the ResFlow model with 10.6M parameters, Figure 6 indicates that PDM consistently outperforms the baseline diffusion in terms of NELBO with smaller variations.
Flow Depth The ﬂow model architecture is a building block that fundamentally determines the family of inference path measures.
Table 4 empirically demonstrates that an inﬂexible variational family results in large variational gap on CIFAR-10.
Additionally, the absolute magnitude of NLL performance keeps decreasing as the number of layers (L) increases.
Sample Generation Figure 7 presents samples from (a) a latent space and (b) a data space.
Since a sample generation on the latent space is unnormalized, we normalize the sampled latent variables to be bounded on the range of [0 , 255] , so we can visualize the latent information in image.
Figureindicates that a diffusion model and a ﬂow model have distinctive and complementary roles in sample generation: 1) the diffusion model constructs the global context and 2) the ﬂow model enriches this global context by colorizing it into a realistic image.
C ONCLUSION
This paper expands the linear diffusing mechanism to be nonlinear by combining an invertible transformation and a diffusion model.
This nonlinear diffusing mechanism learns the forward diffusion process out of variational family of inference path measures, and such optimization provides tighter variational gap compared to a baseline model with linear diffusion.
E THICS S TATEMENT
As our work improves sample quality, our methodology could be a source of fake datasets, i e. fake images.
This is a common threat from the line of deep generative modeling.
R EPRODUCIBILITY S TATEMENT
Every proofs and derivations are explained in details on Appendix A and B. Also, for reproducibility, we would release our code as soon as the discussion period opens through an anonymous repository to reviewers.
We note that most of our code is built based on the released code of Song et al.
(2020) in order to compare with baselines in a fair setting introduced in Song et al.
and Song et al.
Additionally, we provide the details on training/evaluation/neural architecture in Appendix C
A P ROOFS OF T HEOREM 1 AND T HEOREM
Theorem 1.
Suppose that p φ , θ ( x 0 ) is the log-likelihood of the generative random variable x φ , θ 0 .
Then, the log-likelihood is bounded by
[Equation]
where
[Equation]
with σ 2 ( t ) = 1 − e − (cid:82) t 0 β ( s )
d s + (cid:82) t
[ g 2 ( s ) − β ( s )] d s .
Proof of Theorem 1.
From the change of variable, the transformation of z 0 =
h φ ( x 0 ) induces
[Equation]
and thus the entropy of the data distribution becomes
[Equation]
From Theorem 4 of Song et al.
[Equation]
the entropy at t = 0 equals to
[Equation]
[Equation]
where f ( z t , t ) is a drift term of the diffusion for z t and p t is the probability distribution of z t .
Therefore, the negative log-likelihood becomes
[Equation]
Now, from Theorem 1 of Song et al.
[Equation]
the KL-divergence between the path measures becomes
[Equation]
so if we plug in this into the negative log-likelihood
[Equation]
we yield the following:
[Equation]
Also, we have
[Equation]
Therefore,
[Equation]
Now
[Equation]
since p 0 t ( z t | z 0 ) = N ( z t ; µ ( t ) z t , σ 2 ( t ) I ) for µ ( t ) and σ 2 ( t ) determined by β ( t ) and g ( t ) , we have
[Equation]
and we have the desired result
[Equation]
Theorem 2.
Suppose the ﬂow network is ﬂexible enough to transform p r to arbitrary continuous distribution.
For any θ , if ∇ · s θ ( · , t ) is symmetric for any t ∈ [0 , T ] and ∇ · log π ( · ) ≡ s θ ( · , T ) , then there exists φ ∗ such that Gap d ( µ d φ ∗ , ν d φ ∗ , θ ) = 0 .
Furthermore, we have
[Equation]
Proof of Theorem 2.
The symmetry of ∇ · s θ ( · , t ) indicates that s θ ( · , t ) is a closed 1-form.
Therefore, from the Poincare’s lemma (Do Carmo, 2012), s θ ( · , t ) is exact
[Equation]
and there exists a scalar ﬁeld f θ t that satisﬁes s θ ( · , t ) =
∇ · f θ t ( · ) for any t .
If we deﬁne by q θ t ( z t )
=
[Equation]
z , then we have
[Equation]
t , t ) = ∇ z t log q θ t ( z t , t ) for any t .
With this construction, the reverse SDE on the transformed latent space becomes
[Equation]
[Figure] [Figure]
𝑟𝑟 ℎ 𝜙𝜙
𝜋𝜋
ℎ
𝜙𝜙 𝜈𝜈 𝜙𝜙 , 𝜃𝜃𝑑𝑑 𝑝𝑝 𝜙𝜙 , 𝜃𝜃 𝜇𝜇 𝜙𝜙𝑑𝑑 𝜇𝜇 𝜙𝜙𝑙𝑙 𝜈𝜈 𝜙𝜙 , 𝜃𝜃𝑙𝑙 𝑝𝑝 𝜃𝜃
[Figure]
[Figure]
If ρ θ is the path measure of the solution of the above SDE, then the marginal distribution of ρ θ at t would be q θ t .
Therefore, if h − 1 φ ∗ maps p r to q θ 0 , then the forward process on the latent space { z φ ∗ t } Tt =0 becomes the generative process on the latent space { z θ t } Tt =0 .
Thus, we have µ l φ ∗ =
ν l θ in distribution, which leads D KL ( µ l φ ∗ (cid:107) ν l θ )
= 0 .
We get the desired result by the change of variable formula: D KL ( µ d φ ∗ (cid:107) ν
d φ ∗ , θ )
= D KL ( µ l φ ∗ (cid:107) ν l θ ) = 0 .
B D ERIVATIONS B. D ERIVATION OF N ONLINEAR D IFFUSION T ERMS
Throughout this section
[Equation]
we omit φ for the notational simplicity.
Beginning with the linear diffusion on the latent space given by
[Equation]
we have deﬁned x t = h − 1 ( z t ) as the induced random variables.
Then by the multivariate Ito’s
Lemma (Oksendal, 2013)
[Equation]
the k -th component of the induced variable satisﬁes
[Equation]
Plugging the linear diffusion SDE 11 in the SDE of Eq.
12, Eq.
12 is derived by
[Equation]
because ∂ h − 1 k ∂t = 0 .
Then, Eq. 13 in vector form becomes
[Equation]
where vector-valued drift and diffusion terms are given by
[Equation]
) where ∇ 2 z t h − 1 ( z t ) is a 3-dimensional tensor with ( i, j, k ) -th element to be (cid:0) ∇ 2 z t h − 1 k ( z t ) (cid:1) i,j , and the trace operator applied on this tensor results in the vector of (cid:104) tr (cid:0) ∇ 2 z t h
[Equation]
( z
[Equation]
From the inverse function theorem (Rudin et al., 1964), the Jacobian of the inverse function ∇ z t h − 1 ( z t ) equals to the inverse Jacobian (cid:2) ∇ x t h ( x t ) (cid:3) − 1 .
Therefore, Eq. 14 is transformed to
[Equation]
Now, we derive the second term of f in terms of x t as follows: observe that (cid:80) k
∂ h − 1 i ∂ z t,k ∂
h k ∂ x t,j = δ i,j , where δ i,j = 1 if i = j and 0 otherwise.
Differentiating both sides with respect to x t,l , we have (cid:88) k (cid:26) ∂
∂ x t,l (cid:18) ∂ h − 1 i ∂ z t,k (cid:19)(cid:27)
∂
h k ∂ x t,j + ∂ h − 1 i ∂ z t,k (cid:26) ∂
∂ x t,l (cid:18)
∂
h k ∂ x t,j (cid:19)(cid:27) = 0 , where the ﬁrst term is (cid:88) k,m ∂ h m ∂ x t,l (cid:26) ∂
∂ z t,m (cid:18)
∂ h − 1 i ∂ z t,k (cid:19)(cid:27) ∂
h k ∂ x t,j = (cid:88) k,m (cid:0) ∇ x t h ( x t ) (cid:1) T l,m (cid:0) ∇ 2 z t h − 1 i ( z t ) (cid:1) m,k (cid:0) ∇ x t h ( x t ) (cid:1) k,j , using the chain rule, and the second term becomes (cid:88) k
∂ h − 1 i ∂ z t,k (cid:26) ∂
∂ x t,l (cid:18)
∂
h k ∂ x t,j (cid:19)(cid:27) =
(cid:88) k (cid:0) ∇ z t h − 1 ( z t ) (cid:1) i,k (cid:0) ∇ 2 x t h k ( x t ) (cid:1) l,j .
From above, we derive the trace term of f in Eq.
15 as tr
[Equation]
∇
[Equation]
(cid:2) ∇ x t h ( x t ) (cid:3) − 1 (cid:17) ,
[Table]
where ∇ 2 x t h ( x t ) is a 3-dimensional tensor with ( i, j, k ) -th element to be (cid:0) ∇ 2 x t h k ( x t ) (cid:1) i,j .
Also, we deﬁne ∗ operation as the element-wise matrix multiplication given by
[Equation]
Combining all together, thus, we derive nonlinear drift term in Eq.
15 as a function of x t given by
[Equation]
B. D ERIVATION OF V ARIATIONAL G AP
First, the KL divergence between two path measures satisﬁes
[Equation]
[Table]
where x 0: T = { x t } Tt =0 .
Therefore, variational gap is derived by
Gap ( µ , ν θ ) = D KL
[Equation]
The gap Gap d ( µ d φ , ν d φ , θ ) in Eq.
10 is derived to be
E z 0 (cid:104) D KL (cid:0) µ φ ( { z t }|
z
0 ) (cid:107) ν
θ ( { z t }| z 0 ) (cid:1)(cid:105) , analogously.
C I MPLEMENTATION D ETAILS C. M ODEL A RCHITECTURE
Diffusion Model We implement two diffusion models as backbone: NCSN++ (VE)
(Song et al., 2020) and DDPM++ (VP) (Song et al., 2020), where two backbones are one of the best performers in CIFAR-10 dataset.
In our setting, NCSN++ assumes the score network with parametrization of s θ ( z t , log σ 2 ( t )) , where σ 2 ( t ) =
σ 2 min ( σ max σ min ) 2 t is the variance of the transition probability p 0 t ( z t | z 0 ) with VESDE.
As introduced in Song et al.
(2020), we use the Gaussian Fourier embeddings (Tancik et al., 2020) to model the high frequency details across the temporal embedding.
DDPM++ models the score network with parametrization of (cid:15) θ
( z t , t ) , which targets to estimate − σ ( t ) ∇ z t log p t ( z t ) .
We use the Transformer sinusoidal temporal embedding (Vaswani et al., 2017).
We use the U-Net (Ronneberger et al., 2015) architecture for the score networks on both NCSN++ and DDPM++ based on (Ho et al., 2020).
We stack U-Net resblocks of up-and-down convolutions with skip connections that connect the identical-dimensional layers.
Also, we follow Ho et al.
by applying the global attention at the resolution of 16 × 16 .
We use four U-Net resblocks with four feature map resolutions ( 32 × 32 to 4 × 4 ).
There are eight and four resblocks at each resolution on
CIFAR-10 and CelebA, respectively.
Flow Model We implement two normalizing ﬂow models as backbone: the ResFlow (Chen et al., 2019) and the decoupled ResFlow (Ma et al., 2020).
Although Ma et al.
(2020) makes use of the
Glow (Kingma and Dhariwal, 2018) for their ﬂow model, we replace this glow architecture with the
ResFlow, mainly because the ResFlow transforms the variables from the identity function.
We name the experiments on this decoupled ResFlow as PDM (FID, dec) and PDM (NLL, dec).
Otherwise, we train the model with ResFlow as default.
For the ResFlow, we drop three components from the original paper: 1) the activation normalization Kingma and Dhariwal (2018), 2) the batch normalization Ioffe and Szegedy (2015), and 3) the fully connected layers.
On the deep ResFlow, we use resolutions of 32 × 32 and 16 × 16 on both CIFAR-10 and CelebA.
Otherwise, we use a single resolution of 32 × 32 on other experiments whose models have networks with layers less than 32.
We apply the LipSwish activation (Chen et al., 2019) that provides the
Lipschitz coefﬁcients less than a unity.
For the multi-GPU training, we use the Neumann logdeterminant gradient estimator, instead of the memory efﬁcient estimator (Chen et al., 2019).
C. E XPERIMENTAL DETAILS
Training All the models are trained with batch size of 128 and EMA rate of 0.999.
We train each model for around 10 days on four V-100 GPUs with 128Gb GPU memory for all experiments.
At the initial training phase, we train the diffusion model parameters with ﬁxed ﬂow parameters initialized by the identity until the sample generation performance saturates.
After this pretraining phase, we start updating both diffusion and ﬂow models.
In this learning phase, we apply the learning rate scheduling after the sample performance saturation.
For the diffusion model, we drop the learning rate from 2 × 10 − 4 to 10 − 5 .
For the ﬂow model, we drop the learning rate from 10 − 3 to 10 −for VPSDE and 5 × 10 − 5 to 10 − 5 for VESDE.
We emphasize this learning rate scheduling takes additional performance boost.
VESDE and VPSDE have different training details.
VESDE assumes σ min = 10 − 2 on all experiments.
VESDE has σ max = 50 on CIFAR-10 and σ max = 90 on CelebA.
On the other hand,
VPSDE assumes β ( t ) =
β min + ( β max − β min ) t with β min = 0 .
1 and β max = 20 .
Both VESDE and VPSDE truncate the diffusion time on [ (cid:15), T ] in order to stabilize the diffusion model, where (cid:15) =
10 − 5 and T = 1 .
On sample generation, however, DDPM++/PDM with VPSDE draws samples by proceeding the generative diffusion process up to (cid:15)
=
10 − 3 , which corresponds to the diffusion variance to be σ min = 10 − 2 .
On the sample generation, we apply the Predictor-Corrector (PC) sampler Song et al.
(2020) for NCSN++/PDM with VESDE and the Probability Flow (PF) ODE sampler Song et al.
(2020) for DDPM++/PDM.
Throughout the experiments, we provide the identical diffusion model structures to compare the baseline model and the PDM
Evaluation While we use the EMA rates of 0.999 for both diffusion and ﬂow models for training, we use this EMA only for the diffusion model on the evaluation mode because the ﬂow model without the EMA on the evaluation gives better samples.
One peculiarity occurs in the experiment on the decoupled ResFlow.
We empirically ﬁnd that turning off the EMA for both diffusion model and ﬂow model on the evaluation mode provides better performances of 3.11 in test NLL and 3.17 in test
NELBO as reported in Table 1 on PDM (FID, dec).
With the EMA applied to the diffusion model, we have 3.15 for test NLL and 3.22 for test NELBO, which shows slight degradation on the density estimation performance.
We compute the FID score based on Song et al.
(2020) for CIFAR-10 with the modiﬁed Inception V network 1 in order to compare PDM to the baselines (Song et al., 2020; 2021) in a fair setting.
On the other hand, for the CelebA dataset, we compute the clean-FID (Parmar et al., 2021) that provides consistently antialiased results.
C. V ARIANCE R EDUCTION
Flow Training When we train the ﬂow network with L φ , θ (cid:0) { x t } Tt =0 ; g 2 ) , this NELBO contains the integration of L θ (cid:0) { z t } Tt =0 ; g 2 (cid:1) .
However, previous works on the diffusion models (Nichol and Dhariwal, 2021) show that the estimation variance is largely reduced with the importance sampling, which leads the performance variations (Song et al., 2021).
Concretely, the importance sampling chooses an importance weight that is proportional to g 2 ( t ) σ 2 ( t ) , and estimates the integration by L θ (cid:0) { z t } Tt =0 ; g 2 (cid:1)
= (cid:82) T 0 g 2 ( t )
L t (cid:0) { z t } Tt =0 ; θ (cid:1)
d t ≈ (cid:80) Nn
=1 σ 2 ( t n ) L t n (cid:0) { z t } Tt =0 ; θ (cid:1) , where t n is sampled from the importance distribution.
For VESDE, it satisﬁes β ( t ) =
and g ( t ) =
σ min ( σ max σ min ) t (cid:113) 2 log ( σ max σ min ) .
Throughout the experiments, we select σ min = 0 .
01 , and, σ max = 50 , 90 for CIFAR-10 and CelebA, respectively.
The transition probability becomes p 0 t ( z t | z 0 )
=
N ( z t ; z 0 , σ 2 ( t )) , where σ 2 ( t ) =
(cid:82) t
g 2 ( s )
d s = σ 2 min (cid:2) ( σ max σ min ) 2 t − 1 (cid:3) .
Song et al.
(2020) approximates this variance to be σ 2 app ( t )
=
σ 2 min ( σ max σ min ) 2 t so that the variance is proportional to g 2 ( t ) .
Therefore, the importance weight follows the uniform distribution, and the importance sampling is equivalent with choosing the uniform t .
On the other hand, VPSDE satisﬁes β ( t ) =
β min + ( β max − β min ) t with g ( t ) =
(cid:112) β ( t ) .
Then, the transition probability becomes p 0 t ( z t | z 0 )
=
N ( z t ; µ ( t ) z t , σ 2 ( t ) I ) , where µ ( t ) =
e − 12 (cid:82) t 0 β ( s )
d s and σ 2 ( t )
= 1 − e − (cid:82) t 0 β ( s )d s .
Thus, VPSDE has the importance weight of g 2 ( t ) σ 2 (
[Equation]
The Monte-Carlo sample from this importance weight is the solution of the inverse Cumulative
Distribution Function (CDF) of the importance distribution as
[Equation]
where u is a uniform sample from [0 , 1] , F ( t ) is the antiderivative of the importance weight given by F ( t )
= log (1 − e
[Equation]
min )
+ 0 .
5 t 2 ( β max − β min ) + tβ min , and Z is the normalizing constant given by
[Equation]
for T = 1 and (cid:15) =
10 − 5 .
The solution for the inverse CDF in Eq.
16 becomes
[Equation]
The variation of the Monte-Carlo diffusion time depends on the uniform sample of u .
Diffusion Training Now, on the side of training the diffusion network, as described in the main paper, selecting the weighting function heavily inﬂuences the model performances.
For instance,
[Table] [Table]
Song et al.
(2021) compare the weighting functions of g 2 and σ
[Table]
and Song et al.
(2021) conclude that the weighting function of λ ( t ) =
σ 2 ( t ) works the best for the sample generation
[Table]
and λ ( t ) =
g 2 ( t ) yields the best NLL performance.
We train the diffusion model with σ 2 ( t ) for PDM (VE/VP, FID), and with g 2 ( t ) for PDM (VP, NLL) models.
D A DDITIONAL E XPERIMENTAL R ESULTS D. D ETAILED C OMPARISON AND A BLATION ON T OLERANCE
[Figure]
Table 7 has two implications.
First, we ﬁnd that the reported NLL and NELBO performances are slightly better than what we actually obtain in our pretraining step, denoted by DDPM++ (NLL, IS, ours).
Comparing with our results, optimizing PDM is indeed beneﬁcial on density estimation, and variational gap is signiﬁcantly reduced from 0.17 to 0.01.
Second, we ﬁnd the tolerance of 10 − 1 for the numerical ODE solver on computing NLL is the sweet spot on obtaining best NLL of 2.86, which beats LSGM-480M (NLL) on density estimation.
This is noteworthy because our model uses 120M number of parameters, a quarter of parameters in LSGM.
On the other hand, NELBO is independent of the tolerance because it does not use the numerical ODE solver, and Table 7 shows that variational gap is minimized when the tolerance is small enough.
Figure 10 illustrates the ablation study on the tolerance for the ODE solver.
Surprisingly, PDM is robust on the tolerance level up to 10 − 1 , in contrast to LSGM with poor sample quality at 10 −(Vahdat et al., 2021).
From this robustness on the tolerance, we could reduce the sampling wall clock time from 360s (tolerance 10 − 5 ) to 72s (tolerance 10 − 1 ) for 1,024 samples, which is 5x faster, with a minor sacriﬁce in FID score from 6.79 to 8.56 in PDM (FID) with VPSDE.
D. P RECISION AND R ECALL
While FID measures the overall sample quality that counts both ﬁdelity and diversity at once, we provide precision and recall (Kynkäänniemi et al., 2019) based on the modiﬁed Inception V1 network
[Table] [Table]
[Table] [Table]
2 with 50k samples in order to separate the sample ﬁdelity and diversity.
Precision is a metric that measures the sample ﬁdelity, and the recall is for the sample diversity.
Table 8 indicates that both precision and recall remains at the identical scale, except for precision at PDM (FID, dec).
D. E FFECT OF S IGNAL - TO -N OISE AND T EMPERATURE
[Figure]
T
[Figure]
m p e r a t u r e ( )
As the ﬂow colorizes the latent sample, we experiment the effect of temperature (Kingma and Dhariwal, 2018) on this colorization in
Figure 11.
With temperature τ , the normalizing ﬂow puts its latent input scaled by τ to the ﬂow network.
In our PDM, the image color with a higher temperature tends to be brighter, and we ﬁnd that the optimal temperature depends on the experimental settings.
On sample generation, we draw latent samples from either the PC sampling or the PF
ODE sampling.
PC sampling assumes the Signalto-Noise Ratio (SNR) that modiﬁes the Corrector algorithm (Song et al., 2020) of the annealed Langevin dynamics (Song and Ermon, 2019).
From the sampled latent variable, we scale by multiplying the temperature τ (Kingma and
Dhariwal, 2018) to the latent variable.
Since the diffusion model learns the global image structure, the normalizing ﬂow only paints the color of the image, and the temperature controls the brightness of the generated image.
This is a contrastive result to the previous ﬂow models (Kingma and Dhariwal, 2018; Chen et al., 2019) that the temperature in the previous models controls both global context (such as the smoothness) and color.
Therefore, we can create images with different colors by manipulating this temperature.
CIFAR-10 With these hyperparameters of the SNR and the temperatures, we ﬁnd optimal values in the below Tables.
Table 9 computes the FID-5k score of all combinations for (SNR, τ ) in PDM (VE,
FID) on CIFAR-10.
Unlike the SNR of 0.16 that is previously suggested in (Song et al., 2020) for the vanilla diffusion model, we ﬁnd that SNR of 0.145 consistently outperforming other values for SNR in terms of FID.
Similarly, Table 9 presents that the optimal temperature is 1.1.
For VPSDE, Table 10 shows that the temperature of 1.3 performs the best for sample generation in
PDM (VP, FID).
For PDM (VP, FID, dec), we ﬁnd that the temperature of 1.1 performs the best, and for PDM (VP, NLL, dec), the vanilla temperature of 1.0 generates the best samples.
CelebA Table 11 shows the FID-1k performance of PDM (VE, FID) on CelebA.
It shows that the
SNR of 0.17 proposed in (Song et al., 2020) performs the best also in our PDM.
In addition, it shows that the optimal temperature is either 1.0 or 1.05.
Table 12 shows the FID-1k performance of PDM (VP) on CelebA. It is clear in this table that the sample quality is the best on the temperature range of 1.05 to 1.15.
D. R ANDOM SAMPLES
Figure 12 and 13 show the non cherry-picked random samples from PDM (VE, FID) on CIFARand PDM (VP, FID) on CelebA, respectively.