M AXIMUM L IKELIHOOD T RAINING OF P ARAMETRIZED D IFFUSION M ODEL
Anonymous authors
Paper under double-blind review
A BSTRACT
Whereas the diverse variations of the diffusion model exist in image synthesis, the previous variations have not innovated on the topic of the diffusing mechanism that was always kept linear.
Meanwhile, it is intuitive that there would be a more promising diffusion pattern adapted to the data distribution.
This paper introduces such adaptive and nonlinear diffusion method for the score-based diffusion models.
Unlike the static and linear Variance Exploring (VE) or Variance Preserving (VP) Stochastic Differential Equations (SDE) of the previous diffusion models, our Parameterized Diffusion Model (PDM) learns the optimal diffusion process by combining the normalizing ﬂow and the diffusion process.
Speciﬁcally, PDM utilizes the ﬂow to transform non-linearly a data variable into a latent variable, and PDM applies the diffusion process to the transformed latent variable with the linear diffusing mechanism.
Subsequently, PDM enjoys the nonlinear and learned diffusion from the perspective of the data variable.
We train PDM with the variational proxy of the log-likelihood, and we prove that the variational gap between the variational bound and the log-likelihood becomes tight when the normalizing ﬂow is optimal.
I NTRODUCTION
The diffusion models have recently achieved the success on the sample generation, and some researches claim the state-of-the-art performance in sample generation over Generative Adversarial Networks (GAN) structures (Karras et al., 2019).
This particular success is highlighted in the community of the likelihood-based models, including the normalizing ﬂows (Grci´c et al., 2021), the autoregressive models (Parmar et al., 2018), and the variational autoencoders (VAE) (Vahdat and Kautz, 2020).
Moreover, this success is noteworthy because it is achieved merely using the linear diffusing mechanisms, such as VESDEs (Song and Ermon, 2020) and VPSDEs (Ho et al., 2020).
This paper expands these linear diffusing mechanisms of VE/VPSDE to a data-adaptive trainable nonlinear diffusion.
To motivate the expansion, though there are structural similarities between the diffusion models and VAEs, the forward diffusion in the diffusion model has not been trained in existing literature while its counterpart, which is the encoder of VAE, is trainable.
Rather, the current diffusion models assume the linear diffusing mechanism to be ﬁxed throughout the training procedure.
Because of this static nature of the diffusing mechanism, the variational gap (Cremer et al., 2018) between the log-likelihood and the Evidence Lower BOund (ELBO) remains to be strictly positive unless the score perfectly estimates the data score.
This variational gap prevents the score training from being Maximum Likelihood Estimation (MLE).
This gap is the fundamental motivation to develop the trainable encoder of the diffusion model, so that the gap can be tight.
As we tighten the variational gap by training a nonlinear forward diffusion, the biggest challenge comes from the intractable optimization loss.
The denoising diffusion loss requires the transition probability to be closed-form in order to achieve the fast optimization, but the nonlinear diffusing mechanism, in general, has no closed-form perturbation probability.
Hence, our innovation becomes designing a diffusion model with the tractable loss while the diffusion is learnable and ﬂexible, which we achieve by merging the previous diffusion model and the tractable variable transformation.
Our innovation concentrates on the theoretic and the practical aspects of a new diffusion, which is tractable, learnable, and ﬂexible.
Theoretically, we observe that the nonlinear diffusing mechanism can be transformed to a linear diffusion under an invertible transformation, and vice versa.
In practice, we implement PDM by merging the normalizing ﬂow model and the diffusion model, which constructs the tractable, learnable, and ﬂexible nonlinear diffusing mechanism.
From this construction, we prove that the variational gap can be further tightened by training the forward diffusion nonlinearly.
Also, we demonstrate the state-of-the-art performance in CelebA on Fréchet Inception Distance (FID) (Heusel et al., 2017).
We summarize our key contribution as below.
• PDM expands the scope of the forward diffusion process from the linear and ﬁxed dynamics to the nonlinear and trainable dynamics.
• PDM minimizes the variational gap between the log-likelihood and ELBO, lower than the gap of the linear diffusion models because PDM learns the data-adaptive nonlinear diffusion. • PDM explicitly proves in Theorem 2 that the optimal generative distribution equals to the real distribution, which was not the case for the linear diffusion model.
P RELIMINARY
The diffusion model is constructed with the bidirectional stochastic processes.
The forward direction diffuses the input variable to a noise variable, and the reverse direction denoises the random noise to construct the realistic input data.
The diffusion model learns this reverse direction (generative process) by estimating the data score.
We provide the brief summary of the diffusion model, as below.
Forward Diffusion
At the beginning of model build-up, the diffusion process requires the diffusing mechanism on the input variable.
This paper assumes that the diffusion process is governed by the SDE of d x t
= f
( x t , t ) d t + G ( x t , t ) d ω t with x 0 ∼ p r , where p r is the distribution from a real-world dataset and x t is the solution of the SDE.
Reverse Diffusion
The theory of the stochastic calculus guarantees that we could create the identical diffusion process, { x t } Tt =0 , by solving an associated reverse SDE backwards in time (Anderson, 1982).
The associated reverse SDE is
[Equation]
where ω t and ¯ ω t are the standard Wiener processes with the time ﬂows forward and backward, respectively; and p t is the probability law of x t .
Generative Diffusion
The diffusion model approximates the above SDE 1 to eventually yield the estimation on the data distribution of p r .
In SDE 1, the previous literature setup that the drift and the diffusion terms, f and G , are determined a-priori in the forward diffusion.
However, the data score, ∇ x t log p t ( x t ) , is intractable to compute, so we estimate this data score with the score network of s θ ( x t , t ) in order to mimic the reverse diffusion with our generative process.
This score network approximates the reverse diffusion by plugging the estimated score in place of the data score with the below generative process:
[Equation]
The generative process starts from the prior distribution ( π ), and it constructs the time-continuous random variables x θ t by solving the SDE 2 backwards in time.
The generated stochastic process is denoted by { x θ t } Tt =0 , and we omit θ in the superscript if no confusion arises.
With this generative process, we deﬁne the generative distribution, x θ 0 ∼ p θ , as the probability density of the generated random variable at time t = 0 .
Score Estimation We train the score network by the variational bound of the log-likelihood, given by
[Equation]
where L t (cid:0) { x t } Tt =0 ; θ (cid:1) =
E x 0 , x t (cid:2) (cid:107) s θ ( x t , t ) − ∇ x t log p 0 t ( x t | x 0 ) (cid:107) 22 (cid:3) up to a constant.
Here, λ is the weighting function that determines the level of contribution for each diffusion time on the overall diffusion loss, L ( { x t } Tt =0 , λ ; θ ) (Song et al., 2020).
The variational bound holds when the weighting function is the likelihood weighting ( g ) (Song et al., 2021), where G is a scalar-valued g function.
M OTIVATION OF N ONLINEAR D IFFUSING M ECHANISM
Though it has long been theoretically and empirically grounded to train the encoder part in VAE, such solid ground is not accomplished in the diffusion models.
This section analyzes the structural similarity between the VAE model and the diffusion model, which brings the foundation of the data-adaptive nonlinear diffusing mechanism.
3. V ARIATIONAL G AP OF VAE
Given Negative ELBO (NELBO) of Negative Log-Likelihood (NLL) in VAE as
[Equation]
we have a pair of interpretations on this NELBO.
First, when we focus on Eq. 4, NELBO 1) aids the data reconstruction by optimizing E q φ ( z | x ) (cid:2) − log p θ ( x | z ) (cid:3) and 2) regularizes the inference distribution of the encoder to a prior distribution by D KL (cid:0) q φ ( z | x ) (cid:107) p ( z ) (cid:1) .
On the other hand, if we concentrate on Eq. 5, NELBO bounds NLL by approximating the intractable decoder posterior of p θ ( z | x ) with the tractable encoder posterior of q φ ( z | x ) .
The vanilla VAE (Kingma and Welling, 2013) assumes this approximate posterior to be the Gaussian distribution with the mean and the diagonal covariance through amortized inference: q φ ( z | x ) = N (cid:0) z ; µ φ ( x ) , σ 2 φ ( x ) I (cid:1) .
By expanding the ﬂexibility of this approximate posterior into the variational family of general distributions, for instance, Rezende and Mohamed (2015) resulted in tighter NELBO, which leads the optimization of VAE closer to MLE, and their choice of the generalizable model was the normalizing ﬂow.
To connect this NELBO to the diffusion loss, we restate NELBO in the language of a stochastic process.
Having that VAE attains the stochastic process of bivariate random variables, { x , z } , NELBO is reformulated to the KL divergence between two joint distributions modeled in bidirectional ways:
[Equation]
The inequality 6, i.e., D KL ( p r (cid:107) p θ ) ≤ D KL ( q φ ( x , z ) (cid:107) p θ ( x , z )) , is well-known by itself in the ﬁeld of information theory (Duchi, 2016) by the name of the data processing inequality.
This restated bound interprets that VAE is essentially a bimodeling approach of the joint distribution on the bivariate stochastic process, { x , z } .
On the forward direction ( x → z ), the latent variable is conditioned on the data variable, and the joint distribution is modeled by q φ ( x , z )
= p r ( x ) q φ ( z | x ) .
On the reverse direction ( z → x ), the generative data variable is conditioned on the latent variable, and the joint distribution is modeled by p θ ( x , z )
= p ( z ) p θ ( x | z ) .
Under this equivalent framework, we present an analytic tool to measure the closeness of NELBO and NLL as in Table 5 by
[Equation]
and we denote this quantity as the variational gap (Cremer et al., 2018).
Though it is one of the central tasks in VAE to minimize this variational gap (Kingma et al., 2016), only limited works (Burda et al., 2015; Neal, 2001) have estimated this variational gap due to its computational burden.
We introduce the variational gap in the next section of the diffusion model, where the gap is tractably computed.
3. V ARIATIONAL G AP OF D IFFUSION M ODELS
This section constructs the analogy of the diffusion model to VAE.
To begin with, we observe that the classical data processing inequality in VAE holds on the stochastic process with bivariate random variables on a ﬁnite-dimensional Euclidean space.
On the other hand, the time horizon of the diffusion [Figure] [Figure] model is continuous, so it contains an inﬁnite number of random variables in its stochastic as { x t } Tt =0 .
Fortunately, the ﬁnite-dimensional classical data processing inequality on the bivariate random variables is directly extendable to the stochastic process with a continuous index.
Formally, the inﬁnite-variate joint distribution is called the path measure (Oksendal, 2013), which is deﬁned as the probability law on the space of continuous functions induced by { x t } T t =0 .
Therefore, the path measure is an alternative concept of the joint distribution for the stochastic process with a continuous index.
The path measure for the forward diffusion in the diffusion model corresponds to the joint distribution for the encoder part in VAE, and similarly, the path measure for the generative diffusion corresponds to the decoder part in VAE.
If µ is the forward (inference) path measure and ν θ is the generative path measure; then the data processing inequality becomes
[Equation]
The real distribution of p r is the marginal distribution of µ at t = 0 , and the generative distribution of p θ becomes the marginal distribution of ν θ at t = 0 .
Under this concept, the data processing inequality means that the distributional divergence of D KL ( µ (cid:107) ν θ ) between the path measures bounds the divergence of D KL ( p r (cid:107) p θ ) between the sliced path measures at t = 0 .
Previous works (Song et al., 2021; Huang et al., 2021) regard this variational bound being equivalent to the diffusion loss,
[Equation]
where G ( x t , t ) = g ( t ) is a scaler function on t , and λ ( t ) is the square of the diffusion term g 2 ( t ) .
Since the previous diffusion models optimize the diffusion loss, the KL divergence of D KL ( µ (cid:107) ν θ ) is an equivalent target of the optimization on the diffusion loss.
Subsequently, we can credit the variational gap as the training performance to tighten the NELBO to NLL by
[Equation]
This variational gap satisﬁes (see Appendix B.2)
[Equation]
so the variational gap is minimized if the inference posterior, µ ( { x t }| x 0 ) , estimates the intractable generative posterior, ν θ ( { x t }| x 0 ) .
However, unlike VAE, the inference posterior is not trainable, which restricts the variational gap to be strictly positive in previous diffusion models.
We note that the diffusion model provides tractable NLL using the Instantaneous Change of Variable (Song et al., 2020), and this tractable variational gap becomes useful in analyzing the diffusion model quality.
P ARAMETRIZED D IFFUSION M ODEL 4. N O C LOSED - FORM T RANSITION P ROBABILITY ON N ONLINEAR D IFFUSION
As far as we know, all previously proposed SDEs, including VESDE and VPSDE, are categorized as the linear SDEs because they have the linear drift terms of f ( x t , t ) .
This drift term determines the dynamics of the diffusion, so a data-adaptive diffusion could be attained simply by parametrizing the drift and diffusion terms.
However, the transition probability with the nonlinear coefﬁcients for the
SDE, p 0 t ( x t | x 0 ) , is intractable in general, and the diffusion loss becomes intractable to optimize.
In fact, it is worth noting that the closed-form of the transition probability exists only on certain types of SDEs like VESDE and VPSDE. [Figure]
4. P ARAMETRIZED F ORWARD D IFFUSION WITH N ORMALIZING F LOW
To construct a tractable transition probability while keeping the nonlinearity, we observe that the nonlinear diffusion can be decomposed into the linear diffusion and the nonlinear invertible transformation, where the transformation provides the ﬂexible nonlinearity on the diffusion process.
Figureand 2 provide the schematic view of our model.
The normalizing ﬂow transforms the data input to a latent variable, and the diffusion model linearly permeates the latent variable to the noise distribution.
Concretely, let us deﬁne z 0 to be the transformed latent variable, z
=
h φ ( x 0 ) , where φ is the ﬂow model parameter.
Then, the canonical diffusion with the linear dynamics of
[Equation]
describes the diffusing mechanism of the latent variable.
Since the transformation is invertible, if we deﬁne the inverse random variable by x φ t = h − 1 φ ( z t ) , the dynamics of the stochastic process of { x φ t } Tt =0 is governed by the SDE of
[Equation]
where the drift and the diffusion terms are derived by
[Equation]
from the Ito’s lemma (Oksendal, 2013).
See full derivation and explanation in Appendix B.1.
This diffusion process is an induced diffusion on the data space that is deﬁned indirectly by transforming back from the latent space to the data space.
The nonlinearity of this induced diffusion fully depends on the invertible transformation as illustrated in Figure 2.
It is not straightforward to directly construct the generative reverse process on the data space, x φ , θ t .
Figures 1 and 2 build the generative process by red arrows using the transformation.
Starting from the prior distribution of π , we denoise the random variable from the prior to z θ {+
+}
according to the generative process of the linear diffusion on the latent space, and we transform the fully denoised latent of z θ 0 back to the data space x φ , θ
[Equation]
4. T RACTABLE V ARIATIONAL B OUND OF P ARAMETRIZED D IFFUSION M ODEL
The joint modeling of the normalizing ﬂow and the diffusion model enables the tractable training of the nonlinear dynamics.
The log-likelihood on the data space reduces to the log-likelihood on the latent space by adding the log determinant of Jacobian to the latent log-likelihood.
This calculation of log-likelihoods provides Theorem 1 that bounds the log-likelihood with the tractable ELBO.
Theorem 1.
Suppose that p φ , θ ( x 0 ) is the log-likelihood of the generative random variable x φ , θ 0 .
Then, the log-likelihood is bounded by
[Equation]
where
[Equation]
with L
[Equation]
, up to a constant.
PDM ELBO in Eq. 7 is different from the vanilla diffusion ELBO in Eq.
3 from three perspectives.
First, the normalizing ﬂow loss, i.e., the log determinant of Jacobian, is added to the loss of PDM
ELBO.
This normalizing ﬂow loss represents the transformation of the data space into the latent space.
Second, the remaining terms, related to the diffusion model, is the linear diffusion on the latent space with linear dynamics, which provides tractable optimization, rather than the intractable data diffusion of nonlinear dynamics.
Third, the optimization loss in Eq.
7 estimates the score on the latent space by the score network of s θ ( z t , t ) , while the diffusion loss in Eq.
3 estimates the score on the data space by the score network of s θ ( x t , t ) .
4. O PTIMIZATION L OSSES
Previous works (Ho et al., 2020; Song et al., 2021; Vahdat et al., 2021) have empirically demonstrated that training the diffusion model with the likelihood weighting, λ ( t ) =
g 2 ( t ) , could bring worse performances than applying other weighting functions.
For instance, Ho et al.
; Song and
Ermon (2020) introduce the variance weighting function, λ ( t ) =
σ 2 ( t ) , and Ho et al.
(2020) show that this weighting outperforms the likelihood weighting in terms of the sample generation.
Within our framework, we also observe that the variance weighting function is advantageous over the likelihood weighting.
Henceforth, we optimize the ﬂow model and the diffusion model with different losses as follows:
[Equation]
Note that the ﬁrst term of the log Jacobian and the third term of the cross entropy in Eq.
7 are independent of the score parameters, θ .
Therefore, the diffusion parameters are not inﬂuenced by these terms, in which only the second term remains to update the diffusion parameters.
4. T IGHT V ARIATIONAL G AP OF P ARAMETRIZED F ORWARD D IFFUSION
This section theoretically analyzes the variational gap of PDM.
To construct the variational gap of PDM, we enumerate few nomenclatures for the path measures (see Table 6).
On the data space, µ d φ is the path measure for the inference process and ν d φ , θ is the path measure for the generative process.
On the latent space, µ l φ is the path measure for the inference process and ν l θ is the path measure for the generative process.
Here, ν l θ does not depend on φ because the generative diffusion on the latent space starts from the φ -independent noise distribution ( π ) and the diffusion proceeds the denoising process by following the score estimation of s θ ( z t , t ) that depends only on θ .
The variational gap on the data space becomes
[Equation]
9) where the subscript in Gap d represents the data space.
Recall that training the encoder in VAE guarantees the tighter variational gap because the encoder distribution of q φ ( x , z ) approaches toward the decoder distribution of p θ ( x , z ) by minimizing the NELBO of D KL (cid:0) q φ ( x , z ) (cid:107) p θ ( x , z ) (cid:1) .
However, this analogy is not valid in the variational gap in Eq. 8 because both inference and generative path measures depend on φ .
This means that the generative path measure could arbitrarily deviate from the inference path measure when we optimize φ .
Using the change of variable, however, the above variational gap is equivalent to
[Equation]
With this derivation, optimizing the invertible transformation guarantees the tighter variational gap because the generative measure of ν l θ is now ﬁxed on φ .
Also, we note that this variational gap reduces to the KL divergence between the inference posterior and the generative posterior by
[Equation]
which provides another analogy to VAE.
We enumerate the variational gaps of the explained models in Table 5.
In fact, we show in Theorem 2 that the optimum of PDM guarantees that the generative distribution p φ ∗ , θ equals to the data distribution p r , and the variational gap reduces to zero in the optimum.
See Appendix A for the proofs.
Theorem 2.
Suppose the ﬂow network is ﬂexible enough to transform p r to arbitrary continuous distribution.
For any θ , if ∇ · s θ ( · , t ) is symmetric for any t ∈ [0 , T ] and ∇ · log π ( · ) ≡ s θ ( · , T ) , then there exists φ ∗ such that Gap d ( µ d φ ∗ , ν d φ ∗ , θ ) = 0 .
Furthermore, we have
[Equation]
(cid:107) p φ , θ ) .
To the best of our knowledge, Theorem 2 is the ﬁrst analysis in the community of diffusion models that the theoretic optimum of the model completely recovers the data distribution.
To proceed the argument, let us consider the linear diffusion model in the data space with no ﬂow model.
Then, the marginal distribution of the forward process is p T at t = T , which differs from the prior distribution of π .
In addition, suppose that the diffusion model reaches to the optimum point, i.e., the score network estimates the exact data score by s θ ∗ ( x t , t ) =
∇ x t log p t ( x t ) .
{+Then, the generative transition probability equals to the groundtruth (reverse) transition probability of p T 
+}
( x 0 | x T ) .
Therefore, the generative distribution, p θ ∗ ( x 0 ) =
(cid:82) π ( x T ) p T 0 ( x 0 | x T )
d x T , naturally diverges from the real distribution, p r ( x 0 )
= (cid:82) p T ( x T ) p T 0 ( x 0 | x T ) d x T , because of the mismatch of p T and π .
This implies that the original linear diffusion model cannot estimate the real distribution, p r ( x 0 ) .
In contrast to the original diffusion model, applying normalizing ﬂows to the diffusion process enables
PDM to recover the real distribution by Theorem 2.
See Figures 8 and 9 for further illustration.
R ELATED W ORK
[Figure]
Recently, the diffusion model has been jointly modeled in conjunction with other deep generative models.
There are two major previous works on this line of research.
First, Vahdat et al.
(2021) (LSGM) in Figure 3 applies the diffusion model to the latent space of VAE.
Since VAE does not attain the invertible structure, LSGM is not suggesting the expansion of the diffusing mechanism.
In other words, LSGM cannot provide the diffused data learning at the middle of the process because LSGM does not induce the diffusion process on the data space due to the lack of VAE’s invertibility.
Consequently, the analyzing tool for LSGM should be the variational gap of VAE, rather than that of the diffusion model.
[Figure]
Second, Song et al.
(2021) (ScoreFlow) in Figure 4 simply concatenate the normalizing ﬂow on top of the diffusion model to provide the dequantized data distribution only at the initial diffusion time.
Despite of the structural similarity of ScoreFlow and PDM, the major difference of ScoreFlow and PDM lies on the differences between their density model construction: p θ in Scoreﬂow and p φ , θ in PDM.
ScoreFlow constructs p θ only with the diffusion model, so the ﬂow and the diffusion networks are not jointly combined in the loss function of (cid:82) p r ( x ) (cid:82) [0 , 1)
d q φ ( u | x ) log p θ ( x + u ) q φ ( u | x ) d u d x , where u is the dequantization variable that lies in a unit cube, [0 , 1) d .
In other words, ScoreFlow adopts the ﬂow outside of the model density, so the ﬂow is independent of the model density.
In fact, ScoreFlow can be best understood as a dequantization technique (Ho et al., 2019), whose goal is estimating the optimal dequantizing region on [Table] [Table] ) d to minimize the dequantization gap (Hoogeboom et al., 2020).
Therefore, the variational gap of ScoreFlow remains to be strictly positive because training the ﬂow does not optimize the forward diffusing mechanism.
On the other hand, PDM constructs the ﬂow-embedded model density of p φ , θ , and PDM optimizes the loss of (cid:82) p r ( x ) log (cid:82) [0 , 1) d p φ , θ ( x + u ) d u d x that puts the normalizing ﬂow inside of the model density.
Although we train PDM with the uniform dequantization (Theis et al., 2015) by default, we could apply the variational dequantization to PDM as well.
E XPERIMENTS
This section quantitatively and qualitatively analyzes the suggested PDM on CIFAR-10 (Krizhevsky et al., 2009) and CelebA (Liu et al., 2015) 64 × 64 .
We compute NLL for the density estimation in bits per dimension scale by solving the probability ﬂow (Song et al., 2020) backwards in time, and we compare the baseline performances of Song et al.
(2020; 2021) computed on the uniform dequantization.
For the sample generation, we use the Predictor-Corrector algorithm (Song et al., 2020) for PDM on VESDE, and we use the numerical ODE sampler of the probability ﬂow (Song et al., 2020) with RK45 method on VPSDE.
Throughout the experimental results, we use NCSN++ (VE) and DDPM++ (VP) (Song et al., 2020) as the backbones of the diffusion model, and a pair of
ResNet-based ﬂow models (Chen et al., 2019; Ma et al., 2020) as the backbones of the ﬂow model.
We give a full description on models for our experiments in Appendix C.
We name PDM (FID) and
PDM (NLL) for the experiments that use weighting functions as σ 2 and g 2 , respectively.
6. Q UANTITATIVE R ESULTS
We compare PDM to the baselines in Table 1 on VESDE and VPSDE as the linear diffusing mechanisms.
Although minor performance degradation on the sample performance, Table 1 empirically demonstrates the reduced variational gap.
In all experimental settings, the variational gaps signiﬁcantly drop less than 0.1.
On the other hand, the variational gaps of the baseline diffusion on VPSDE for the FID model and NLL model are 0.16 and 0.19, respectively.
We conclude that PDM effectively decreases the variational gap by training the parametrized inference path measure.
Table 2 compares the experiments on CelebA to the current state-of-the-art baseline models in various disciplines: the diffusion model of UDM (Kim et al., 2021) and DDGM (Nachmani et al., 2021), the VAE model of CR-NVAE (Sinha and Dieng, 2021), the ﬂow model of DenseFlow (Grci´c et al., 2021), and the GAN model of Styleformer (Park and Kim, 2021).
We observe a pair of implications.
First, training the diffusing mechanism with PDM largely reduces the variational gap.
In particular, PDM reduces this inference gap from 1.57 to 0.09 on the NCSN++ backbone with VESDE and from 0.40 to 0.05 on the DDPM++ backbone with VPSDE.
Also, we emphasize that NELBO of PDM largely outperforms NELBO of the baseline models.
Second, our PDM reports the state-of-the-art performance with 2.23 FID score in the CelebA sample generation.
This opposite trend on FID (compared to the results in CIFAR-10) indicates that the nonlinear diffusion is more effective on datasets of higher dimensions.
In fact, this is a quite intuitive result because as the data dimension increases, the family of feasible diffusion expands exponentially, and the data-optimal diffusing mechanism is more likely to deviate from linear diffusions. [Figure] [Figure] [Figure]
L( # params) NLL NELBO Gap
PDM 4(1.2M) 3.
ResFlow 5.
PDM 8(2.3M) 3.
ResFlow 4.
PDM 32(10.6M) 3.
[Figure]
[Figure]
[Figure]
ResFlow 3.
6. Q UALITATIVE R ESULTS
Test NELBO To ﬁgure out the effect of the nonlinear diffusing mechanism on the training procedure,
Figure 5 (a) shows test NELBO by training iteration on the diffusion model with 35M parameters.
Without any modiﬁcation on the hyperparameters, we train PDM with the shallow ResFlow (Chen et al., 2019) of 3M parameters, and Figure 5 (a) shows that the training on PDM surprisingly stabilizes test NELBO.
Figure 5 (b) illustrates that such training stability can be attributed to the nearly identical score scales.
As described in Figure 5 (b), the score scale on the data space is unbounded as the diffusion time goes to zero, and this extreme scale variation eventually leads to the training instability on the baseline diffusion model.
On the other hand, adopting the ﬂow transformation on the data space mitigates such extreme scale variations that destabilize the training.
Figure 6 shows test NELBO on the diffusion model with 107.6M number of parameters.
To stabilize the diffusion model, we apply the AdamW optimizer (Loshchilov and Hutter, 2017) with 0.01 weight decay, and Exponential Moving Average (Song et al., 2020) of 0.9999.
Using the ResFlow model with 10.6M parameters, Figure 6 indicates that PDM consistently outperforms the baseline diffusion in terms of NELBO with smaller variations.
Flow Depth The ﬂow model architecture is the building block that fundamentally determines the family of the inference path measures.
Table 4 empirically demonstrates that the inﬂexible variational family results in a large variational gap on CIFAR-10.
Additionally, the absolute magnitude of the NLL performance keeps decreasing as the number of layers (L) increases.
Sample Generation Figure 7 presents the samples from (a) the latent space and (b) the data space.
Since the sample generation on the latent space is unnormalized, we normalize the sampled latent variables to be bounded on the range of [0 , 255] , so we can visualize the latent information in image.
Figure 7 indicates that the diffusion model and the ﬂow model have distinctive and complementary roles in sample generation: 1) the diffusion model constructs the global context and 2) the ﬂow model enriches this global context by colorizing it into a realistic image.
C ONCLUSION
This paper expands the linear diffusing mechanism to be nonlinear by combining the invertible transformation and the diffusion model.
This nonlinear diffusing mechanism learns the forward diffusion process out of the variational family of inference path measures, and such optimization provides the tighter variational gap compared to the baseline model with linear diffusion.
E THICS S TATEMENT
As our work improves the sample quality, our methodology could be a source of fake datasets, i.e. fake images.
This is a common threat from the line of work in the deep generative modeling community.
R EPRODUCIBILITY S TATEMENT
Every proofs and derivations are explained in details on Appendix A and B. Also, for the reproducibility, we would release our code as soon as the discussion period opens through an anonymous repository to the reviewers.
We note that most of our code is built based on the released code of Song et al.
(2020) in order to compare with the baselines in a fair setting introduced in Song et al.
(2020) and Song et al.
Additionally, we provide the details on training/evaluation/neural architecture in Appendix C
A P ROOFS OF T HEOREM 1 AND T HEOREM
Theorem 1.
Suppose that p φ , θ ( x 0 ) is the log-likelihood of the generative random variable x φ , θ 0 .
Then, the log-likelihood is bounded by
[Equation]
where
[Equation]
with σ 2 ( t ) = 1 − e − (cid:82) t 0 β ( s )
d s + (cid:82) t
[ g 2 ( s ) − β ( s )] d s .
Proof of Theorem 1.
From the change of variable, the transformation of z 0 =
h φ ( x 0 ) induces
[Equation]
and thus the entropy of the data distribution becomes
[Equation]
From Theorem 4 of Song et al.
[Equation]
the entropy at t = 0 equals to
[Equation]
[Equation]
where f ( z t , t ) is the drift term of the diffusion for z t and p t is the probability distribution of z t .
Therefore, the negative log-likelihood becomes
[Equation]
Now, from Theorem 1 of Song et al.
[Equation]
the KL-divergence between the path measures becomes
[Equation]
so if we plug in this into the negative log-likelihood
[Equation]
we yield the following:
[Equation]
Also, we have
[Equation]
Therefore,
[Equation]
Now
[Equation]
since p 0 t ( z t | z 0 ) = N ( z t ; µ ( t ) z t , σ 2 ( t ) I ) for µ ( t ) and σ 2 ( t ) determined by β ( t ) and g ( t ) , we have
[Equation]
and we have the desired result
[Equation]
Theorem 2.
Suppose the ﬂow network is ﬂexible enough to transform p r to arbitrary continuous distribution.
For any θ , if ∇ · s θ ( · , t ) is symmetric for any t ∈ [0 , T ] and ∇ · log π ( · ) ≡ s θ ( · , T ) , then there exists φ ∗ such that Gap d ( µ d φ ∗ , ν d φ ∗ , θ ) = 0 .
Furthermore, we have
[Equation]
Proof of Theorem 2.
The symmetry of ∇ · s θ ( · , t ) indicates that s θ ( · , t ) is a closed 1-form.
Therefore, from the Poincare’s lemma (Do Carmo, 2012), s θ ( · , t ) is exact
[Equation]
and there exists a scalar ﬁeld f θ t that satisﬁes s θ ( · , t ) =
∇ · f θ t ( · ) for any t .
If we deﬁne by q θ t ( z t )
=
[Equation]
z , then we have
[Equation]
t , t ) = ∇ z t log q θ t ( z t , t ) for any t .
With this construction, the reverse SDE on the transformed latent space becomes
[Equation]
[Figure]
[Figure]
𝑟𝑟 ℎ 𝜙𝜙
𝜋𝜋
ℎ
𝜙𝜙 𝜈𝜈 𝜙𝜙 , 𝜃𝜃𝑑𝑑 𝑝𝑝 𝜙𝜙 , 𝜃𝜃 𝜇𝜇 𝜙𝜙𝑑𝑑 𝜇𝜇 𝜙𝜙𝑙𝑙 𝜈𝜈 𝜙𝜙 , 𝜃𝜃𝑙𝑙 𝑝𝑝 𝜃𝜃
[Figure] [Figure] If ρ θ is the path measure of the solution of the above SDE, then the marginal distribution of ρ θ at t would be q θ t .
Therefore, if h − 1 φ ∗ maps p r to q θ 0 , then the forward process on the latent space { z φ ∗ t } Tt =0 becomes the generative process on the latent space { z θ t } Tt =0 .
Thus, we have µ l φ ∗ =
ν l θ in distribution, which leads D KL ( µ l φ ∗ (cid:107) ν l θ )
= 0 .
We get the desired result by the change of variable formula: D KL ( µ d φ ∗ (cid:107) ν
d φ ∗ , θ ) = D KL ( µ l φ ∗ (cid:107) ν l θ ) = 0 .
B D ERIVATIONS B. D ERIVATION OF N ONLINEAR D IFFUSION T ERMS
Throughout this section
[Equation]
we omit φ for the notational simplicity.
Beginning with the linear diffusion on the latent space given by
[Equation]
we have deﬁned x t = h − 1 ( z t ) as the induced random variables.
Then by the multivariate Ito’s
Lemma (Oksendal, 2013) [Equation] the k -th component of the induced variable satisﬁes
[Equation]
Plugging the linear diffusion SDE 11 in the SDE of Eq.
12, Eq.
12 is derived by
[Equation]
because ∂ h − 1 k ∂t = 0 .
Then, Eq. 13 in vector form becomes
[Equation]
where the vector-valued drift and diffusion terms are given by
[Equation]
) where ∇ 2 z t h − 1 ( z t ) is a 3-dimensional tensor with ( i, j, k ) -th element to be (cid:0) ∇ 2 z t h − 1 k ( z t ) (cid:1) i,j , and the trace operator applied on this tensor results in the vector of (cid:104) tr (cid:0) ∇ 2 z t h [Equation] 1 ( z
[Equation] From the inverse function theorem (Rudin et al., 1964), the Jacobian of the inverse function ∇ z t h − 1 ( z t ) equals to the inverse Jacobian (cid:2) ∇ x t h ( x t ) (cid:3) − 1 .
Therefore, Eq. 14 is transformed to
[Equation]
Now, we derive the second term of f in terms of x t as follows: observe that (cid:80) k
∂ h − 1 i ∂ z t,k ∂
h k ∂ x t,j = δ i,j , where δ i,j = 1 if i = j and 0 otherwise.
Differentiating both sides with respect to x t,l , we have (cid:88) k (cid:26) ∂
∂ x t,l (cid:18) ∂ h − 1 i ∂ z t,k (cid:19)(cid:27)
∂
h k ∂ x t,j + ∂ h − 1 i ∂ z t,k (cid:26) ∂
∂ x t,l (cid:18)
∂
h k ∂ x t,j (cid:19)(cid:27) = 0 , where the ﬁrst term is (cid:88) k,m ∂ h m ∂ x t,l (cid:26) ∂
∂ z t,m (cid:18)
∂ h − 1 i ∂ z t,k (cid:19)(cid:27) ∂
h k ∂ x t,j = (cid:88) k,m (cid:0) ∇ x t h ( x t ) (cid:1) T l,m (cid:0) ∇ 2 z t h − 1 i ( z t ) (cid:1) m,k (cid:0) ∇ x t h ( x t ) (cid:1) k,j , using the chain rule, and the second term becomes (cid:88) k
∂ h − 1 i ∂ z t,k (cid:26) ∂
∂ x t,l (cid:18)
∂
h k ∂ x t,j (cid:19)(cid:27) =
(cid:88) k (cid:0) ∇
[Equation]
h [Equation] ( z [Equation]
i
[Equation]
∇ 2 x t h
[Equation]
( x t ) (cid:1) l,j .
From above, we derive the trace term of f in Eq.
15 as tr
[Equation]
∇
[Equation] (cid:2) ∇ x t h ( x t ) (cid:3) − 1 (cid:17) , [Table] where ∇ 2 x t h ( x t ) is a 3-dimensional tensor with ( i, j, k ) -th element to be (cid:0) ∇ 2 x t h k ( x t ) (cid:1) i,j .
Also, we deﬁne ∗ operation as the element-wise matrix multiplication given by
[Equation]
Combining all together, thus, we derive the nonlinear drift term in Eq.
15 as a function of x t given by
[Equation]
B. D ERIVATION OF V ARIATIONAL G AP
First, the KL divergence between two path measures satisﬁes
[Equation]
[Table]
where x 0: T = { x t } Tt =0 .
Therefore, the variational gap is derived by
Gap
[Equation]
The gap Gap d ( µ d φ , ν d φ , θ ) in Eq.
10 is derived to be
E z 0 (cid:104) D KL (cid:0) µ φ ( { z t }|
z
0 ) (cid:107) ν
θ ( { z t }| z 0 ) (cid:1)(cid:105) , analogously.
C I MPLEMENTATION D ETAILS C. M ODEL A RCHITECTURE
Diffusion Model We implement two diffusion models as backbone: NCSN++ (VE)
(Song et al., 2020) and DDPM++ (VP) (Song et al., 2020), where two backbones are one of the best performers in CIFAR-10 dataset.
In our setting, NCSN++ assumes the score network with parametrization of s θ ( z t , log σ 2 ( t )) , where σ 2 ( t ) =
σ 2 min ( σ max σ min ) 2 t is the variance of the transition probability p 0 t ( z t | z 0 ) with VESDE.
As introduced in Song et al.
(2020), we use the Gaussian Fourier embeddings (Tancik et al., 2020) to model the high frequency details across the temporal embedding.
DDPM++ models the score network with parametrization of (cid:15) θ ( z t , t ) , which targets to estimate − σ ( t ) ∇ z t log p t ( z t ) .
We use the Transformer sinusoidal temporal embedding (Vaswani et al., 2017).
We use the U-Net (Ronneberger et al., 2015) architecture for the score networks on both NCSN++ and DDPM++ based on (Ho et al., 2020).
We stack U-Net resblocks of up-and-down convolutions with skip connections that connect the identical-dimensional layers.
Also, we follow Ho et al.
by applying the global attention at the resolution of 16 × 16 .
We use four U-Net resblocks with four feature map resolutions ( 32 × 32 to 4 × 4 ).
There are eight and four resblocks at each resolution on
CIFAR-10 and CelebA, respectively.
Flow Model We implement two normalizing ﬂow models as backbone: the ResFlow (Chen et al., 2019) and the decoupled ResFlow (Ma et al., 2020).
Although Ma et al.
(2020) makes use of the
Glow (Kingma and Dhariwal, 2018) for their ﬂow model, we replace this glow architecture with the
ResFlow, mainly because the ResFlow transforms the variables from the identity function.
We name the experiments on this decoupled ResFlow as PDM (FID, dec) and PDM (NLL, dec).
Otherwise, we train the model with ResFlow as default.
For the ResFlow, we drop three components from the original paper: 1) the activation normalization Kingma and Dhariwal (2018), 2) the batch normalization Ioffe and Szegedy (2015), and 3) the fully connected layers.
On the deep ResFlow, we use resolutions of 32 × 32 and 16 × 16 on both CIFAR-10 and CelebA.
Otherwise, we use a single resolution of 32 × 32 on other experiments whose models have networks with layers less than 32.
We apply the LipSwish activation (Chen et al., 2019) that provides the
Lipschitz coefﬁcients less than a unity.
For the multi-GPU training, we use the Neumann logdeterminant gradient estimator, instead of the memory efﬁcient estimator (Chen et al., 2019).
C. E XPERIMENTAL DETAILS
Training All the models are trained with batch size of 128 and EMA rate of 0.999.
We train each model for around 10 days on four V-100 GPUs with 128Gb GPU memory for all experiments.
At the initial training phase, we train the diffusion model parameters with ﬁxed ﬂow parameters initialized by the identity until the sample generation performance saturates.
After this pretraining phase, we start updating both diffusion and ﬂow models.
In this learning phase, we apply the learning rate scheduling after the sample performance saturation.
For the diffusion model, we drop the learning rate from 2 × 10 − 4 to 10 − 5 .
For the ﬂow model, we drop the learning rate from 10 − 3 to 10 −for VPSDE and 5 × 10 − 5 to 10 − 5 for VESDE.
We emphasize this learning rate scheduling takes additional performance boost.
VESDE and VPSDE have different training details.
VESDE assumes σ min = 10 − 2 on all experiments.
VESDE has σ max = 50 on CIFAR-10 and σ max = 90 on CelebA.
On the other hand,
VPSDE assumes β ( t ) =
β min + ( β max − β min ) t with β min = 0 .
1 and β max = 20 .
Both VESDE and VPSDE truncate the diffusion time on [ (cid:15), T ] in order to stabilize the diffusion model, where (cid:15) =
10 − 5 and T = 1 .
On sample generation, however, DDPM++/PDM with VPSDE draws samples by proceeding the generative diffusion process up to (cid:15)
=
10 − 3 , which corresponds to the diffusion variance to be σ min = 10 − 2 .
On the sample generation, we apply the Predictor-Corrector (PC) sampler Song et al.
(2020) for NCSN++/PDM with VESDE and the Probability Flow (PF) ODE sampler Song et al.
(2020) for DDPM++/PDM.
Throughout the experiments, we provide the identical diffusion model structures to compare the baseline model and the PDM
Evaluation While we use the EMA rates of 0.999 for both diffusion and ﬂow models for training, we use this EMA only for the diffusion model on the evaluation mode because the ﬂow model without the EMA on the evaluation gives better samples.
One peculiarity occurs in the experiment on the decoupled ResFlow.
We empirically ﬁnd that turning off the EMA for both diffusion model and ﬂow model on the evaluation mode provides better performances of 3.11 in test NLL and 3.17 in test
NELBO as reported in Table 1 on PDM (FID, dec).
With the EMA applied to the diffusion model, we have 3.15 for test NLL and 3.22 for test NELBO, which shows slight degradation on the density estimation performance.
We compute the FID score based on Song et al.
(2020) for CIFAR-10 with the modiﬁed Inception V network 1 in order to compare PDM to the baselines (Song et al., 2020; 2021) in a fair setting.
On the other hand, for the CelebA dataset, we compute the clean-FID (Parmar et al., 2021) that provides consistently antialiased results.
C. V ARIANCE R EDUCTION
Flow Training When we train the ﬂow network with L φ , θ (cid:0) { x t } Tt =0 ; g 2 ) , this NELBO contains the integration of L θ (cid:0) { z t } Tt =0 ; g 2 (cid:1) .
However, previous works on the diffusion models (Nichol and Dhariwal, 2021) show that the estimation variance is largely reduced with the importance sampling, which leads the performance variations (Song et al., 2021).
Concretely, the importance sampling chooses an importance weight that is proportional to g 2 ( t ) σ 2 ( t ) , and estimates the integration by L θ (cid:0) { z t } Tt =0 ; g 2 (cid:1)
= (cid:82) T 0 g 2 ( t )
L t (cid:0) { z t } Tt =0 ; θ (cid:1)
d t ≈ (cid:80) Nn
=1 σ 2 ( t n ) L t n (cid:0) { z t } Tt =0 ; θ (cid:1) , where t n is sampled from the importance distribution.
For VESDE, it satisﬁes β ( t ) =
and g ( t ) =
σ min ( σ max σ min ) t (cid:113) 2 log ( σ max σ min ) .
Throughout the experiments, we select σ min = 0 .
01 , and, σ max = 50 , 90 for CIFAR-10 and CelebA, respectively.
The transition probability becomes p 0 t ( z t | z 0 )
=
N ( z t ; z 0 , σ 2 ( t )) , where σ 2 ( t ) =
(cid:82) t
g 2 ( s )
d s = σ 2 min (cid:2) ( σ max σ min ) 2 t − 1 (cid:3) .
Song et al.
(2020) approximates this variance to be σ 2 app ( t )
=
σ 2 min ( σ max σ min ) 2 t so that the variance is proportional to g 2 ( t ) .
Therefore, the importance weight follows the uniform distribution, and the importance sampling is equivalent with choosing the uniform t .
On the other hand, VPSDE satisﬁes β ( t ) =
β min + ( β max − β min ) t with g ( t ) =
(cid:112) β ( t ) .
Then, the transition probability becomes p 0 t ( z t | z 0 )
=
N ( z t ; µ ( t ) z t , σ 2 ( t ) I ) , where µ ( t ) =
e − 12 (cid:82) t 0 β ( s )
d s and σ 2 ( t )
= 1 − e − (cid:82) t 0 β ( s )d s .
Thus, VPSDE has the importance weight of g 2 ( t ) σ 2 (
[Equation]
The Monte-Carlo sample from this importance weight is the solution of the inverse Cumulative
Distribution Function (CDF) of the importance distribution as
[Equation]
where u is a uniform sample from [0 , 1] , F ( t ) is the antiderivative of the importance weight given by F ( t )
= log (1 − e
[Equation]
min )
+ 0 .
5 t 2 ( β max − β min ) + tβ min , and Z is the normalizing constant given by
[Equation]
for T = 1 and (cid:15) =
10 − 5 .
The solution for the inverse CDF in Eq.
16 becomes
[Equation]
The variation of the Monte-Carlo diffusion time depends on the uniform sample of u .
Diffusion Training Now, on the side of training the diffusion network, as described in the main paper, selecting the weighting function heavily inﬂuences the model performances.
For instance,
[Table]
[Table]
Song et al.
(2021) compare the weighting functions of g 2 and σ [Table] and Song et al.
(2021) conclude that the weighting function of λ ( t ) =
σ 2 ( t ) works the best for the sample generation [Table] and λ ( t ) =
g 2 ( t ) yields the best NLL performance.
We train the diffusion model with σ 2 ( t ) for PDM (VE/VP, FID), and with g 2 ( t ) for PDM (VP, NLL) models. D A DDITIONAL E XPERIMENTAL R ESULTS D. D ETAILED C OMPARISON AND A BLATION ON T OLERANCE [Figure] Table 7 has two implications.
First, we ﬁnd that the reported NLL and NELBO performances are slightly better than what we actually obtain in our pretraining step, denoted by DDPM++ (NLL, IS, ours). Comparing with our results, optimizing PDM is indeed beneﬁcial on density estimation, and the variational gap is signiﬁcantly reduced from 0.17 to 0.01. Second, we ﬁnd the tolerance of 10 − 1 for the numerical ODE solver on computing NLL is the sweet spot on obtaining best NLL of 2.86, which beats LSGM-480M (NLL) on density estimation. This is noteworthy because our model uses 120M number of parameters, a quarter of parameters in LSGM.
On the other hand, NELBO is independent of the tolerance because it does not use the numerical ODE solver, and Table 7 shows that the variational gap is minimized when the tolerance is small enough. Figure 10 illustrates the ablation study on the tolerance for the ODE solver. Surprisingly, PDM is robust on the tolerance level up to 10 − 1 , in contrast to LSGM with poor sample quality at 10 −(Vahdat et al., 2021). From this robustness on the tolerance, we could reduce the sampling wall clock time from 360s (tolerance 10 − 5 ) to 72s (tolerance 10 − 1 ) for 1,024 samples, which is 5x faster, with a minor sacriﬁce in FID score from 6.79 to 8.56 in PDM (FID) with VPSDE.
D. P RECISION AND R ECALL
While FID measures the overall sample quality that counts both ﬁdelity and diversity at once, we provide precision and recall (Kynkäänniemi et al., 2019) based on the modiﬁed Inception V1 network [Table] [Table] [Table]
[Table] 2 with 50k samples in order to separate the sample ﬁdelity and diversity. Precision is a metric that measures the sample ﬁdelity, and the recall is for the sample diversity. Table 8 indicates that both precision and recall remains at the identical scale, except for precision at PDM (FID, dec).
D. E FFECT OF S IGNAL - TO -N OISE AND T EMPERATURE
[Figure] T [Figure]
m p e r a t u r e ( )
As the ﬂow colorizes the latent sample, we experiment the effect of temperature (Kingma and Dhariwal, 2018) on this colorization in
Figure 11. With temperature τ , the normalizing ﬂow puts its latent input scaled by τ to the ﬂow network. In our PDM, the image color with a higher temperature tends to be brighter, and we ﬁnd that the optimal temperature depends on the experimental settings. On sample generation, we draw latent samples from either the PC sampling or the PF
ODE sampling.
PC sampling assumes the Signalto-Noise Ratio (SNR) that modiﬁes the Corrector algorithm (Song et al., 2020) of the annealed Langevin dynamics (Song and Ermon, 2019).
From the sampled latent variable, we scale by multiplying the temperature τ (Kingma and Dhariwal, 2018) to the latent variable.
Since the diffusion model learns the global image structure, the normalizing ﬂow only paints the color of the image, and the temperature controls the brightness of the generated image.
This is a contrastive result to the previous ﬂow models (Kingma and Dhariwal, 2018; Chen et al., 2019) that the temperature in the previous models controls both global context (such as the smoothness) and color.
Therefore, we can create images with different colors by manipulating this temperature.
CIFAR-10 With these hyperparameters of the SNR and the temperatures, we ﬁnd optimal values in the below Tables.
Table 9 computes the FID-5k score of all combinations for (SNR, τ ) in PDM (VE,
FID) on CIFAR-10.
Unlike the SNR of 0.16 that is previously suggested in (Song et al., 2020) for the vanilla diffusion model, we ﬁnd that SNR of 0.145 consistently outperforming other values for SNR in terms of FID.
Similarly, Table 9 presents that the optimal temperature is 1.1.
For VPSDE, Table 10 shows that the temperature of 1.3 performs the best for sample generation in
PDM (VP, FID).
For PDM (VP, FID, dec), we ﬁnd that the temperature of 1.1 performs the best, and for PDM (VP, NLL, dec), the vanilla temperature of 1.0 generates the best samples.
CelebA Table 11 shows the FID-1k performance of PDM (VE, FID) on CelebA.
It shows that the SNR of 0.17 proposed in (Song et al., 2020) performs the best also in our PDM.
In addition, it shows that the optimal temperature is either 1.0 or 1.05.
Table 12 shows the FID-1k performance of PDM (VP) on CelebA. It is clear in this table that the sample quality is the best on the temperature range of 1.05 to 1.15.
D. R ANDOM SAMPLES
Figure 12 and 13 show the non cherry-picked random samples from PDM (VE, FID) on CIFARand PDM (VP, FID) on CelebA, respectively.