P
SEUDO
N
UMERICAL M ETHODS FOR D IFFUSION M ODELS ON M ANIFOLDS
Anonymous authors
Paper under double-blind review
A BSTRACT
Denoising Diffusion Probabilistic Models (DDPMs) can generate high-quality samples such as image and audio samples.
However, DDPMs require hundreds to thousands of iterations to produce ﬁnal samples.
Several prior works have successfully accelerated DDPMs through adjusting the variance schedule (e.g.,
Improved Denoising Diffusion Probabilistic Models) or the denoising equation (
e.g., Denoising Diffusion
Implicit Models (DDIMs)).
However, these acceleration methods cannot maintain the quality of samples and even introduce new noise at a high speedup rate, which limit their practicability.
To accelerate the inference process while keeping the sample quality, we provide a fresh perspective that
DDPMs should be treated as solving differential equations on manifolds.
Under such a perspective, we propose pseudo numerical methods for diffusion models (PNDMs).
Speciﬁcally, we ﬁgure out how to solve differential equations on manifolds and show that DDIMs are simple cases of pseudo numerical methods.
We change several classical numerical methods to corresponding pseudo numerical methods and ﬁnd that pseudo linear multi-step method is the best method in most situations.
According to our experiments, by directly using pre-trained models on
Cifar10, CelebA and LSUN, PNDMs can generate higher quality synthetic images with only 50 steps compared with 1000-step DDIMs (20x speedup), signiﬁcantly outperform DDIMs with 250 steps (by around 0.4 in FID) and have good generalization on different variance schedules.
I NTRODUCTION
Denoising Diffusion Probabilistic Models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020) is a class of generative models which model the data distribution through an iterative denoising process reversing a multi-step noising process.
DDPMs have been applied successfully to a variety of applications, including image generation (Ho et al., 2020;
Song et al., 2020b; Nichol & Dhariwal, 2021), text generation (Hoogeboom et al., 2021;
Austin et al., 2021), 3D point cloud generation (Luo & Hu, 2021), text-to-speech (Kong et al., 2021; Chen et al., 2020) and image super-resolution (Saharia et al., 2021).
Unlike Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), which require careful hyperparameter tuning according to different model structures and datasets, DDPMs can use similar model structures and be trained by a simple denoising objective which makes the models ﬁt the noise in the data.
To generate samples, the iterative denoising process starts from white noise and progressively denoises it into the target domain according to the noise predicted by the model at every step.
However, a critical drawback of DDPMs is that DDPMs require hundreds to thousands of iterations to produce high-quality samples and need to pass through a network at least once at every step, which makes the generation of a large number of samples extremely slow and infeasible.
In contrast, GANs only need one pass through a network.
There have been many recent works focusing on improving the speed of the denoising process.
Some works search for better variance schedules, including Nichol & Dhariwal (2021) and Watson et al.
Some works focus on changing the inference equation, including Song et al.
(2020a)
and
Song et al.
(2020b).
Denoising Diffusion
Implicit Models (DDIMs) (Song et al., 2020a) relying on a non-Markovian process accelerate the denoising process by taking multiple steps every iteration.
Probability Flows (PFs) (Song et al., 2020b) build a connection between the denoising process and solving ordinary differential equations and use numerical methods of differential equations to accelerate the denoising process of DDPMs.
Additional, we introduce more related works in Appendix
A.1.
However, this direct connection between DDPMs and numerical methods (e.g., forward Euler method, linear multi-step method and RungeKutta method (Timothy, 2017)) has weaknesses in both speed and effect (see Section 3.1).
Some numerical methods are straightforward and simple, like forward Euler method, but they can only trade quality for speed.
Some numerical methods can accelerate the reverse process without loss of quality, like Runge-Kutta method, but they need to propagate forward more times along a neural network at every step.
Furthermore, we also notice that numerical methods can introduce noticeable noise at a high speedup rate, limiting their practicability.
To ﬁgure out the reason for the performance degradation in classical numerical methods, we ﬁnd that classical numerical methods may sample data far away from the main distribution area of the data, and the inference equations of DDPMs do not satisfy a necessary condition of numerical methods at the last several steps (see Section 3.2).
To tackle these problems, we design new numerical methods called pseudo numerical methods for diffusion models (PNDMs) to generate samples along a speciﬁc manifold in R n , which is the highdensity region of the data.
We ﬁrst give a rigorous formula transformation to build a theoretical connection between DDPMs and numerical methods directly and self-consistently.
Considering that classical numerical methods cannot guarantee to generate samples on certain manifolds, we provide brand-new numerical methods called pseudo numerical methods based on our theoretical analyses and formula transformation.
We ﬁnd that DDIMs are simple cases of pseudo numerical methods, which means that we also provide a new way to understand DDIMs better.
Furthermore, we search for the most suitable numerical method for diffusion models and ﬁnd that pseudo linear multi-step method is the fastest method under similar generated quality.
Besides, we provide a detailed theoretical analysis of our new theory and give visualization results to support our theory intuitively.
According to our experiments, our methods have several advantages:
• Our methods only need to change the inference equation in the code and use pre-trained models directly regardless of the size of the images and the type of the datasets.
• Our methods can make DDIMs 20x faster without loss of quality on Cifar10 and CelebA.
Furthermore, we theoretically prove that our new methods PNDMs are second-order convergent while DDIMs are ﬁrst-order convergent.
• Our methods can reduce the best FID of pre-trained models with even shorter sampling time.
With only 250 steps, our new denoising process can reduce the best FID by around 0.4 points Cifar10 and CelebA.
We achieve a new SOTA FID score of 2.71 on CelebA. •
Our methods work well with different variance schedules, which means that our methods have a good generalization and can be used together with those works introducing better variance schedules to accelerate the denoising process further.
B ACKGROUND
In this section, we introduce some backgrounds.
Firstly, we present the classical understanding of
DDPMs.
Then we provide another understanding based on Song et al.
(2020b), which inspires us to use numerical methods to accelerate the denoising process of diffusion models.
After that, we introduce some background on numerical methods that will be used later in this paper.
2. D ENOISING D IFFUSION P ROBABILISTIC M ODELS
DDPMs model the data distribution from Gaussian distribution to image distribution through an iterative denoising process.
Let x 0 be an image, then the diffusion process is a Markov process and the reverse process has a similar form to the diffusion process, which satisﬁes:
[Equation]
Here, β t controls the speed of adding noise to the data, calling them the variance schedule.
N is the total number of steps of the denoising process.
µ θ and β θ are two neural networks, and θ are their parameters.
Ho et al.
(2020) get some statistics estimations of µ θ and β θ .
According to the properties of the conditional Gaussian distribution, we have:
[Equation]
Here, α t = 1 − β t , ¯ α t = ∏ ti =
[Equation]
i , ¯
[Equation]
x t and ¯ β t =
[Equation]
¯
[Equation]
t
[Equation]
1 − ¯ α t β t .
Then this paper sets β θ = ¯ β t and designs a objective function to help neural networks to represent µ θ .
Objective Function
The objective function is deﬁned by:
[Equation]
Here, x t ( x 0 , ϵ ) =
√ ¯ α t x 0 + √ 1 − ¯ α t ϵ, ϵ ∼ N (0 , 1) , ϵ θ is an estimate of the noise ϵ .
The relationship between µ θ and ϵ θ is µ θ = 1 √ α t ( x t − β t √ 1 − ¯
α t ϵ θ ) .
Because ϵ ∼ N (0 , 1) , we assume that the mean and variance of ϵ θ are 0 and 1.
2. S TOCHASTIC D IFFERENTIAL E QUATION
According to Song et al.
(2020b);
Anderson (1982), there is another understanding of DDPMs.
The denoising process of DDPMs satisﬁes a certain stochastic differential equation:
[Equation]
This is Variance Preserving stochastic differential equations (VP-SDEs).
Here, we change the domain of t from [1 , N ] to [0 , 1] .
When N tends to inﬁnity, { β i } Ni =1 , { x i } Ni =1 become continuous functions β ( t ) and x ( t ) on [0 , 1] .
Song et al.
(2020b) also show that this equation has an ordinary differential equation (ODE) version with the same marginal probability density as Equation (4):
[Equation]
This is Probability Flows (PFs), which remove the random item of VP-SDEs.
These two equations show us a new possibility that we can use numerical methods to accelerate the reverse process.
As far as we know, DDIMs ﬁrst try to remove this random item, so PFs can also be treated as an acceleration of DDIMs, while VP-SDEs are an acceleration of DDPMs.
2. N UMERICAL M ETHOD
Many classical numerical methods can be used to solve ODEs, including forward Euler method,
Runge-Kutta method and linear multi-step method (Timothy, 2017).
Forward Euler Method For a certain differential equation satisfying dxdt =
f ( x, t ) .
The trivial numerical method is forward Euler method satisfying x t + δ = x t + δf ( x t , t ) .
Runge-Kutta Method Runge-Kutta method uses more information at every step, so it can achieve higher accuracy 2 .
Runge-Kutta method satisﬁes:
[Equation]
Linear Multi-Step Method Linear multi-step method is another numerical method and satisﬁes:
[Equation]
P SEUDO N UMERICAL M ETHOD FOR DDPM
In this section, we ﬁrst give a rigorous formula transformation to build a direct connection between
DDPMs and numerical methods.
As a byproduct, we can directly use pre-trained models from
DDPMs.
After establishing this connection, we dive into the structure of numerical methods by dividing their equations into a gradient part and a transfer part and providing more detailed analyses on the weakness of classical numerical methods.
To solve the problems in classical numerical methods, we deﬁne pseudo numerical methods by introducing nonlinear transfer parts and ﬁnd that
DDIMs can be regarded as simple pseudo numerical methods.
Then, We explore the pros and cons of different numerical methods and choose linear multi-step method to make numerical methods faster.
Finally, we summarize our ﬁndings and analyses and safely propose our novel pseudo numerical methods for diffusion models (PNDMs), which combine our proposed transfer part and the gradient part of linear multi-step method.
Furthermore, we analyze the convergence order of pseudo numerical methods to demonstrate the effectiveness of our methods theoretically.
3. F ORMULA T RANSFORMATION
According to Song et al.
(2020a), the reverse process of DDPMs and DDIMs satisﬁes:
[Equation]
Here, σ t controls the ratio of random noise.
If σ t equals one, Equation (8) represents the reverse process of DDPMs; if σ t equals zero, this equation represents the reverse process of DDIMs.
And only when σ t equals zero, this equation removes the random item and becomes a discrete form of a certain ODE.
Theoretically, the numerical methods can be used on differential equations with random items are limited.
And Song et al.
(2020b) have done enough research in this case.
Empirically,
Song et al.
(2020a) have showed that DDIMs have better acceleration effect when the number of total steps is relatively small.
Therefore, our work concentrate on the case σ t equals zero.
To ﬁnd the corresponding ODE of Equation (8), we replace discrete t − 1 with a continuous version t − δ according to (Song et al., 2020a) and change this equation into a differential form, namely, subtract x t from both sides of this equation:
[Equation]
[Equation]
Because δ is a continuous variable from 0 to t , we can now compute the derivative of the generation data x t and get that lim δ → 0 x t − x t − δ δ
[Equation]
Here, ¯ α ( t ) is the continuous version of { ¯ α i } Ni =1 like the deﬁnition of x ( t ) .
Therefore, the corresponding ODE when δ tends to zero of Equation (9) is:
[Equation]
Equation (10) is the continuous version of DDIMs.
In this paper, our main target is ﬁnding better numerical methods to solve this ODE better and faster, and we successfully ﬁnd new numerical methods which have high convergence order and good behavior even when the number of steps is relatively small.
3. G RADIENT AND T RANSFER P ART OF N UMERICAL M ETHOD
Ernst & Gerhard (1996) show several numerical methods to solve differential equations on manifolds that have analytic expressions.
In our problem, target manifolds are the high-density region of the data x t of DDPMs, which is deﬁned by x t ( x 0 , ϵ ) =
√ ¯ α t x 0 + √ 1 − ¯ α t ϵ , ϵ ∼ N (0 , 1) .
Unfortunately, it’s challenging to use this expression of manifolds, while we do not know the target x 0 in the reverse process and random items ϵ are hard to handle, too.
To avoid this problem, we design a different way to solve differential equations on these manifolds.
We divide the classical numerical methods into two parts, a gradient part and a transfer part.
The gradient part determines the gradient at each step, while the transfer part generates the result at the next step and keeps it on target manifolds.
For example, linear multi-step method can be divided into the gradient part f ′ =
δ 24 (55 f t − 59 f t − δ + 37 f t − 2 δ − 9 f t − 3 δ ) and the transfer part x t + δ = x t + δf ′ .
All classical numerical methods have the same transfer part while their gradient parts are different.
[Figure]
n
[Figure]
However, this ﬁxed transfer part cannot keep the results on certain manifolds.
Equation (2) shows that the data x t is generated along a curve close to an arc.
According to Figure 2, most of x t is concentrated in a band with a width of around 0.1, namely the red area in Figure 2.
This means that the neural network ϵ θ cannot get enough examples away from this area.
Therefore, if the denoising process samples an example x ′ t away from the main area and uses x ′ t to compute the noise ϵ θ ( x ′ t , t ) at the next step, the prediction will be inaccurate.
In Section 4.3 we will give more visualization results to support this.
There is another problem that limits the effectiveness of numerical methods mentioned in Section 2.3.
We ﬁnd that for most linear variance schedules β t ,
Equation (10) tends to inﬁnity when t tends to zero, namely Equation (10) is unbounded (see Appendix A.4), which does not satisfy the condition of numerical methods mentioned in Section 2.3.
This is an apparent theoretical weakness that previous works have not explored.
However, in the original DDPMs and DDIMs, the prediction of the sample x t and the noise ϵ θ in the data are more and more precise as the index of step t tends to zero (see Appendix A.5).
This means the discrete version of DDPMs does not make a signiﬁcant error in the last several steps.
In contrast, using numerical methods of the continuous version will introduce obvious error when t tends to zero.
These two problems lead us to change the transfer part to solve our target unbounded ODE Equation (10).
3. P SEUDO N UMERICAL M ETHOD
We deﬁne that those numerical methods which use a nonlinear transfer part as pseudo numerical methods.
And an expected transfer part should have the property that when the result from the gradient part is precise, then the result of the transfer part is as close to the manifold as possible and the error of this result is as small as possible.
We ﬁnd that Equation (9) satisﬁes such property.
Property 3.1 If ϵ is the precise noise in x t , then the result of x t − δ from Equation (9) is also precise.
And we put the proof of this property in Appendix A.5.
Therefore, we use:
[Equation]
as the transfer part and ϵ θ as the gradient part.
That if ϵ θ is precise, the result of x t − δ is also precise, which means that ϵ θ can determine the direction of the denoising process to generate the ﬁnal results.
Therefore, such a choice also satisﬁes the deﬁnition of a gradient part.
Now, we have our gradient part ϵ
θ
and transfer part ϕ .
This combination solves the two problems mentioned above successfully.
Firstly, our new transfer parts do not introduce new errors.
This property also means that it keeps the results at the next step on the target manifold because generating samples away is a kind of error.
This shows that we solve the ﬁrst problem.
Secondly, we know that the prediction of ϵ θ is more and more precise in the reverse process in the above subsection.
And our new transfer part can generate precise results according to the precise prediction of ϵ θ .
Therefore, our generation results are more and more precise using pseudo numerical methods, while classical numerical methods can introduce obvious error at the last several steps.
This shows that we solve the second problem, too.
We also ﬁnd that their combination ϕ ( x t , ϵ θ ( x t , t ) , t, t − 1) is just the inference equation used by DDIMs, so DDIMs is a simple case of pseudo numerical methods.
Here, we deﬁne DDIMs as DDIMs*, emphasizing that it is a pseudo numerical method.
Because we split numerical methods into two parts, we can use the same gradient part from different classical numerical methods freely (e.g., linear multi-step method and Runge-Kutta method), although we change the transfer part of our inference equation.
Our theoretical analyses and experiments show that the gradient part from different classical methods can work well with our new transfer part (see Section 3.6, 4.2).
By using the same gradient part of linear multi-step method, we have:
[Equation] [Equation]
By using the same gradient part of Runge-Kutta method, we have:
[Equation] [Equation]
1: x T ∼ N (0 , I ) 2: for t = T − 1 , · · · , 1 , 0 do 3: x t = ϕ
( x t +1 , ϵ θ
( x t +1 , t + 1) , t + 1 , t ) 4: end for 5: return x
Algorithm
1: x T ∼ N (0 , I ) 2: for t = T − 1 , T − 2 , T − 3 do 3: x t , e t =
PRK ( x t +1 , t + 1 , t ) 4: end for 5: for t =
T − 4 , · · · , 1 , 0 do 6: x t , e t =
PLMS ( x t +1 , { e p } p>t , t +1 , t ) 7: end for 8: return x
Abbreviate Equation (12) and (13) as x t + δ , e t = PLMS ( x t , { e p } p<t , t, t + δ ) , x t + δ , e 1 t =
PRK ( x t , t, t + δ ) .
3. G RADIENT P ART
In the above section, we have provided three kinds of pseudo numerical methods.
Although advanced numerical methods can accelerate the denoising process, some may have to compute the gradient part ϵ θ more times at every step, like the Runge-Kutta method.
Propagating forward four times along a neural network makes the denoising process slower.
However, we ﬁnd that linear multi-step method can reuse the result of ϵ θ four times and only compute ϵ θ once at every step.
And theoretical analyses tell us that Runge-Kutta method and linear multi-step method have the same convergence order and should have similar generated results.
[Table]
Therefore, we use the gradient part of linear multi-step method and our new transfer part as our main pseudo numerical methods for diffusion models (PNDMs).
In Table 1, we show the relationship between different numerical methods.
Here, we can see PNDMs combine the beneﬁts of higher-order classical numerical methods (in the gradient part) and DDIMs (in the transfer part).
3. A LGORITHM
We can provide our whole algorithm of the denoising process of DDIMs now.
According to Song et al.
(2020a), the algorithm of the original method satisﬁes Algorithm 1.
And our new algorithm of
DNPMs uses pseudo linear multi-step method and pseudo Runge-Kutta method, which satisﬁes Algorithm 2.
Here, we cannot use linear multi-step at the beginning because linear multi-step method cannot start automatically, which needs at least three previous steps’ information to generate results.
So we use Runge-Kutta method to compute the ﬁrst three steps’ results and then use linear multi-step method to calculate the remaining.
We also use the gradient parts of two second-order numerical methods to get another pseudo numerical method.
We introduce the details of this method in Appendix A.3.
We call it S-PNDMs, because its gradient part uses information from two steps at every step.
Similarly, we also call our ﬁrst PNDMs F-PNDMs, which use data from four steps, when we need to distinguish them.
3. C ONVERGENCE O RDER
Change the transfer part of numerical methods may introduce unknown error.
To determine the inﬂuence of our new transfer part theoretically, we compute the local and global error between the theoretical result of Equation (10) x
( t + δ ) and our new methods, we ﬁnd that x ( t + δ ) − x DDIM ( x + δ )
=
O ( δ 2 ) and
[Equation] [Equation]
If the target ODE satisﬁes Lipschitz condition and local error e local = O ( δ k ) , then there are C and h such
that the global error e global satisﬁes e global ≤ Cδ k (1 + e h
+ e 2 h + · · · ) ≤ C ′
δ k − 1 .
And we have that the convergence order is equal to the order of the global error.
The detailed proof can be found in Appendix A.6.
Therefore, we get the following property:
Property 3.2 S/F-PNDMs have third order local error and are second order convergent.
E XPERIMENT 4. S ETUP
We conduct unconditional image generation experiments on four datasets: Cifar10 ( 32 × 32 ) (Krizhevsky et al., 2009), CelebA ( 64 × 64 )
(Liu et al., 2015), LSUN-church ( 256 × 256 ) and
LSUN-bedroom ( 256 × 256 ) (Yu et al., 2016).
According to the analysis in Section 3.1, we can use pre-trained models from prior works in our experiments.
The pre-trained models for Cifar10, LSUNchurch and LSUN-bedroom are taken from Ho et al.
and the pre-trained model for CelebA is taken from Song et al.
(2020a).
In these models, the number of total steps N is 1000 and the variance schedule is linear variance schedule.
And we also use a pre-trained model for Cifar10, which uses a cosine variance schedule from improved denoising diffusion probabilistic models (iDDPMs (Nichol & Dhariwal, 2021)).
4. S AMPLE E FFICIENCY AND Q UALITY
To analyze the acceleration effect, we test Fenchel Inception Distance (FID (Heusel et al., 2018)) on different datasets under different steps and different numerical methods, including DDIMs,
S-PNDMs, F-PNDMs and classical fourth-order numerical methods (FONs) (e.
Kutta
[Table]
method and linear multi-step method).
On Cifar10 and CelebA, we ﬁrst provide the results of previous works DDIMs.
Then, we use the same pre-trained models to test numerical methods mentioned in this paper and put the results in Cifar10 / CelebA (linear).
We also use models from iDDPMs to test nonlinear variance schedules and put the results in Cifar10 (cosine).
Song et al.
(2020b) do not provide detailed FID results of probability ﬂows (PFs) under different steps, so we retest the results using its pretrained models by ourselves.
Our two baselines are DDIM and PF.
DDIM is a simple case of pseudo numerical methods and PF is a case of classical numerical methods.
However, PF uses a much bigger model than
DDIM and uses some tricks to improve the sample quality.
To ensure the experiment’s fairness, we use fourth-order numerical methods on Equation (10) and the model from DDIM.
In Table 2, we ﬁnd that the performance of FON is limited when the number of steps is small.
By contrast, our new methods, including S-PNDM and F-PNDM, can improve the generated results regardless of whether the number of steps used is large or small.
According to Cifar10 / CelebA (linear), F-PNDM can achieve lower FID than 1000 steps DDIM using only 50 steps, making diffusion models 20x faster without losing quality.
[Figure]
We draw a line chart of computation cost with FID according to the results of Cifar10 (linear) above in Figure 3.
Because F-PNDM uses pseudo Runge-Kutta method to generate the ﬁrst three steps, it is slower than other methods at the ﬁrst several steps.
Therefore, S-PNDM can achieve the best FID initially, then F-PNDM becomes the best and the acceleration is signiﬁcant.
Quality When the number of steps is relatively big, the results of FON become more and more similar to that of pseudo numerical methods.
This is because all the methods are solving Equation (10) and their convergent results should be the same.
However, pseudo numerical methods still work better when using a large number of steps empirically.
F-PNDM can improve the best FID around 0.4 using pre-trained models and achieves a new SOTA FID score of 2.71 on CelebA, which shows that our work can not only accelerate diffusion models but also improve the sample quality topline.
We also notice that the FID results of F-PNDM converge after more than 250 steps.
The FID results will ﬂuctuate around a value then.
This phenomenon is more pronounced when we test our methods on LSUN (see Table 5, 6).
According to Cifar10 (cosine), cosine variance schedule can make FID lower using a relatively big number of steps.
More analyses about variance schedule can be found in Appendix A.7.
What’s more, we test our methods on other datasets and provide our FID results in Appendix A.9 and image results in Appendix A.10.
We can draw similar conclusions on our methods’ acceleration and sampling quality, regardless of the datasets and the size of the images.
[Figure]
4. S AMPLE ON M ANIFOLDS
Here, we design visualization experiments to show the effort of our new methods and support our analyses.
Because it is hard to visualize high-dimensional data, we use the change of a global characteristic norm and a local characteristic pixel to show the change of the data under different steps.
For pixel, we randomly choose two positions p 1 , p 2 .
Then for a series of images x T , x T − k , · · · , xderived from the reverse process, we denote y kt as the value of x t at position p k .
Then we draw a polyline ( y 1 t , y 2 t ) t
= T, ··· in R 2 .
For norm, we ﬁrst count the distribution of the norm of the training datasets under different steps and use this to make a heat map as the background.
After that, we draw the norm of our generated results using different methods and steps above this heat map.
In Figure 4, we can see that the FON may run far away from the high-density area of the data, which explains why FON may introduce noticeable noise.
However, PNDM can avoid this problem and ﬁt the target result more appropriately.
More visualization results supporting our analysis can be found in Appendix A.11.
What’s more, we design a toy example to test our new methods without the inﬂuence of neural networks and get similar conclusions as the real cases above.
We put the detailed results in Appendix A.8.
D ISCUSSION
In this paper, we provided DNPMs, a new numerical method suitable for solving the corresponding
ODEs of DDPMs.
DNPMs can generate high-quality images using fewer steps without loss of quality successfully.
Based on the idea of this work, further improvement can be explored in our future works: 1) ﬁnd a better variance schedule for DNPMs: although we tested DNPMs on linear variance schedule and cosine variance schedule in this work, there might be another variance schedule more suitable for our proposed numerical methods. 2)
Find higher-order convergent pseudo numerical methods: we analyzed the convergence order of S/F-DNPMs, which are both second-order convergent.
However, F-DNPMs achieve better FID than S-DNPMs in most cases.
We think this is because the result between our transfer part and target ODE has a higher-order error, which limits the convergence order of F-DNPMs.
This error from the change of the transfer part is theoretical but does not inﬂuence the quality of images according to the property of Equation (11).
However, making the transfer part higher-order convergent and ﬁnding the inﬂuence of such change is still exciting and need more research.
3) Extend PNDMs to more general applications: when we proved the convergence order of DNPMs, we found that other kinds of transfer parts can keep the convergence order unchanged too, which means that pseudo numerical methods can be used on more general applications, like certain neural ODEs (Chen et al., 2019; Dupont et al., 2019).
A A PPENDIX A. R ELATED W ORK
DDPMs have been well developed in the last few years.
Some works concentrate on improving the quality and the speed of DDPMs and making DDPMs more practical.
Song et al.
(2020a) introduce new inference equations to accelerate DDPMs.
Nichol & Dhariwal (2021), Watson et al.
(2021) and
Kong & Ping (2021) choose to ﬁnd better variance schedules to improve the images quality.
Vahdat et al.
(2021) combine the advantages of DDPMs and Variational Autoencoders and get better results.
Furthermore, Kim et al.
(2021) try to solve an existing bottleneck that the inference equations of
DDPMs are unbounded in some situations.
Song et al.
(2020b)
ﬁnd the similarity between DDPMs and noise conditional score networks (NCSNs (Song & Ermon, 2020b)), which is that they both use a process similar to Langevin dynamics to produce samples.
Therefore, some works (Song & Ermon, 2020a; Kim et al., 2021) that can improve the results of NCSNs also can be used in DDPMs.
Additional, Song et al.
(2020b) combine DDPMs and NCSNs under the framework of neural differential equations (Chen et al., 2019;
Dupont et al., 2019).
Therefore, numerical methods widely used in neural differential equations can also be applied to accelerate DDPMs.
Our work successfully combines the advantages of Song et al.
(2020a) and Song et al. (2020b).
We use a transfer part from DDIMs and use different gradient parts from different numerical methods.
Although we and Song et al.
(2020b) solve certain differential equations derived from DDPMs, we use different target different equations and different numerical methods, which get better results.
The application of DDPMs is not limited to unconditional image generation.
Some works apply
DDPMs to various types of data successfully, including text-to-speech (Chen et al., 2020; Lam et al
A. C ONVERGENT O RDER OF M ETHOD
We use the forward Euler method and linear multi-step method to show what is the order of a method.
Assume that x t is precise and compute the error at x t + δ .
For forward Euler method, we have:
[Equation]
Here, we assume that f ′′ is continuous, so f ′′ is bounded in a close area.
x ( t + δ ) is the precise result and x t,δ is the numerical result from t to t + δ .
For linear multi-step method, we have:
[Equation]
Here, { b s } satisfy ∑ 4 s =1 b s = 1 and the following equations for j ∈ { 1 , · · · , 3 } :
[Equation]
We call the error at x t,δ local error, and the error at x t + Mδ
(M is big enough but ﬁnite) global error.
Assume the local error of our method has order k+1 and the target ODE satisﬁes Lipschitz condition, then:
[Equation]
Therefore, the global error will be one order lower than the local error.
From Equation (15), we can see the forward Euler method has local error O ( δ 2 ) and global error O ( δ ) , so we call it ﬁrst-order numerical method.
And linear multi-step method has local error O
( δ 5 ) and global error O ( δ 4 ) , and we call it fourth-order numerical method.
In addition, assuming that a numerical method has kth-order global error, we can compute the convergence speed of this numerical method:
[Equation]
Here, x δt + T is the result at t+T and move δ every step.
This shows that fourth-order method can converge to the exact solution faster than ﬁrst-order method when δ → 0 , which means that we can use bigger iteration interval δ to achieve similar global error and a bigger iteration interval means that we can iterate fewer times to get results with high quality.
A. P SEUDO S ECOND -O RDER M ETHOD
We introduce two second-order numerical methods.
First is improved Euler method satisfying:
[Equation]
Second is another linear multi-step method called second-order linear multi-step method satisfying:
[Equation]
And the corresponding pseudo improved Euler methods satisfying:
[Equation]
Pseudo second-order linear multi-step method satisfying:
[Equation]
Similar to what we do to get F-PNDMs, We combine them to get S-PNDMs.
Abbreviate Equation (22) and (23) as x t + δ
[Equation]
e t
[Equation] [Equation]
t
[Equation]
e p } p
[Equation]
t , t, t + δ
[Equation]
x t + δ
[Equation]
e 1 t
[Equation]
PLMS ′
[Equation]
t
[Equation]
δ ) .
1: x T ∼ N (0 , I ) 2: for t = T − 1 do 3: x t , e t =
PIE ( x t +1 , t + 1 , t ) 4: end for 5: for t =
T − 2 , · · · , 1 , 0 do 6: x t , e t = PLMS ′
( x t +1 , { e p } p>t , t +1 , t ) 7: end for 8: return x
A. T HE E XISTENCE OF A D ERIVATIVE
Because ¯ α t is usually obtained by multiplying a linear variance schedule β t .
So we have
[Equation]
and ¯ α 0 = 1 , so c = 0 .
Now we have
[Equation]
To make lim δ → 0 x t − δ − x t δ | t =0 is well-deﬁned, b must equal to zero, or (2 at + b )
ϵ θ ( x t ,t ) √
1 − ¯ α t will tend to inﬁnity.
This is a strong condition that most variance schedules do not satisfy.
In practice, DDPMs can choose the variance schedule very freely.
This means that treating DDPMs as ODEs directly is not proper and has theoretical weakness.
A. R ELATIONSHIP BETWEEN t , ϵ θ AND x t
Relationship between t and ϵ θ .
In Figure 5, we can see that the denoising process tends to converge, whether in the ϵ θ domain or the sample/image domain when the step-index tends to zero.
Therefore, we can say that the noise becomes more and more precise when step, namely t , tends to zero.
[Figure]
assume that
[Equation]
the neural network ϵ θ =
( x t , t ) .
Because we assume that the gradient part is precise, then we have ϵ θ =
ϵ .
Then for all t ′ ≤ t , we have:
[Equation]
Here, we can ﬁnd that x t ′ =
√ ¯ α t ′ x 0 + √ 1 − ¯ α t ′
ϵ is also precise, so Property 3.1 is true.
A. O RDER A NALYSIS OF P SEUDO M ETHOD
For the convenience of theoretical analysis, we generalize the problem.
Let ϕ ( x ( t ) , ϵ, t, δ ) =
f ( x ( t ) , t, δ ) + g ( t, δ ) ϵ
( x ( t ) , t ) and we have the property f ( x ( t ) , t, 0) =
g ( t, 0) = 0 .
Then we have:
[Equation]
Now, Equation (10) becomes a special case of this more general version and, in this special case
[Equation]
we have:
[Equation]
Now, we compute the local error of S-PNDMs.
We ﬁrst compute the theoretical and numerical results of different numerical methods.
We have:
[Equation]
and
[Equation]
Then we compute the difference between the theoretical and numerical results.
We have:
[Equation]
In this special case
[Equation]
we compute the derivatives of some items needed in Equation (31).
We have:
[Equation]
Now we can compute the ﬁnal result of Equation (31).
We split it into three parts and the values of the ﬁrst two terms.
We have:
[Equation]
=
[Equation]
39) and
[Equation] [Equation]
Finally, we get the ﬁnal result of Equation (31):
[Equation]
And the computation of the convergence order of F-PNDMs is similar, and we ignore it here.
Therefore, Property 3.2 is true.
A. V ARIANCE SCHEDULE
According to Cifar10 (cosine) in Table 2, PNDMs can be used on both linear variance schedule and cosine variance schedule.
However, we also notice cosine variance schedule can make FID lower when we use relatively big generation steps, but the effort is limited when the number of steps is small.
F-PNDM uses information from four consecutive steps, so the smoothness of the schedule is more important for F-PNDM than DDIM.
According to this experiment, our work can be used with works that pay attention to variance schedules to improve the acceleration effect further.
However, a variance schedule that ﬁts pseudo numerical methods better remains to be found in further work.
A. T OY EXAMPLE
Here, we design a toy example to test our new methods without the inﬂuence of neural networks.
We randomly generate the initial input x 1 = ( m 1 , m 2 ) , m i ∼ U (0 , 1) and use a simple analytic equations ϵ θ ( x ) =
(sin x [0] , cos x [1]) to replace the neural networks in real cases.
Let ϕ in Equation (11) is unchanged and ¯ α t
= α ( t ) = 1 − t , then we get:
[Equation]
[Equation]
Here, we use three different numerical methods to generate x 0 .
For DDIM, we have:
[Equation]
For FON, according to Equation (10), we have:
[Equation]
For F-PNDM, we have:
[Equation]
Then we draw the corresponding generation curves in Figure 6.
We ﬁnd that the result is similar to the real cases.
The main difference here is that FON can correct its results while the real case cannot.
The reason is that the gradient is well-deﬁned everywhere, while in real cases, the gradient is meaningful on the high-density region of the data x t of DDPMs.
A. M ORE FID R ESULTS
Here, We provide our more detailed FID results on Cifar10, CelebA, LSUN-church and LSUNbedroom in Table 3, 4, 5, 6.
[Figure]
steps
[Figure]
DDIM 13.
DDIM* 44.
[Table] [Table]
[Table] [Table]
[Table]
A. M ORE I MAGE R ESULTS
Here, we show more generated images on Cifar10, CelebA, LSUN-church and LSUN-bedroom in
Figure 7, 9, 8, 10, 11, 12.
[Figure]
A. M ORE V ISUALIZATION R ESULTS
Here, we put more visualization results similar to Figure 4 in Figure
[Figure]
A. FID RESULT WITH S TANDARD D EVIATION
Here, we report the mean and standard deviation of FID results, tested over four sampling runs.
[Figure]
Cifar10(linear)
[Figure]
* 18.50 ± .
10.86 ± .
6.95 ± .
5.49 ± .
[Table]