DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
Abstract
Diffusion probabilistic models (DPMs) are emerging powerful generative models.
Despite their high-quality generation performance, DPMs still suffer from their slow sampling as they generally need hundreds or thousands of sequential function evaluations (steps) of large neural networks to draw a sample.
Sampling from
DPMs can be viewed alternatively as solving the corresponding diffusion ordinary differential equations (ODEs).
In this work, we propose an exact formulation of the solution of diffusion ODEs.
The formulation analytically computes the linear part of the solution, rather than leaving all terms to black-box ODE solvers as adopted in previous works.
By applying change-of-variable, the solution can be equivalently simpliﬁed to an exponentially weighted integral of the neural network.
Based on our formulation, we propose DPM-Solver , a fast dedicated high-order solver for diffusion ODEs with the convergence order guarantee.
DPM-Solver is suitable for both discrete-time and continuous-time DPMs without any further training.
Experimental results show that DPM-Solver can generate high-quality samples in only 10 to 20 function evaluations on various datasets.
We achieve 4.70 FID in 10 function evaluations and 2.87 FID in 20 function evaluations on the
CIFAR10 dataset, and a 4 ∼ 16 × speedup compared with previous state-of-the-art training-free samplers on various datasets.
Introduction
Diffusion probabilistic models (DPMs) [1–3] are emerging powerful generative models with promising performance on many tasks, such as image generation [4, 5], video generation [6], text-to-image generation [7], speech synthesis [8, 9] and lossless compression [10].
DPMs are deﬁned by discretetime random processes [1, 2] or continuous-time stochastic differential equations (SDEs) [3], which learn to gradually remove the noise added to the data points.
Compared with the widely-used generative adversarial networks (GANs) [11] and variational auto-encoders (VAEs) [12], DPMs can not only compute exact likelihood [3], but also achieve even better sample quality for image generation [4].
However, to obtain high-quality samples, DPMs usually need hundreds or thousands of sequential steps of large neural network evaluations, thereby resulting in a much slower sampling speed than the single-step GANs or VAEs.
Such inefﬁciency is becoming a critical bottleneck for the adoption of
DPMs in downstream tasks, leading to an urgent request to design fast samplers for DPMs. [Figure]
Existing fast samplers for DPMs can be divided into two categories.
The ﬁrst category includes knowledge distillation [13, 14] and noise level or sample trajectory learning [15–18].
Such methods require a possibly expensive training stage before they can be used for efﬁcient sampling.
Furthermore, their applicability and ﬂexibility might be limited.
It might require nontrivial effort to adapt the method to different models, datasets, and number of sampling steps.
The second category consists of training-free [19–21] samplers, which are suitable for all pre-trained DPMs in a simple plug-andplay manner.
Training-free samplers include adopting implicit [19] or analytical [21] generation process, advanced differential equation (DE) solvers [3, 20, 22–24] and dynamic programming [18].
However, these methods still require ∼ 50 function evaluations [21] to generate high-quality samples (comparable to those generated by plain samplers in about 1000 function evaluations), thereby are still time-consuming.
In this work, we bring the efﬁciency of training-free samplers to a new level to produce high-quality samples in the “ few-step sampling ” regime, where the sampling can be done within around 10 steps of sequential function evaluations.
We tackle the alternative problem of sampling from DPMs as solving the corresponding diffusion ordinary differential equations (ODEs) of DPMs, and carefully examine the structure of diffusion ODEs.
Diffusion ODEs have a semi-linear structure — they consist of a linear function of the data variable and a nonlinear function parameterized by neural networks.
Such structure is omitted in previous training-free samplers [3, 20], which directly use black-box
DE solvers.
To utilize the semi-linear structure, we derive an exact formulation of the solutions of diffusion ODEs by analytically computing the linear part of the solutions, avoiding the corresponding discretization error.
Furthermore, by applying change-of-variable, the solutions can be equivalently simpliﬁed to an exponentially weighted integral of the neural network.
Such integral is very special and can be efﬁciently approximated by the numerical methods for exponential integrators [25].
Based on our formulation of solutions, we propose DPM-Solver , a fast dedicated solver for diffusion
ODEs by approximating the above integral.
Speciﬁcally, we propose ﬁrst-order, second-order and third-order versions of DPM-Solver with convergence order guarantees.
We further propose an adaptive step size schedule for DPM-Solver.
In general, DPM-Solver is applicable to both continuoustime and discrete-time DPMs, and also conditional sampling with classiﬁer guidance [4].
Fig.demonstrates the speedup performance of a Denoising Diffusion Implicit Models (DDIM) [19] baseline and DPM-Solver, which shows that DPM-Solver can generate high-quality samples with as few as 10 function evaluations and is much faster than DDIM on the ImageNet 256x256 dataset [26].
Our additional experimental results show that DPM-Solver can greatly improve the sampling speed of both discrete-time and continuous-time DPMs, and it can achieve excellent sample quality in around 10 function evaluations, which is much faster than all previous training-free samplers of DPMs.
Diffusion Probabilistic Models
We review diffusion probabilistic models and their associated differential equations in this section.
2.1 Forward Process and Diffusion SDEs
Assume that we have a D -dimensional random variable x 0 ∈ R D with an unknown distribution q 0 ( x 0 ) .
Diffusion Probabilistic Models (DPMs) [1–3, 10] deﬁne a forward process { x t } t ∈ [0 ,T ] with
T > 0 starting with x 0 , such that for any t ∈ [0 , T ] , the distribution of x t conditioned on x 0 satisﬁes
[Equation]
where α ( t ) , σ ( t ) ∈ R + are differentiable functions of t with bounded derivatives, and we denote them as α t , σ t for simplicity.
The choice for α t and σ t is referred to as the noise schedule of a
DPM.
Let q t ( x t ) denote the marginal distribution of x t , DPMs choose noise schedules to ensure that q T ( x T )
≈ N ( x T | 0 , ˜
σ 2 I ) for some ˜ σ > 0 , and the signal-to-noise-ratio (SNR) α 2 t /σ 2 t is strictly decreasing w.r.t.
t [10].
Moreover, Kingma et al.
[10] prove that the following stochastic differential equation (SDE) has the same transition distribution q 0 t ( x t | x 0 ) as in Eq.
(2.1) for any t ∈ [0 , T ] :
[Equation]
where w t ∈ R D is the standard Wiener process, and
[Equation]
Under some regularity conditions, Song et al.
[3] show that the forward process in Eq.
(2.2) has an equivalent reverse process from time T to 0 , starting with the marginal distribution q T ( x T ) :
[Equation]
where ¯ w t is a standard Wiener process in the reverse time.
The only unknown term in Eq.
(2.4) is the score function ∇ x log q t ( x t ) at each time t .
In practice, DPMs use a neural network (cid:15) θ ( x t , t ) parameterized by θ to estimate the scaled score function: − σ t ∇ x log q t ( x t ) .
The parameter θ is optimized by minimizing the following objective [2, 3]:
[Equation]
where ω ( t ) is a weighting function, (cid:15) ∼ q ( (cid:15) )
=
N ( (cid:15) | 0 , I ) , x t = α t x 0 + σ t (cid:15) , and C is a constant independent of θ .
As (cid:15) θ ( x t , t ) can also be regarded as predicting the Gaussian noise added to x t , it is usually called the noise prediction model .
Since the ground truth of (cid:15) θ ( x t , t ) is − σ t ∇ x log q t ( x t ) ,
DPMs replace the score function in Eq.
(2.4) by − (cid:15) θ ( x t , t ) /σ
t and deﬁne a parameterized reverse process ( diffusion SDE ) from time T to 0 , starting with x T ∼ N ( 0 , ˜
σ 2 I ) :
[Equation]
Samples can be generated from DPMs by solving the diffusion SDE in Eq.
(2.5) with numerical solvers, which discretize the SDE from T to 0 .
Song et al.
[3] proved that the traditional ancestral sampling method for DPMs [2] can be viewed as a ﬁrst-order SDE solver for Eq.
However, these ﬁrst-order methods usually need hundreds of or thousands of function evaluations to converge [3], leading to extremely slow sampling speed.
2.2 Diffusion (Probability Flow) ODEs
When discretizing SDEs, the step size is limited by the randomness of the Wiener process [27, Chap.
A large step size (small number of steps) often causes non-convergence, especially in high dimensional spaces.
For faster sampling, one can consider the associated probability ﬂow ODE [3], which has the same marginal distribution at each time t as that of the SDE.
Speciﬁcally, for DPMs,
Song et al.
[3] proved that the probability ﬂow ODE of Eq.
(2.4) is
[Equation]
where the marginal distribution of x t is also q t ( x t ) .
By replacing the score function with the noise prediction model, Song et al.
[3] deﬁned the following parameterized ODE ( diffusion ODE ):
[Equation]
Samples can be drawn by solving the ODE from T to 0 .
Comparing with SDEs, ODEs can be solved with larger step sizes as they have no randomness.
Furthermore, we can take advantage of efﬁcient numerical ODE solvers to accelerate the sampling.
Song et al.
[3] used the RK45 ODE solver [28] for the diffusion ODEs, which generates samples in ∼ 60 function evaluations to reach comparable quality with a 1000-step SDE solver for Eq.
(2.5) on the CIFAR-10 dataset [29].
However, existing general-purpose ODE solvers still cannot generate satisfactory samples in the few-step ( ∼ 10 steps) sampling regime.
To the best of our knowledge, there is still a lack of training-free samplers for
DPMs in the few-step sampling regime, and the sampling speed of DPMs is still a critical issue.
Customized Fast Solvers for Diffusion ODEs
As highlighted in Sec. 2.2, discretizing SDEs is generally difﬁcult in high dimensions [27, Chap.
and it is hard to converge within few steps.
In contrast, ODEs are easier to solve, yielding a potential for fast samplers.
However, as mentioned in Sec. 2.2, the general black-box ODE solver used in previous work [3] empirically fails to converge in few steps.
This motivates us to design a dedicated solver for diffusion ODEs to enable fast and high-quality few-step sampling.
We start with a detailed investigation of the speciﬁc structure of diffusion ODEs.
3.1 Simpliﬁed Formulation of Exact Solutions of Diffusion ODEs
The key insight of this work is that given an initial value x s at time s > 0 , the solution x t at each time t < s of diffusion ODEs in Eq.
(2.7) can be simpliﬁed into a very special exact formulation which can be efﬁciently approximated.
Our ﬁrst key observation is that a part of the solution x t can be exactly computed by considering the particular structure of diffusion ODEs.
The r.h.s.
of diffusion ODEs in Eq.
(2.7) consists of two parts: the part f ( t ) x t is a linear function of x t , and the other part g 2 ( t ) 2 σ t (cid:15) θ ( x t , t ) is generally a nonlinear function of x t because of the neural network (cid:15) θ ( x t , t ) .
This type of ODE is referred to as semi-linear
ODE.
The black-box ODE solvers adopted by previous work [3] are ignorant of this semi-linear structure as they take the whole h θ ( x t , t ) in Eq.
(2.7) as the input, which causes discretization errors of both the linear and nonlinear term.
We note that for semi-linear ODEs, the solution at time t can be exactly formulated by the “variation of constants” formula [30]:
[Equation]
This formulation decouples the linear part and the nonlinear part.
In contrast to black-box ODE solvers, the linear part is now exactly computed, which eliminates the approximation error of the linear term.
However, the integral of the nonlinear part is still complicated because it couples the coefﬁcients about the noise schedule (i.e., f ( τ ) , g ( τ ) , σ τ ) and the complex neural network (cid:15) θ , which is still hard to approximate.
Our second key observation is that the integral of the nonlinear part can be greatly simpliﬁed by introducing a special variable.
Let λ t :=
log( α t /σ t ) (one half of the log-SNR), then λ t is a strictly decreasing function of t (due to the deﬁnition of DPMs as discussed in Sec. 2.1).
We can rewrite g ( t ) in Eq.
(2.3) as
[Equation]
Combining with f ( t ) =
d log α t / d t in Eq.
(2.3), we can rewrite Eq.
(3.1) as
[Equation]
As λ ( t ) =
λ t is a strictly decreasing function of t , it has an inverse function t λ ( · ) satisfying t = t λ ( λ ( t )) .
We further change the subscripts of x and (cid:15) θ from t to λ and denote ˆ x λ := x t λ ( λ ) , ˆ (cid:15) θ
(ˆ x λ , λ ) := (cid:15)
θ ( x t λ ( λ ) , t λ ( λ )) .
Rewrite Eq.
(3.3) by “change-of-variable” for λ , then we have:
Proposition 3.1 (Exact solution of diffusion ODEs) .
Given an initial value x s at time s > 0 , the solution x t at time t ∈ [0 , s ] of diffusion ODEs in Eq.
(2.7) is: [Equation]
We call the integral (cid:82) e − λ ˆ (cid:15) θ
(ˆ x λ , λ )d λ the exponentially weighted integral of ˆ (cid:15) θ , which is very special and highly related to the exponential integrators in the literature of ODE solvers [25].
To the best of our knowledge, such formulation has not been revealed in prior work of diffusion models.
Eq.
(3.4) provides a new perspective for approximating the solutions of diffusion ODEs.
Speciﬁcally, given x s at time s , According to Eq.
(3.4), approximating the solution at time t is equivalent to directly approximating the exponentially weighted integral of ˆ (cid:15) θ from λ s to λ t , which avoids the error of the linear terms and is well-studied in the literature of exponential integrators [25, 31].
Based on this insight, we propose fast solvers for diffusion ODEs, as detailed in the following sections.
3.2 High-Order Solvers for Diffusion ODEs
In this section, we propose high-order solvers for diffusion ODEs with convergence order guarantee by leveraging our proposed solution formulation Eq.
The proposed solvers and analysis are highly motivated by the methods of exponential integrators [25, 31] in the ODE literature.
{+Speciﬁcally, given an initial value x T at time T and M + 1 time steps { t i } Mi =0 decreasing from t 
+}
= T to t M = 0 .
{+Let ˜ x t 
+}
= x T be the initial value.
The proposed solvers use M steps to iteratively compute a sequence { ˜ x t i } Mi =0 to approximate the true solutions at time steps { t i } Mi =0 .
In particular, the last iterate ˜ x t M approximates the true solution at time 0 .
In order to reduce the approximation error between ˜ x t M and the true solution at time 0 , we need to reduce the approximation error for each ˜ x t i at every step [30].
Starting with the previous value ˜ x t i −at time t i − 1 , according to Eq.
(3.4), the exact solution x t i − 1 → t i at time t i is given by
[Equation]
Therefore, to compute the value ˜ x t i for approximating x t i − 1 → t i , we need to approximate the exponentially weighted integral of ˆ (cid:15) θ from λ t i − 1 to λ t i .
Denote h
i :=
λ t i − λ t i − 1 , and ˆ (cid:15) ( n ) θ (ˆ x λ , λ ) : = d n ˆ (cid:15) θ (ˆ x λ ,λ
[Equation]
d λ n as the n -th order total derivative of ˆ (cid:15) θ (ˆ x λ , λ ) w.r.t.
λ . For k ≥ 1 , the ( k − 1) -th order Taylor expansion of ˆ (cid:15) θ (ˆ x λ , λ ) w.r.t. λ at λ t i − 1 is
[Equation] Substituting the above Taylor expansion into Eq. (3.5) yields
[Equation]
where the integral (cid:82) e − λ ( λ − λ ti − 1 ) n n !
d λ can be analytically computed by repeatedly applying n times of integration-by-parts (see Appendix B.2). Therefore, to approximate x t i − 1 → t i , we only need to approximate the n -th order total derivatives ˆ (cid:15) ( n ) θ (ˆ x λ , λ ) for n ≤ k − 1 , which is a well-studied problem in the ODE literature [31, 32]. By dropping the O ( h k +1 i ) error term and approximating the ﬁrst ( k − 1) -th total derivatives with the “stiff order conditions” [31, 32], we can derive k -th-order
ODE solvers for diffusion ODEs. We name such solvers as DPM-Solver overall, and DPM-Solver- k for a speciﬁc order k . Here we take k = 1 for demonstration. In this case, Eq.
(3.6) becomes [Equation]
By dropping the high-order error term O ( h 2 i ) , we can obtain an approximation for x t i − 1 → t i . As k = 1 here, we call this solver DPM-Solver-1 , and the detailed algorithm is as following.
DPM-Solver-1.
Given an initial value x T and M + 1 time steps { t i } Mi {+=
+} decreasing from t 
 = T to t M = 0 . Starting with ˜ x t 0 = x T , the sequence { ˜ x t i } Mi =1 is computed iteratively as follows:
[Equation] h [Equation] For k ≥ 2 , approximating the ﬁrst k terms of the Taylor expansion needs additional intermediate points between t and s [31].
The derivation is more technical so we defer it to Appendix B. Below we propose algorithms for k = 2 , 3 and name them as DPM-Solver-2 and DPM-Solver-3 , respectively. Require: initial value x T , time steps { t i } Mi =0 , model (cid:15) θ 1: ˜ x t [-
-]0 ← x T 2: for i ← 1 to M do 3: s i ← t λ (cid:16) λ ti [Equation] 1 +
[Equation]
ti(cid:17) 4: u
[Equation]
←
[Equation]
) 5: ˜ x
[Equation]
←
[Equation]
(cid:15)
[Equation]
( u
[Equation]
, s
[Equation]
6: end for 7: return ˜ x
t M Require: initial value x T , time steps { t i } Mi =0 , model (cid:15) [Table] 1: ˜
[Table]
x
[Table]
, r
[Table]
13 , r
[Table]
2: for i ← 1 to M do
[Table]
[Table]
(cid:0) λ
[Table]
i
[Table]
+ r
[Table]
h
[Table]
(cid:1) , s
[Table]
t
[Table]
(cid:0) λ t i −
[Table]
+ r
[Table]
h
[Table]
(cid:1) 4: u
[Table]
D
[Table]
[Table]
D
[Table]
← (cid:15)
[Table]
[Table]
: ˜
[Table]
←
[Table]
[Table]
[Table]
9: end for 10: return ˜ x
t M
Here, t λ ( · ) is the inverse function of λ ( t ) , which has an analytical formulation for the practical noise schedule used in [2, 16], as shown in Appendix D. The chosen intermediate points are ( s i , u i ) for
{+DPM-Solver-2 and ( s 
+}
i − 1 , u 2 i − 1 ) and ( s 2 i , u 2 i ) for DPM-Solver-3.
As shown in the algorithm,
DPM-Solver- k requires k function evaluations per step for k = 1 , 2 , 3 .
Despite the more expensive steps, higher-order solvers ( k = 2 , 3 ) are usually more efﬁcient since they require much fewer steps to converge, due to their higher convergence order.
We show that DPM-Solver- k is k -th-order solver, as stated in the following theorem.
The proof is in Appendix B.
Theorem 3.2 (DPM-Solver- k as a k -th-order solver) .
Assume (cid:15) θ
( x t , t ) follows the regularity conditions detailed in Appendix B.1, then for k = 1 , 2 , 3 , DPM-Solver- k is a k -th order solver for diffusion
ODEs, i.e., for the sequence { ˜ x t i } M i =1 computed by DPM-Solver- k , the approximation error at timesatisﬁes ˜ x
t M − x 0 =
O ( h k max ) , where h max = max 1 ≤ i ≤ M ( λ t i − λ t
i − 1 ) .
Finally, solvers with k ≥ 4 need much more intermediate points as shown by previous work [31, 32] for exponential integrators.
Therefore, we only consider k from 1 to 3 in this work, while leaving the solvers with higher k for future study. 3.3 Step Size Schedule The proposed solvers in Sec. 3.2 need to specify the time steps { t i } Mi =0 in advance. We propose two choices of the time step schedule.
One choice is handcrafted, which is to uniformly split the interval [ λ T , λ 0 ], i.e. λ t i = λ T + iM ( λ 0 − λ T ) , i = 0 , . . , M . Note that this is different from previous work [2, 3] which chooses uniform steps for t i .
Empirically, DPM-Solver with uniform time steps λ t i can already generate quite good samples in few steps, where results are listed in Appendix E.
As the other choice, we propose an adaptive step size algorithm, which dynamically adjusts the step size by combining different orders of DPM-Solver. The adaptive algorithm is inspired by [20] and we defer its implementation details to Appendix C. For few-step sampling, we need to use up all the number of function evaluations (NFE). When the
NFE is not divisible by 3 , we ﬁrstly apply DPM-Solver-3 as much as possible, and then add a single step of DPM-Solver-1 or DPM-Solver-2 (dependent on the reminder of K divided by 3 ), as detailed in Appendix D. In the subsequent experiments, we use such combination of solvers with the uniform step size schedule for NFE ≤ 20 , and otherwise the adaptive step size schedule. 3.4 Sampling from Discrete-Time DPMs Discrete-time DPMs [2] train the noise prediction model at N ﬁxed time steps { t n } Nn =1 , and the noise prediction model is parameterized by ˜ (cid:15) θ ( x n , n ) for n = 0 , . . .
, N − 1 , where each x n is corresponding to the value at time t n +1 . We can transform the discrete-time noise prediction model to the continuous version by letting (cid:15) θ ( x , t ) : = ˜ (cid:15) θ ( x , ( N − 1) t T ) , for all x ∈ R d , t ∈ [0 , T ] .
Note that the input time of ˜ (cid:15) θ may not be integers, but we ﬁnd that the noise prediction model can still work well, and we hypothesize that it is because of the smooth time embeddings (e.g., position embeddings [2]). By such reparameterization, the noise prediction model can adopt the continuous-time steps as input, and thus we can also use DPM-Solver for fast sampling.  Comparison with Existing Fast Sampling Methods Here, we discuss the relationship and highlight the difference between DPM-Solver and existing
ODE-based fast sampling methods for DPMs.
We further brieﬂy discuss the advantage of training-free samplers over those training-based ones. 4.1 DDIM as DPM-Solver-  Denoising Diffusion Implicit Models (DDIM) [19] design a deterministic method for fast sampling from DPMs. For two adjacent time steps t i − 1 and t i , assume that we have a solution ˜ x t i − 1 at time t i − 1 , then a single step of DDIM from time t i − 1 to time
t i is [Equation] Although motivated by entirely different perspectives, we show that the updates of DPM-Solverand Denoising Diffusion Implicit Models (DDIM) [19] are identical.
By the deﬁnition of λ , we have σ ti −α ti − 1 = e − λ ti − 1 and σ ti α ti = e − λ ti . Plugging these and h i =
λ t i − λ t i − 1 to Eq.
(4.1) results in exactly a step of DPM-Solver-1 in Eq.
However, the semi-linear ODE formulation of DPM-Solver allows for principled generalization to higher-order solvers and convergence order analysis. Recent work [13] also show that DDIM is a ﬁrst-order discretization of diffusion ODEs by differentiating both sides of Eq. However, they cannot explain the difference between DDIM and the ﬁrst-order Euler discretization of diffusion ODEs. In contrast, by showing that DDIM is a special case of DPM-Solver, we reveal that DDIM makes full use of the semi-linearity of diffusion ODEs, which explains its superiority over traditional Euler methods.
4.2 Comparison with Traditional Runge-Kutta Methods
One can obtain a high-order solver by directly applying traditional explicit Runge-Kutta (RK) methods to the diffusion ODE in Eq.
Speciﬁcally, RK methods write the solution of Eq.
(2.7) in the [Table] following integral form: [Equation]
and use some intermediate time steps between [ t, s ] and combine the evaluations of h θ at these time steps to approximate the whole integral.
The approximation error of explicit RK methods depends on h θ , which consists of the error corresponding to both the linear term f ( τ ) x τ and the nonlinear noise prediction model (cid:15) θ .
However, the error of the linear term may increase exponentially because the exact solution of the linear term has an exponential coefﬁcient (as shown in Eq.
There are many empirical evidence [25, 31] showing that directly using explicit RK methods for semi-linear ODEs may suffer from unstable numerical issues for large step size.
We also demonstrate the empirical difference of the proposed DPM-Solver and the traditional explicit RK methods in Sec. 5.1, which shows that DPM-Solver have smaller discretization errors than the RK methods with the same order.
4.3 Training-based Fast Sampling Methods for DPMs
Samplers that need extra training or optimization include knowledge distillation [13, 14], learning the noise level or variance [15, 16, 33], and learning the noise schedule or sample trajectory [17, 18].
Although the progressive distillation method [13] can obtain a fast sampler within 4 steps, it needs further training costs and loses part of the information in the original DPM (e.g., after distillation, the noise prediction model cannot predict the noise (score function) at every time step between [0 , T ] ).
In contrast, training-free samplers can keep all the information of the original model, and thereby can be directly extended to the conditional sampling by combining the original model and an external classiﬁer [4] (e.g. see Appendix D for the conditional sampling with classiﬁer guidance).
Beyond directly designing fast samplers for DPMs, several works also propose novel types of
DPMs which supports faster sampling.
For instance, deﬁning a low-dimensional latent variable for
DPMs [34]; designing special diffusion processes with bounded score functions [35]; combining
GANs with the reverse process of DPMs [36].
The proposed DPM-Solver may also be suitable for accelerating the sampling of these DPMs, and we leave them for future work.
Experiments
In this section, we show that as a training-free sampler, DPM-Solver can greatly speedup the sampling of existing pre-trained DPMs, including both continuous-time and discrete-time ones, with both linear noise schedule [2, 19] and cosine noise schedule [16].
We vary different number of function evaluations (NFE) which is the number of calls to the noise prediction model (cid:15) θ ( x t , t ) , and compare the sample quality between DPM-Solver and other methods.
For each experiment, We draw 50K samples and use the widely adopted FID score [37] to evaluate the sample quality, where lower FID usually implies better sample quality.
Unless explicitly mentioned, we always use the solver combination with the uniform step size schedule in Sec.
3.3 if the NFE budget is less than 20, and otherwise the DPM-Solver-3 with the adaptive step size schedule in Sec. We refer to Appendix D for other implementation details of DPM-Solver and Appendix E for detailed settings. [Figure]
5.1 Comparison with Continuous-Time Sampling Methods
We ﬁrstly compare DPM-Solver with other continuous-time sampling methods for DPMs.
The compared methods include the Euler-Maruyama discretization for diffusion SDEs [3], the adaptive step size solver for diffusion SDEs [20] and the RK methods for diffusion ODEs [3, 28] in Eq.
We compare these methods for sampling from a pre-trained continuous-time “VP deep” model [3] on the CIFAR-10 dataset [29] with the linear noise schedule.
Fig.
2a shows the efﬁciency of compared solvers.
We use uniform time steps with 50, 200, 1000 NFE for the diffusion SDE with Euler discretization, and vary the tolerance hyperparameter [3, 20] for the adaptive step size SDE solver [20] and RK45 ODE solver [28] to control the NFE.
DPM-Solver can generate good sample quality within around 10 NFE, while other solvers have large discretization error even in 50 NFE, which shows that DPM-Solver can achieve ∼ 5 speedup of the previous best solver.
In particular, we achieve 4.70 FID with 10 NFE, 3.75 FID with 12 NFE, 3.24 FID with
NFE, and 2.87 FID with 20 NFE, which is the fastest sampler on CIFAR-10.
As an ablation study, we also compare the second-order and third-order DPM-Solver and RK methods, as shown in Table 1.
We compare RK methods for diffusion ODEs w.r.t.
both time t in Eq.
(2.7) and half-log-SNR λ by applying change-of-variable (see detailed formulations in Appendix E.1).
The results show that given the same NFE, the sample quality of DPM-Solver is consistently better than
RK methods with the same order.
The superior efﬁciency of DPM-Solver is particularly evident in the few-step regime under 15 NFE, where RK methods have rather large discretization errors.
This is mainly because DPM-Solver analytically computes the linear term, avoiding the corresponding discretization error.
Besides, the higher order DPM-Solver-3 converges faster than DPM-Solver-2, which matches the order analysis in Theorem 3.2.
5.2 Comparison with Discrete-Time Sampling Methods
We use the method in Sec.
3.4 for using DPM-Solver in discrete-time DPMs, and then compare
DPM-Solver with other discrete-time training-free samplers, including DDPM [2], DDIM [19],
Analytic-DDPM [21], Analytic-DDIM [21], PNDM [22], FastDPM [38] and Itô-Taylor [24].
We also compare with GGDM [18], which uses the same pre-trained model but needs further training for the sampling trajectory.
We compare the sample quality by varying NFE from 10 to 1000.
Speciﬁcally, we use the discrete-time model trained by L simple in [2] on the CIFAR-10 dataset with linear noise schedule; the discrete-time model in [19] on CelebA 64x64 [39] with linear noise schedule; the discrete-time model trained by L hybrid in [16] on ImageNet 64x64 [26] with cosine noise schedule; the discrete-time model with classiﬁer guidance in [4] on ImageNet 128x128 [26] with linear noise schedule; the discrete-time model in [4] on LSUN bedroom 256x256 [40] with linear noise schedule.
For the models trained on ImageNet, we only use their “mean” model and omit the “variance” model.
As shown in Fig.
2, on all datasets, DPM-Solver can obtain reasonable samples within 12 steps (FID 4.65 on CIFAR-10, FID 3.71 on CelebA 64x64 and FID 19.97 on
ImageNet 64x64, FID 4.08 on ImageNet 128x128), which is 4 ∼ 16 × faster than the previous fastest training-free sampler.
DPM-Solver even outperforms GGDM, which requires additional training.
Conclusions
We tackle the problem of fast and training-free sampling from DPMs.
We propose DPM-Solver, a fast dedicated training-free solver of diffusion ODEs for fast sampling of DPMs in aroundsteps of function evaluations.
DPM-Solver leverages the semi-linearity of diffusion ODEs and it directly approximates a simpliﬁed formulation of exact solutions of diffusion ODEs, which consists of an exponentially weighted integral of the noise prediction model.
Inspired by numerical methods for exponential integrators, we propose ﬁrst-order, second-order and third-order DPMSolver to approximate the exponentially weighted integral of noise prediction models with theoretical convergence guarantee.
We propose both handcrafted and adaptive step size schedule, and apply
DPM-Solver for both continuous-time and discrete-time DPMs.
Our experimental results show that
DPM-Solver can generate high-quality samples in around 10 function evaluations on various datasets, and it can achieve 4 ∼ 16 × speedup compared with previous state-of-the-art training-free samplers.
Limitations and broader impact Despite the promising speedup performance, DPM-Solver is designed for fast sampling, which may be not suitable for accelerating the likelihood evaluations of
DPMs.
Besides, compared to the commonly-used GANs, diffusion models with DPM-Solver are still not fast enough for real-time applications.
In addition, like other deep generative models, DPMs may be used to generate adverse fake contents, and the proposed solver may further amplify the potential undesirable inﬂuence of deep generative models for malicious applications.
Acknowledgements
This work was supported by National Key Research and Development Project of China (No.
2021ZD0110502)
; NSF of China Projects (Nos. 62061136001, 61620106010, 62076145, U19B2034,
U1811461, U19A2081, 6197222, 62106120); Beijing NSF Project (No. JQ19016); Beijing Outstanding Young Scientist Program NO.
BJJWZYJH012019100020098; a grant from Tsinghua Institute for Guo Qiang; the NVIDIA NVAIL Program with GPU/DGX Acceleration; the High Performance
Computing Center, Tsinghua University; the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (22XNKJ13).
J.Z is also supported by the XPlorer Prize.
A Sampling with Invariance to the Noise Schedule
In this section, we discuss more about the exact solution in Proposition 3.1 and give some insights about the formulation.
Below we ﬁrstly restate the proposition w.r.t.
λ (i.e. the half-logSNR).
Proposition 3.1 (Exact solution of diffusion ODEs) .
Given an initial value ˆ x λ s at time s with the corresponding half-logSNR λ s , the solution ˆ x λ t at time t of diffusion ODEs in Eq.
(2.7) with the corresponding half-logSNR λ t is:
[Equation] (A. [Equation]
In the following subsections, we will show that such formulation decouples the model (cid:15) θ from the speciﬁc noise schedule, and thus is invariant to the noise schedule.
Moreover, such change-ofvariable for λ in Proposition 3.1 is highly related to the maximum likelihood training of diffusion models [10, 41].
We show that both the maximum likelihood training and the sampling of diffusion models have invariance formulations that are independent of the noise schedule.
A.1 Decoupling
the Sampling Solution from the Noise Schedule
In this section, we show that Proposition 3.1 can decouples the exact solutions of the diffusion ODEs from the speciﬁc noise schedules (i.e. choice of the functions α t = α ( t ) and σ t = σ ( t ) ).
Namely, given a starting point λ s , a ending point λ t , an initial value ˆ x λ s at λ s and a noise prediction model ˆ (cid:15) θ , the solution of ˆ x λ t is invariant of the noise schedule between λ s and λ t .
We ﬁrstly consider the VP type diffusion models, which is equivalent to the original DDPM [2, 3].
For VP type diffusion models, we always have α 2 t + σ 2 t = 1 , so deﬁning the noise schedule is equivalent to deﬁning the function α t
= α ( t ) (For example, DDPM [2] uses a noise schedule such that β ( t )
= dlog α t d t is a linear function of t , and i-DDPM
[16] uses a noise schedule such that β ( t )
= dlog α t d t is a cosine function of t ).
{+As λ t = log α t − log σ t , we have α t = (cid:113) 1 
+}
{++ e − 2 λt and σ t = (cid:113) 1 
+}
+ e 2 λt .
Thus, we can directly compute the α t and σ t for a given λ t .
Denote ˆ α λ := (cid:113) 1 1+ e − 2 λ , we have
[Equation]
We should notice that the integrand e − λ ˆ (cid:15) θ (ˆ x λ , λ ) is a function of λ , so its integral from λ s to λ t is only dependent on the starting point λ s , the ending point λ t and the function ˆ (cid:15) θ , which is independent of the intermediate values.
As other coefﬁcients ( ˆ α λ s and ˆ α λ t ) are also only dependent on the starting point λ s and the ending point λ t , we can conclude that ˆ x λ t is invariant of the speciﬁc choice of the noise schedules.
Intuitively, this is because we converts the original integral of time t in Eq.
(3.1) to the integral of λ , and the functions f ( t ) and g ( t ) are converted to an analytical formulation e − λ , which is invariant to the speciﬁc choices of f ( t ) and g ( t ) .
Finally, for other types of diffusion models (such as the VE type and the subVP type), they are all equivalent to the VP type by equivalently rescaling the noise prediction models, as proved in [10].
Therefore, the solutions of these types also have such property.
In summary, Proposition 3.1 decouples the solution of diffusion ODEs from the noise schedules, which gives us an opportunity to design tailor-made samplers for DPMs.
In fact, as shown in
Sec. 3.2, the only approximation of the proposed DPM-Solver is about the Taylor expansion of the neural network ˆ (cid:15) θ w.r.t.
λ , and DPM-Solver analytically computes other coefﬁcients (which are corresponding to the speciﬁc noise schedules).
Intuitively, DPM-Solver keeps the known information as much as possible, and only approximates the intractable integral of the neural network, so it can generate comparable samples within much fewer steps.
A.2 Choosing Time Steps for λ is Invariant to the Noise Schedule
As mentioned in Appendix A.1, the formulation of Proposition 3.1 decouples the sampling solution from the noise schedule.
The solution depends on the starting point λ s and the ending point λ t , and is invariant to the intermediate noise schedule.
Similarly, the updating equations of the algorithm of DPM-Solver are also invariant to the intermediate noise schedule.
Therefore, if we have chosen the time steps { λ i } Mi =0 , then the solution of DPM-Solver is also determined and is invariant to the intermediate noise schedule.
A simple way for choosing time steps for λ is uniformly splitting [ λ T , λ (cid:15) ] , which is the setting in our experiments.
However, we believe that there exists more precise ways for choosing the time steps, and we leave it for future work.
A.3 Relationship with the Maximum Likelihood Training of Diffusion Models
Interestingly, the maximum likelihood training of diffusion SDEs in continuous time also has such invariance property [10].
Below we brieﬂy review the maximum likelihood training loss of diffusion
SDEs, and then propose a new insight for understanding diffusion models.
Denote the data distribution as q 0 ( x 0 ) , the distribution of the forward process at each time t as q t ( x t ) , the distribution of the reverse process at each time t as p t ( x t ) with p T = N ( 0 , I ) .
In [3], it is proved that the KL-divergence between q 0 and p 0 can be bounded by a weighted score matching loss:
[Equation]
where x t = α t x 0 + σ t (cid:15) and C is a constant independent of θ .
As shown in Sec.
3.1, we have
[Equation]
so by applying change-of-variable w
[Equation]
r
[Equation]
t
[Equation]
λ , we have
[Equation]
which is equivalent to the importance sampling trick in [41, Sec. 5.1] and the continuous-time diffusion loss in [10, Eq.
Compared to Proposition 3.1, we can ﬁnd that the sampling and the maximum likelihood training of diffusion models can both be converted to an integral w.r.t.
λ , such that the formulation is invariant to the speciﬁc noise schedules, and we summarize it in Table 2.
Such invariance property for both training and sampling brings a new insight for understanding diffusion models.
For instance, we can directly deﬁne the noise prediction model (cid:15) θ w.r.t.
the (half-)logSNR λ instead of the time t , then the training and sampling for diffusion models can be done without further choosing any ad-hoc noise schedules .
Such ﬁnding may unify the different ways of the training and the inference of diffusion models, and we leave it for future study.
B Proof of Theorem 3. B.1 Assumptions
Throughout this section, we denote x s as the solution of the diffusion ODE Eq.
(2.7) starting from x T .
For DPM-Solver- k we make the following assumptions:
Assumption B.1.
The total derivatives d j ˆ (cid:15) θ (ˆ x λ ,λ ) d λ j (as a function of λ ) exist and are continuous for 0 ≤ j ≤ k + 1 .
Assumption B.2.
The function (cid:15) θ ( x , s ) is Lipschitz w.r.t.
to its ﬁrst parameter x .
Assumption B.3.
h max =
O (1 /M ) .
We note that the ﬁrst assumption is required by Taylor’s theorem Eq.
(3.6), and the second assumption is used to replace (cid:15)
θ (˜ x s , s ) with (cid:15) θ ( x s , s )
+ O ( x s − ˜ x s ) so that the Taylor expansion w.r.t.
λ s is applicable.
The last one is a technical assumption to exclude a signiﬁcantly large step-size.
B.2 General Expansion of the Exponentially Weighted Integral
Firstly, we derive the Taylor expansion of the exponentially weighted integral.
Let t <
s
and then λ t > λ s .
Denote h :=
λ t − λ s , and the k -th order total derivative ˆ (cid:15) ( k ) θ (ˆ x λ , λ ) : = d k ˆ (cid:15) θ (ˆ x λ ,λ ) d λ k . For n ≥ 0 , the n -th order Taylor expansion of ˆ (cid:15) θ (ˆ x λ , λ ) w.r.t.
λ is
[Equation]
To expand the exponential integrator, we further deﬁne [31]:
[Equation] and it satisﬁes ϕ k = 1 k {+! and a recurrence relation ϕ k +
+}
( z ) = [Equation] By taking the Taylor expansion of ˆ (cid:15) θ (ˆ x λ , λ ) , the exponential integrator can be rewritten as [Equation]
So the solution of x t in Eq.
(3.4) can be expanded as
[Equation] Finally, we list the closed-forms of ϕ k for k = 1 , 2 , 3 : [Equation] [Equation]
[Equation]
B.3 Proof of Theorem 3.2 when k =
Proof.
Taking n = 0 , t
= t i , s = t
i − 1 in Eq.
(B.4)
, we obtain
[Equation]
B.
[Equation]
By Assumption B.2 and Eq.
(3.7), it holds that
[Equation]
− 1 ˜ x t i − 1 − σ t
i ( e h
i − 1) (cid:0) (cid:15) θ ( x t i − 1 , t i − 1 )
+ O (˜ x t i − 1 − x t
i − 1 ) (cid:1) =
[Equation]
t i α t i − 1 x t i − 1 − σ t
i ( e h
i − 1) (cid:15) θ ( x t i − 1 , t i − 1 )
+ O (˜ x t i − 1 − x t
i − 1 )
= x t i + O ( h 2 max ) + O
(˜ x t i − 1 − x t i − 1 ) .
Repeat this argument, we ﬁnd that
[Equation]
and thus completes the proof.
B.4 Proof of Theorem 3.2 when k =
We prove the discretization error of the general form of DPM-Solver-2 in Algorithm 4.
Proof.
First, we consider the following update for 0 < t < s < T, h :=
λ t − λ s .
[Equation]
Note that the above update is the same as a single step of DPM-Solver-2 with s
=
t i − 1 and t =
t i , except that ˜ x t i − 1 is replaced with the exact solution x t i − 1 .
Once we have proven that ¯ x t = x t + O ( h 3 ) , we can show that ˜ x t i =
x
t i + O
[Equation]
h 3 max )
+ O
[Equation]
˜ x t i
[Equation]
1 − x
[Equation]
i − 1 ) by a similar argument as in Appendix B.3, and therefore completes the proof.
In this remaining part we prove that ¯ x t = x t + O ( h 3 ) .
Taking n = 1 in Eq.
(B.4)
, we obtain
[Equation]
From Eq.
(B.1), we have
[Equation]
Note that by the Lipschitzness of (cid:15) θ w.r.t.
x (Assumption B.2),
[Equation]
where the last equation follows from a similar argument in the proof of k = 1 .
Since e h − 1 =
O ( h ) , the second term of the above display is O ( h 3 ) .
As λ
[Equation]
− λ
[Equation]
= r 1 h , ϕ
i ( h ) =
( e h − 1) /h and ϕ
[Equation]
[Equation]
e h − h − 1) /h 2 , we ﬁnd
[Equation]
Then, the proof is completed by noticing that
[Equation]
he h ) / 2 =
O ( h
[Equation]
B.5 Proof of Theorem 3.2 when k =
Proof.
As in Appendix B.4, it sufﬁces to show that the following update has error ¯ x t = x t + O ( h 4 ) for 0 < t < s < T and h
= λ s − λ t .
[Equation]
First, we prove that
[Equation] [Equation]
{+Similar to the proof in Appendix B.4, since e r 2 h − 1 r 2 h − 
+}
= O ( h ) and ¯ u 1 = x s 1 + O ( h 2 ) , then
[Equation]
{+Let h 
+}
= r 2 h
[Equation]
then following the same line of arguments in the proof of Appendix B.4, it sufﬁces to check that
[Equation] which holds by applying Taylor {+expansion. Using ¯ u 2 = x s 
+}
+ O ( h 3 ) and λ s 2 − λ s = r 2 h = 23 h [Equation] we ﬁnd that
[Equation]
Comparing with the Taylor expansion in Eq. (B.4) with n = 2 : [Equation] we need to check the following conditions:
[Equation]
The ﬁrst two conditions are clear.
The last condition follows from [Equation] Therefore, ¯ x t = x t + O ( h 4 ) . B.6 Connections to Explicit Exponential Runge-Kutta (expRK) Methods
Assume we have an ODE with the following form: [Equation] where α ∈ R and N ( x t , t ) ∈ R D is a non-linear function of x t . Given an initial value x t at time t , for h > 0 , the true solution at time t
+ h is
[Equation] The exponential Runge-Kutta methods [25, 31] use some intermediate points to approximate the integral (cid:82) e − ατ N ( x t + τ , t + τ )d τ . Our proposed DPM-Solver is inspired by the same technique for approximating the same integral with α = 1 and N = ˜ (cid:15) θ .
However, DPM-Solver is different from the expRK methods, because their linear term e αh x t is different from our linear term α t + h α t x t . In summary, DPM-Solver is inspired by the same technique of expRK for deriving high-order approximations of the exponentially weighted integral, but the formulation of DPM-Solver is different from expRK, and DPM-Solver is customized for the speciﬁc formulation of diffusion ODEs. C Algorithms of DPM-Solvers
We ﬁrstly list the detailed DPM-Solver-1, 2, 3 in Algorithms 3, 4, 5. Note that DPM-Solver-2 is the general case with r 1 ∈ (0 , 1) , and we usually set r 1 = 0 . 5 for DPM-Solver-2, as in Sec. 3. Algorithm 3 DPM-Solver-1.
Require: initial value x T , time steps { t i } Mi =0 , model (cid:15) θ 1: def dpm-solver-1 ( ˜ x t i − 1 , t i − 1 , t i ) : 2: h [Equation]
←
[Equation] [Equation]
4: return ˜ x t i 5: ˜ x t 0 ← x T 6: for i ← 1 to M do 7: ˜ x t i ← dpm-solver-1 (˜ x t i − 1 , t i − 1 , t i ) 8: end for 9: return ˜ x
t M
Algorithm 4 DPM-Solver-2 ( version).
Require: initial value x T , time steps { t i } Mi [-=
-]=0 , model (cid:15) θ , r 1 = 0 .1:
def dpm-solver-2 (˜ x t i − 1 , t
i − 1 , t i , r 1 ) : 2: h i ← λ t i − λ t i −3: s i ← t λ (cid:0)
λ t i − 1 + r 1 h i (cid:1) 4: u i ← α si α ti − 1 ˜ x t i − 1 − σ s i (cid:0) e r 1 h i − 1 (cid:1) (cid:15) θ (˜ x t i − 1 , t i − 1 ) 5: ˜ x t i ← α t
i
α ti − 1 ˜ x t i − 1 − σ t i ( e h i − 1) (cid:15)
θ
( ˜ x t i − 1 , t i − 1 ) − σ t i 2 r 1 ( e h i − 1)( (cid:15) θ ( u i , s i ) − (cid:15) θ ( ˜ x t i − 1 , t i − 1 )) 6: return ˜ x t i 7: ˜ x t 0 ← x T 8: for i ← 1 to M do 9: ˜ x t i ← dpm-solver-2 (
˜ x t i − 1 , t i − 1 , t i , r 1 ) 10: end for 11: return ˜ x t M
Then we list the adaptive step size algorithms, named as DPM-Solver-12 (combining 1 and 2; Algorithm 6) and DPM-Solver-23 (combining 2 and 3; Algorithm 7).
We follow [20] to let the absolute tolerance (cid:15) atol = x max − x minfor image data, which is 0 . 0078 for VP type DPMs. We can tune the relative tolerance (cid:15) rtol to balance the accuracy and NFE, and we ﬁnd that (cid:15) rtol = 0 . 05 is good enough and can converge quickly.
In practice, the inputs of the adaptive step size solvers are batch data. We simply choose E 2 and E 3 as the maximum value of all the batch data. Besides, we implement the comparison s > (cid:15) by | s − (cid:15) | > 10 − 5 to avoid numerical issues.
[Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure] [Table] [Figure]
[-DPM-Solver-2 and ( s 
-][Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure]
i 10: ˜ x t 0 ← x T 11: for i ← 1 to M do 12: ˜ x t i ← dpm-solver-3 (˜ x t i − 1 , t i − 1 , t i , r 1 , r 2 ) 13: end for 14: return ˜ x
t M
Require: start time T , end time (cid:15) , initial value x T , model (cid:15) θ , data dimension D , hyperparameters (cid:15) rtol = 0 .
05 , (cid:15) atol = 0 .
0078 , h init = 0 . 05 , θ = 0 . Ensure: the approximated solution x (cid:15) at time (cid:15) 1: s ← T , h ← h init , x ← x T , x prev ← x T , r 1 ← 12 , NFE ←2: while s > (cid:15) do 3: t ← t λ ( λ s + h ) 4: x 1 ← dpm-solver-1 ( x , s, t ) 5: x 2 ← dpm-solver-2 ( x , s, t, r 1 ) (Share the same function value (cid:15) θ ( x , s ) with dpm-solver-1.)
6: δ ← max( (cid:15) atol , (cid:15) rtol max( | x 1 | , | x prev | )) 7: E 2 ← 1 √ D (cid:107) x 1 − x 2 δ (cid:107)8: if E 2 ≤ 1 then 9: x prev ← x 1 , x ← x 2 , s ← t 10: end if 11: h ← min( θhE − 12 2 , λ (cid:15) − λ s ) 12:
NFE ← NFE +13: end while 14: return x , NFE D Implementation Details of DPM-Solver D.1 End Time of Sampling Theoretically, we need to solve diffusion ODEs from time T to time 0 to generate samples.
Practically, the training and evaluation for the noise prediction model (cid:15) θ ( x t , t ) usually start from time T to time (cid:15) to avoid numerical issues for t near to 0 , where (cid:15) > 0 is a hyperparameter [3].
In contrast to the sampling methods based on diffusion SDEs [2, 3], We don’t add the “denoising” trick at the ﬁnal step at time (cid:15) (which is to set the noise variance to zero), and we just solve diffusion ODEs from T to (cid:15) by DPM-Solver, since we ﬁnd it performs well enough.
For discrete-time DPMs, we ﬁrstly convert the model to continuous time (see Appendix D.2), and then solver it from time T to time t . Require: start time T , end time (cid:15) , initial value x T , model (cid:15) θ , data dimension D , hyperparameters (cid:15) rtol = 0 .
05 , (cid:15) atol
= 0 . 0078 , h init = 0 .
05 , θ = 0 .
Ensure: the approximated solution x (cid:15) at time (cid:15) 1: s ← T , h ← h init , x ← x T , x prev ← x T , r 1 ← 13 , r 2 ← 23 , NFE ←2: while s > (cid:15) do 3: t ← t λ ( λ s + h ) 4: x 2 ← dpm-solver-2 ( x , s, t, r 1 ) 5: x 3 ← dpm-solver-3 ( x , s, t, r 1 , r 2 ) (Share the same function values with dpm-solver-2.) 6: δ ← max( (cid:15) atol , (cid:15) rtol max( | x 2 | , | x prev | )) 7: E 3 ← 1 √ D (cid:107) x 2 − x 3 δ (cid:107)8: if E 3 ≤ 1 then 9: x prev ← x 2 , x ← x 3 , s ← t 10: end if 11: h ← min( θhE − 13 3 , λ (cid:15) − λ s ) 12:
NFE ← NFE +13: end while 14: return x , NFE D.2 Sampling from Discrete-Time DPMs In this section, we discuss the more general case for discrete-time DPMs, in which we consider the 1000-step DPMs [2] and the 4000-step DPMs [16], and we also consider the end time (cid:15) for sampling.
Discrete-time DPMs [2] train the noise prediction model at N ﬁxed time steps { t n } Nn =1 .
In practice,
N = 1000 or N = 4000 , and the implementation of the 4000-step DPMs [16] converts the time steps of 4000-step DPMs to the range of 1000-step DPMs.
Speciﬁcally, the noise prediction model is parameterized by ˜ (cid:15) θ ( x n , 1000 n N ) for n = 0 , . . . , N − 1 , where each x n is corresponding to the value at time t n +1 . In practice, these discrete-time DPMs usually choose uniform time steps between [0 , T ] , thus t n = nTN , for n = 1 , . . .
, N . However, the discrete-time noise prediction model cannot predict the noise at time less than the smallest time t 1 . As the smallest time step t 1 = TN and the corresponding discrete-time noise prediction model at time t 1 is ˜ (cid:15) θ ( x 0 , 0) , we need to “scale” the discrete time steps [ t 1 , t N ] = [ TN , T ] to the continuous time range [ (cid:15), T ] .
We propose two types of scaling as following. Type-1. Scale the discrete time steps [ t 1 , t N ] = [ TN , T ] to the continuous time range [ TN , T ] , and let (cid:15)
[Equation]
[Equation] · , TN ) for t ∈ [ (cid:15), TN ] . In this case, we can deﬁne the continuous-time noise prediction model by [Equation]
where the continuous time t ∈ [ (cid:15), T N ] maps to the discrete input 0 , and the continuous time T maps to the discrete input 1000( N − 1) N . Type-2. Scale the discrete time steps [ t 1 , t N ] = [ TN , T ] to the continuous time range [0 , T ] .
In this case, we can deﬁne the continuous-time noise prediction model by [Equation] where the continuous time 0 maps to the discrete input 0 , and the continuous time T maps to the discrete input 1000( N − 1) N . Note that the input time of ˜ (cid:15) θ may not be integers, but we ﬁnd that the noise prediction model can still work well, and we hypothesize that it is because of the smooth time embeddings (e.g., position embeddings [2]).
By such reparameterization, the noise prediction model can adopt the continuous-time steps as input, and thus we can also use DPM-Solver for fast sampling. In practice, we have T = 1 , and the smallest discrete time t 1 = 10 − 3 . For ﬁxed K number of function evaluations, we empirically ﬁnd that for small K , the Type-1 with (cid:15) = 10 − 3 may have better sample quality, and for large K , the Type-2 with (cid:15) = 10 − 4 may have better sample quality.
We refer to Appendix E for detailed results.
D.3 DPM-Solver in 20 Function Evaluations
Given a ﬁxed budget K ≤ 20 of the number of function evaluations, we uniformly divide the interval [ λ T , λ (cid:15) ] into M = ( (cid:98) K/ 3 (cid:99) + 1) segments, and take M steps to generate samples. The M steps are dependent on the remainder R of K mod 3 to make sure the total number of function evaluations is exactly K .
• If R = 0 , we ﬁrstly take M − 2 steps of DPM-Solver-3, and {+then take 1 step of DPM-Solverand 1 step of DPM-Solver-1. The total number of function evaluations is 3 · ( K 3 − 1)+2+
 =+}
K .
• If R = 1 , we ﬁrstly take M − 1 steps of DPM-Solver-3 and then take 1 step of DPM-Solver-1.
The total number of function evaluations is 3 · ( K − 1 3 ) + 1 = K .
• If R = 2 , we ﬁrstly take M − 1 steps of DPM-Solver-3 and then take 1 step of DPM-Solver-2.
The total number of function evaluations is 3 · ( K − 2 3 ) + 2 = K .
We empirically ﬁnd that this design of time steps can greatly improve the generation quality, and
DPM-Solver can generate comparable samples in 10 steps and high-quality samples in 20 steps. D.4 Analytical Formulation of the function t λ ( · ) (the inverse function of λ ( t ) ) The costs of computing t λ ( · ) is negligible, because for the noise schedules of α t and σ t used in previous DPMs (“linear” and “cosine”) [2, 16], both λ ( t ) and its inverse function t λ ( · ) have analytic formulations.
We mainly consider the variance preserving type here, since it is the most widely-used type. The functions of other types (variance exploding and sub-variance preserving type) can be similarly derived. Linear Noise Schedule [2]. We have
[Equation] where β 0 = 0 . 1 and β 1 = 20 , following [3]. As σ t = (cid:112) 1 − α 2 t , we can compute λ t analytically.
Moreover, the inverse function is
[Equation] To reduce the inﬂuence of numerical issues, we can compute t λ by the following equivalent formulation: [Equation] And we solve diffusion ODEs between [ (cid:15), T ] , where T = 1 .
Cosine Noise Schedule [16].
Denote
[Equation]
where s = 0 .
008 , following [16].
As [16] clipped the derivatives to ensure the numerical stability, we also clip the maximum time T = 0 . 9946 .
As σ t = (cid:112) 1 − α 2 t , we can compute λ t analytically.
Moreover, given a ﬁxed λ , let
[Equation]
which computes the corresponding log α for λ
Then the inverse function is
[Equation]
−
[Equation] And we solve diffusion ODEs between [ (cid:15), T ] , where T = 0 . 9946 . D.5 Conditional Sampling by DPM-Solver DPM-Solver can also be used for conditional sampling, with a simple modiﬁcation.
The conditional generation needs to sample from the conditional diffusion ODE [3, 4] which includes the conditional noise prediction model.
We follow the classiﬁer guidance method [4] to deﬁne the conditional noise prediction model as (cid:15) θ ( x t , t, y ) : = (cid:15) θ ( x t , t ) − s · σ t ∇ x log p t ( y | x t ; θ ) , where p t ( y | x t ; θ ) is a pre-trained classiﬁer and s is the classiﬁer guidance scale (default is 1.0). Thus, we can use DPM-Solver to solve this diffusion ODE for fast conditional sampling, as shown in Fig.
D.6 Numerical Stability
As we need to compute e h i − 1 in the algorithm of DPM-Solver, we follow [10] to use expm1( h i ) instead of exp( h i )-1 to improve numerical stability. E Experiment Details We test our method for sampling the most widely-used variance-preserving (VP) type DPMs [1, 2].
In this case, we have α 2 t + σ 2 t = 1 for all t ∈ [0 , T ] and ˜ σ = 1 .
In spite of this, our method and theoretical results are general and independent of the choice of the noise schedule α t and σ t . For all experiments, we evaluate DPM-Solver on NVIDIA A40 GPUs. However, the computation resource can be other types of GPU, such as NVIDIA GeForce RTX 2080Ti, because we can tune the batch size for sampling. E.1 Diffusion ODEs w.r.t. λ
Alternatively, the diffusion ODE can be reparameterized to the λ domain. In this section, we propose the formulation of diffusion ODEs w.r.t. λ for VP type, and other types can be similarly derived.
For a given λ , denote ˆ α λ :=
α t ( λ ) , ˆ σ λ := σ t ( λ ) .
As ˆ α 2 λ + ˆ σ 2 λ
= 1 , we can prove that d λ dˆ α λ =
[Equation]
so dlog ˆ α λ d λ = ˆ σ 2 λ .
Applying change-of-variable to Eq. (2.7), we have
[Equation]
The ODE Eq.
(E.1) can be also solved directly by RK methods, and we use such formulation for the experiments of RK2 ( λ ) and RK3 ( λ ) in Table 1. E.2 Code Implementation We implement our code with both JAX (for continuous-time DPMs) and PyTorch (for discrete-time DPMs), and our code is released at https://github.com/LuChengTHU/dpm-solver .
Table 3 shows the detailed FID results, which is corresponding to Fig. 2a. We use the ofﬁcial code and checkpoint in [3], the code license is Apache License 2.0.
We use their released “checkpoint_8” of the “VP deep” type. We compare methods for (cid:15) = 10 − 3 and (cid:15) = 10 − 4 . We ﬁnd that the sampling methods based on diffusion SDEs can achieve better sample quality with (cid:15) = 10 − 3 ; and that the sampling methods based on diffusion ODEs can achieve better sample quality with (cid:15) = 10 − 4 .
For
DPM-Solver, we ﬁnd that DPM-Solver with less than 15 NFE can achieve better FID with (cid:15) = 10 −than (cid:15) = 10 − 4 , while DPM-Solver with more than 15 NFE can achieve better FID with (cid:15) = 10 −than (cid:15) =
10 − 3 .
For the diffusion SDEs with Euler discretization, we use the PC sampler in [3] with “euler_maruyama” predictor and no corrector, which uses uniform time steps between T and (cid:15) . We add the “denoise” trick at the ﬁnal step, which can greatly improve the FID score for (cid:15) =
10 − 3 .
For the diffusion SDEs with Improved Euler discretization [20], we follow the results in their original paper, which only includes the results with (cid:15) =
10 − 3 . The corresponding relative tolerance (cid:15) rel are 0 .
10 and 0 . 05 , respectively.
[Table]
For the diffusion ODEs with RK45 Solver, we use the code in [3]
[Table]
and tune the atol and rtol of the solver.
For the NFE from small to large, we use the same atol = rtol = 0 . 001 for the results of (cid:15) =
10 − 3 , and the same atol = rtol = 0 . 1 , 0 . 05 , 0 . 001 for the results of (cid:15) =
10 − 4 , respectively. For the diffusion ODEs with DPM-Solver, we use the method in Appendix D.3 for NFE ≤ 20 , and the adaptive step size solver in Appendix C. For (cid:15)
=
10 − 3 , we use DPM-Solver-12 with relative tolerance (cid:15) rtol = 0 . 05 . For (cid:15) = 10 − 4 , we use DPM-Solver-23 with relative tolerance (cid:15) rtol = 0 . 05 .
E.4 Sample Quality Comparison with RK Methods
Table 1 shows the different performance of RK methods and DPM-Solver-2 and 3.
We list the detailed settings in this section. Assume we have an ODE with [Equation]
Starting with [-10 NFE, 3.75 FID with 12 NFE, 3.24 FID with 
 NFE, and 2.87 FID with 20 NFE, which is the fastest sampler on CIFAR-10.-]˜ x
t i − 1 at time t i − 1 , we use RK2 to approximate the solution ˜ x t i at time t i in the following formulation (which is known as the explicit midpoint method): [Equation] And we use the following RK3 to approximate the solution ˜ x t i at time t i (which is known as “Heun’s third-order method”), because it is very similar to our proposed DPM-Solver-3:
[Equation] We use F ( x t , t ) = h θ ( x t , t ) in Eq.
(2.7) for the results with RK2 ( t ) and RK3 ( t ), and F (ˆ x λ , λ )
= ˆ h θ (ˆ x λ , λ ) in Eq.
(E.1) for the results with RK2 ( λ ) and RK3 ( λ ). For all experiments, we use the uniform step size w.r.t.
t or λ .
E.5 Sample Quality Comparison with Discrete-Time Sampling Methods
We compare DPM-Solver with other discrete-time sampling methods for DPMs, as shown in Tableand Table 5. We use the code in [19] for sampling with DDPM and DDIM, and the code license
[Table] .50 Analytic-DDPM [21] Discrete 28. 5.21 Analytic-DDIM [21]
Discrete 15. 3.13 DDIM [19] Discrete 10. DPM-Solver (Type-1 discrete) (cid:15) =
10 −7.
4.23 (NFE = 36) (cid:15) = 10 −6.
2.71 (NFE = 36)
DPM-Solver (Type-2 discrete) (cid:15) =
10 −7.
6.68 (NFE = 36) (cid:15) = 10 −5.
3.10 (NFE = 36)
ImageNet 64 × 64 (discrete-time model [16], cosine noise schedule)
DDPM [2]
Discrete 305.
17.58 Analytic-DDPM [21]
Discrete 60.
16.14 Analytic-DDIM [21]
Discrete 70.
17.57 † GGDM [18]
Discrete 37.
\ \ \ DDIM [19]
Discrete 67.
DPM-Solver (Type-1 discrete) (cid:15) =
10 −24.
[Table]
[Table]
[Table]
is MIT License.
We use the code in [21] for sampling with Analytic-DDPM and Analytic-DDIM, whose license is unknown.
We directly follow the best results in the original paper of GGDM [18]. For the CIFAR-10 experiments, we use the pretrained checkpoint by [2], which is also provided in the released code in [19]. We use quadratic time steps for DDPM and DDIM, which empirically has better FID performance than the uniform time steps [19]. We use the uniform time steps for
Analytic-DDPM and Analytic-DDIM.
For DPM-Solver, we use both Type-1 discrete and Typediscrete methods to convert the discrete-time model to the continuous-time model.
We use the method in Appendix D.3 for NFE ≤ 20 , and the adaptive step size solver in Appendix C for NFE > 20 .
For all the experiments, we use DPM-Solver-12 with relative tolerance (cid:15) rtol = 0 . 05 .
For the CelebA 64x64 experiments, we use the pretrained checkpoint by [19].
We use quadratic time steps for DDPM and DDIM, which empirically has better FID performance than the uniform time steps [19].
We use the uniform time steps for Analytic-DDPM and Analytic-DDIM.
For DPM-Solver, we use both Type-1 discrete and Type-2 discrete methods to convert the discrete-time model to the continuous-time model.
We use the method in Appendix D.3 for NFE ≤ 20 , and the adaptive step size solver in Appendix C for NFE > 20 .
For all the experiments, we use DPM-Solver-12 with relative tolerance (cid:15) rtol = 0 . 05 .
Note that our best FID results on CelebA 64x64 is even better than the 1000-step DDPM (and all the other methods).
For the ImageNet 64x64 experiments, we use the pretrained checkpoint by [16], and the code license is MIT License.
We use the uniform time steps for DDPM and DDIM, following [19].
We use the uniform time steps for Analytic-DDPM and Analytic-DDIM.
For DPM-Solver, we use both Typediscrete and Type-2 discrete methods to convert the discrete-time model to the continuous-time model.
We use the method in Appendix D.3 for NFE ≤ 20 , and the adaptive step size solver in
Appendix C for NFE > 20 .
For all the experiments, we use DPM-Solver-23 with relative tolerance (cid:15) rtol = 0 . 05 .
Note that the ImageNet dataset includes real human photos and it may have privacy issues, as discussed in [42].
For the ImageNet 128x128 experiments, we use classiﬁer guidance for sampling with the pretrained checkpoints (for both the diffusion model and the classiﬁer model) by [4], and the code license is
MIT License.
We use the uniform time steps for DDPM and DDIM, following [19].
For DPM-Solver, we only use Type-1 discrete method to convert the discrete-time model to the continuous-time model.
We use the method in Appendix D.3 for NFE ≤ 20 , and the adaptive step size solver DPM-Solverwith relative tolerance (cid:15) rtol = 0 .
(detailed in Appendix C) for NFE > 20 .
For all the experiments, we set the classiﬁer guidance scale s = 1 . 25 , which is the best setting for DDIM in [4] (we refer to their Table 14 for details).
For the LSUN bedroom 256x256 experiments, we use the unconditional pretrained checkpoint by [4], and the code license is MIT License.
We use the uniform time steps for DDPM and DDIM, following [19].
For DPM-Solver, we only use Type-1 discrete method to convert the discrete-time model to the continuous-time model.
We use the method in Appendix D.3 for DPM-Solver.
We also compare the sample quality of the different orders of DPM-Solver, as shown in Table 6. We use DPM-Solver-1,2,3 with uniform time steps w.r.t. λ , and the fast version in Appendix D.3 for NFE less than 20, and we name it as DPM-Solver-fast .
For the discrete-time models, we only compare the
Type-2 discrete method, and the results of Type-1 are similar.
As the actual NFE of DPM-Solver-2 is 2 × (cid:98) NFE / 2 (cid:99) and the actual NFE of DPM-Solver-3 is 3 × (cid:98) NFE / 3 (cid:99) , which may be smaller than NFE, we use the notation † to note that the actual NFE is less than the given NFE. We ﬁnd that for NFE less than 20, the proposed fast version (DPM-Solverfast) is usually better than the single order method, and for larger NFE, DPM-Solver-3 is better than DPM-Solver-2, and DPM-Solver-2 is better than DPM-Solver-1, which matches our proposed convergence rate analysis. E.7 Runtime Comparison between DPM-Solver and DDIM Theoretically, for the same NFE, the runtime of DPM-Solver and DDIM are almost the same (linear to NFE) because the main computation costs are the serial evaluations of the large neural network (cid:15) θ and the other coefﬁcients are analytically computed with [-classiﬁer guidance in [4] on ImageNet 128x
-]ignorable costs.
[Table]
DPM-Solver7.
DPM-Solver- † 24.
DPM-Solver-fast 6.
\ \ \ (cid:15) = 10 −
DPM-Solver13.
DPM-Solver11.
DPM-Solver- † 67.
DPM-Solver-fast 10.
CelebA 64 × 64 (discrete-time model [19], linear noise schedule), DPM-Solver with Type-2 discrete (cid:15) = 10 − DPM-Solver18. DPM-Solver5.
DPM-Solver- † 11.
DPM-Solver-fast 7.
\ \ \ (cid:15) = 10 −
DPM-Solver13.
DPM-Solver4.
DPM-Solver- † 49.
DPM-Solver-fast 5.
ImageNet 64 × 64 (discrete-time model [16], cosine noise schedule), DPM-Solver with Type-2 discrete (cid:15) = 10 −
DPM-Solver32.
DPM-Solver29.
DPM-Solver- † 57.
DPM-Solver-fast 24.
\ \ \ (cid:15) = 10 −
DPM-Solver32.
DPM-Solver33. DPM-Solver- † 162. DPM-Solver-fast 27. Table 7 shows the runtime of DPM-Solver and DDIM on a single NVIDIA A40, varying different datasets and NFE.
We use torch.cuda.
Event and torch.cuda.synchronize for accurately computing the runtime. We use the discrete-time pretrained diffusion models for each dataset.
We evaluate the runtime for 8 batches and computes the mean and std of the runtime. We use 64 batch size for LSUN bedroom 256x256 due to the GPU memory limitation, and 128 batch size for other datasets.
For DDIM, we use the ofﬁcial implementation 3 . We ﬁnd that our implementation of DPM-Solver reduces some repetitive computation of the coefﬁcients, so under the same NFE, DPM-Solver is slightly faster than DDIM of their implementation. Nevertheless, the runtime evaluation results show
[Table]
that the runtime of DPM-Solver and DDIM are almost the same for the same NFE, and the runtime is approximately linear to the NFE.
Therefore, the speedup for the NFE is almost the actual speedup of the runtime, so the proposed DPM-Solver can greatly speedup the sampling of DPMs. E.8 Conditional Sampling on ImageNet 256x
For the conditional sampling in Fig. 1, we use the pretrained checkpoint in [4] with classiﬁer guidance (ADM-G), and the classiﬁer scale is 1 . 0 .
The code license is MIT License.
We use uniform time step for DDIM, and the fast version for DPM-Solver in Appendix D.3 (DPM-Solver-fast) with 10, 15,and 100 steps.
Fig. 3 shows the conditional sample results by DDIM and DPM-Solver. We ﬁnd that DPM-Solver with 15 NFE can generate comparable samples with DDIM with 100 NFE. E.9 Additional Samples
Additional sampling results on CIFAR-10, CelebA 64x64, ImageNet 64x64, LSUN bedroom 256x256 [40], ImageNet 256x256 are reported in Figs. NFE = NFE = NFE =
NFE = [Figure] [Figure] [Figure]
NFE
[Figure] NFE [Figure] NFE =
NFE = [Figure] [Figure] [Equation]
[Figure]
NFE
[Figure]
NFE =
NFE =
NFE =
[Figure]
[Figure]