DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
Abstract
Diffusion probabilistic models (DPMs) are emerging powerful generative models.
Despite their high-quality generation performance, DPMs still suffer from theirslow sampling as they generally need hundreds or thousands of sequential functionevaluations (steps) of large neural networks to draw a sample.
Sampling from
DPMs can be viewed alternatively as solving the corresponding diffusion ordinarydifferential equations (ODEs).
In this work, we propose an exact formulation ofthe solution of diffusion ODEs.
The formulation analytically computes the linearpart of the solution, rather than leaving all terms to black-box ODE solvers asadopted in previous works.
By applying change-of-variable, the solution can beequivalently simpliﬁed to an exponentially weighted integral of the neural network.
Based on our formulation, we propose DPM-Solver , a fast dedicated high-ordersolver for diffusion ODEs with the convergence order guarantee.
DPM-Solveris suitable for both discrete-time and continuous-time DPMs without any furthertraining.
Experimental results show that DPM-Solver can generate high-qualitysamples in only 10 to 20 function evaluations on various datasets.
We achieve4.70 FID in 10 function evaluations and 2.87 FID in 20 function evaluations on the
CIFAR10 dataset, and a 4 ∼ 16 × speedup compared with previous state-of-the-arttraining-free samplers on various datasets.
Introduction
Diffusion probabilistic models (DPMs) [1–3] are emerging powerful generative models with promising performance on many tasks, such as image generation [4, 5], video generation [6], text-to-imagegeneration [7], speech synthesis [8, 9] and lossless compression [10].
DPMs are deﬁned by discretetime random processes [1, 2] or continuous-time stochastic differential equations (SDEs) [3], whichlearn to gradually remove the noise added to the data points.
Compared with the widely-used generative adversarial networks (GANs) [11] and variational auto-encoders (VAEs) [12], DPMs can not onlycompute exact likelihood [3], but also achieve even better sample quality for image generation [4].
However, to obtain high-quality samples, DPMs usually need hundreds or thousands of sequentialsteps of large neural network evaluations, thereby resulting in a much slower sampling speed than thesingle-step GANs or VAEs.
Such inefﬁciency is becoming a critical bottleneck for the adoption of
DPMs in downstream tasks, leading to an urgent request to design fast samplers for DPMs.
Existing fast samplers for DPMs can be divided into two categories.
The ﬁrst category includesknowledge distillation [13, 14] and noise level or sample trajectory learning [15–18].
Such methodsrequire a possibly expensive training stage before they can be used for efﬁcient sampling.
Furthermore,their applicability and ﬂexibility might be limited.
It might require nontrivial effort to adapt themethod to different models, datasets, and number of sampling steps.
The second category consistsof training-free [19–21] samplers, which are suitable for all pre-trained DPMs in a simple plug-and-
[Figure]
play manner.
Training-free samplers include adopting implicit [19] or analytical [21] generationprocess, advanced differential equation (DE) solvers [3, 20, 22–24] and dynamic programming [18].
However, these methods still require ∼ 50 function evaluations [21] to generate high-quality samples(comparable to those generated by plain samplers in about 1000 function evaluations), thereby arestill time-consuming.
In this work, we bring the efﬁciency of training-free samplers to a new level to produce high-qualitysamples in the “ few-step sampling ” regime, where the sampling can be done within around 10 stepsof sequential function evaluations.
We tackle the alternative problem of sampling from DPMs assolving the corresponding diffusion ordinary differential equations (ODEs) of DPMs, and carefullyexamine the structure of diffusion ODEs.
Diffusion ODEs have a semi-linear structure — they consistof a linear function of the data variable and a nonlinear function parameterized by neural networks.
Such structure is omitted in previous training-free samplers [3, 20], which directly use black-box
DE solvers.
To utilize the semi-linear structure, we derive an exact formulation of the solutions ofdiffusion ODEs by analytically computing the linear part of the solutions, avoiding the correspondingdiscretization error.
Furthermore, by applying change-of-variable, the solutions can be equivalentlysimpliﬁed to an exponentially weighted integral of the neural network.
Such integral is very specialand can be efﬁciently approximated by the numerical methods for exponential integrators [25].
Based on our formulation of solutions, we propose DPM-Solver , a fast dedicated solver for diffusion
ODEs by approximating the above integral.
Speciﬁcally, we propose ﬁrst-order, second-order andthird-order versions of DPM-Solver with convergence order guarantees.
We further propose anadaptive step size schedule for DPM-Solver.
In general, DPM-Solver is applicable to both continuoustime and discrete-time DPMs, and also conditional sampling with classiﬁer guidance [4].
Fig.
1demonstrates the speedup performance of a Denoising Diffusion Implicit Models (DDIM)
[19]baseline and DPM-Solver, which shows that DPM-Solver can generate high-quality samples with asfew as 10 function evaluations and is much faster than DDIM on the ImageNet 256x256 dataset [26].
Our additional experimental results show that DPM-Solver can greatly improve the sampling speed ofboth discrete-time and continuous-time DPMs, and it can achieve excellent sample quality in around10 function evaluations, which is much faster than all previous training-free samplers of DPMs.
Diffusion Probabilistic Models
We review diffusion probabilistic models and their associated differential equations in this section.
2.1 Forward Process and Diffusion SDEs
Assume that we have a D -dimensional random variable x 0 ∈ R D with an unknown distributionq 0 ( x 0 ) .
Diffusion Probabilistic Models (DPMs) [1–3, 10] deﬁne a forward process { x t } t ∈ [0 ,T ] with
T > 0 starting with x 0 , such that for any t ∈ [0 , T ] , the distribution of x t conditioned on x 0 satisﬁes
[Equation]
where α ( t ) , σ ( t ) ∈ R + are differentiable functions of t with bounded derivatives, and we denotethem as α t , σ t for simplicity.
The choice for α t and σ t is referred to as the noise schedule of a
DPM.
Let q t ( x t ) denote the marginal distribution of x t , DPMs choose noise schedules to ensurethat q T ( x T )
≈ N ( x T | 0 , ˜
σ 2 I ) for some ˜ σ > 0 , and the signal-to-noise-ratio (SNR) α 2 t /σ 2 t is strictlydecreasing w.r.t.
t [10].
Moreover, Kingma et al.
[10] prove that the following stochastic differentialequation (SDE) has the same transition distribution q 0 t ( x t | x 0 ) as in Eq.
(2.1) for any t ∈ [0 , T ] :
[Equation]
where w t ∈ R D is the standard Wiener process, and
[Equation]
Under some regularity conditions, Song et al.
[3] show that the forward process in Eq.
(2.2) has anequivalent reverse process from time T to 0 , starting with the marginal distribution q T ( x T ) :
[Equation]
where ¯ w t is a standard Wiener process in the reverse time.
The only unknown term in Eq.
(2.4) isthe score function ∇ x log q t ( x t ) at each time t .
In practice, DPMs use a neural network (cid:15) θ ( x t , t )parameterized by θ to estimate the scaled score function: − σ t ∇ x log q t ( x t ) .
The parameter θ isoptimized by minimizing the following objective [2, 3]:
[Equation]
where ω ( t ) is a weighting function, (cid:15) ∼ q ( (cid:15) )
=
N ( (cid:15) | 0 , I ) , x t = α t x 0 + σ t (cid:15) , and C is a constantindependent of θ .
As (cid:15) θ ( x t , t ) can also be regarded as predicting the Gaussian noise added to x t , itis usually called the noise prediction model .
Since the ground truth of (cid:15) θ ( x t , t ) is − σ t ∇ x log q t ( x t ) ,
DPMs replace the score function in Eq.
(2.4) by − (cid:15) θ ( x t , t ) /σ
t and deﬁne a parameterized reverseprocess ( diffusion SDE ) from time T to 0 , starting with x T ∼ N ( 0 , ˜
σ 2 I ) :
[Equation]
Samples can be generated from DPMs by solving the diffusion SDE in Eq.
(2.5) with numericalsolvers, which discretize the SDE from T to 0 .
Song et al.
[3] proved that the traditional ancestralsampling method for DPMs [2] can be viewed as a ﬁrst-order SDE solver for Eq.
However, theseﬁrst-order methods usually need hundreds of or thousands of function evaluations to converge [3],leading to extremely slow sampling speed.
2.2 Diffusion (Probability Flow) ODEs
When discretizing SDEs, the step size is limited by the randomness of the Wiener process [27, Chap.11].
A large step size (small number of steps) often causes non-convergence, especially in highdimensional spaces.
For faster sampling, one can consider the associated probability ﬂow ODE [3],which has the same marginal distribution at each time t as that of the SDE.
Speciﬁcally, for DPMs,
Song et al.
[3] proved that the probability ﬂow ODE of Eq.
(2.4) is
[Equation]
where the marginal distribution of x t is also q t ( x t ) .
By replacing the score function with the noiseprediction model, Song et al.
[3] deﬁned the following parameterized ODE ( diffusion ODE ):
[Equation]
Samples can be drawn by solving the ODE from T to 0 .
Comparing with SDEs, ODEs can be solvedwith larger step sizes as they have no randomness.
Furthermore, we can take advantage of efﬁcientnumerical ODE solvers to accelerate the sampling.
Song et al.
[3] used the RK45 ODE solver [28]for the diffusion ODEs, which generates samples in ∼ 60 function evaluations to reach comparablequality with a 1000-step SDE solver for Eq.
(2.5) on the CIFAR-10 dataset [29].
However, existinggeneral-purpose ODE solvers still cannot generate satisfactory samples in the few-step ( ∼ 10 steps)sampling regime.
To the best of our knowledge, there is still a lack of training-free samplers for
DPMs in the few-step sampling regime, and the sampling speed of DPMs is still a critical issue.
Customized Fast Solvers for Diffusion ODEs
As highlighted in Sec. 2.2, discretizing SDEs is generally difﬁcult in high dimensions [27, Chap.
11]and it is hard to converge within few steps.
In contrast, ODEs are easier to solve, yielding a potentialfor fast samplers.
However, as mentioned in Sec. 2.2, the general black-box ODE solver used inprevious work [3] empirically fails to converge in few steps.
This motivates us to design a dedicatedsolver for diffusion ODEs to enable fast and high-quality few-step sampling.
We start with a detailedinvestigation of the speciﬁc structure of diffusion ODEs.
3.1 Simpliﬁed Formulation of Exact Solutions of Diffusion ODEs
The key insight of this work is that given an initial value x s at time s > 0 , the solution x t at eachtime t < s of diffusion ODEs in Eq.
(2.7) can be simpliﬁed into a very special exact formulationwhich can be efﬁciently approximated.
Our ﬁrst key observation is that a part of the solution x t can be exactly computed by considering theparticular structure of diffusion ODEs.
The r.h.s.
of diffusion ODEs in Eq.
(2.7) consists of two parts:the part f ( t ) x t is a linear function of x t , and the other part g 2 ( t ) 2 σ t (cid:15) θ ( x t , t ) is generally a nonlinearfunction of x t because of the neural network (cid:15) θ ( x t , t ) .
This type of ODE is referred to as semi-linear
ODE.
The black-box ODE solvers adopted by previous work [3] are ignorant of this semi-linearstructure as they take the whole h θ ( x t , t ) in Eq.
(2.7) as the input, which causes discretization errorsof both the linear and nonlinear term.
We note that for semi-linear ODEs, the solution at time t canbe exactly formulated by the “variation of constants” formula [30]:
[Equation]
This formulation decouples the linear part and the nonlinear part.
In contrast to black-box ODEsolvers, the linear part is now exactly computed, which eliminates the approximation error of thelinear term.
However, the integral of the nonlinear part is still complicated because it couples thecoefﬁcients about the noise schedule (i.e., f ( τ ) , g ( τ ) , σ τ ) and the complex neural network (cid:15) θ , whichis still hard to approximate.
Our second key observation is that the integral of the nonlinear part can be greatly simpliﬁed byintroducing a special variable.
Let λ t :=
log( α t /σ t ) (one half of the log-SNR), then λ t is a strictlydecreasing function of t (due to the deﬁnition of DPMs as discussed in Sec. 2.1).
We can rewrite g ( t )in Eq.
(2.3) as
[Equation] [Equation]
Combining with f ( t ) =
d log α t / d t in Eq.
(2.3), we can rewrite Eq.
(3.1) as
[Equation]
As λ ( t ) =
λ t is a strictly decreasing function of t , it has an inverse function t λ ( · ) satisfyingt =
t λ ( λ ( t )) .
We further change the subscripts of x and (cid:15) θ from t to λ and denote ˆ x λ := x t λ ( λ ) ,ˆ (cid:15) θ
(ˆ x λ , λ ) := (cid:15)
θ ( x t λ ( λ ) , t λ ( λ )) .
Rewrite Eq.
(3.3) by “change-of-variable” for λ , then we have:
Proposition 3.1 (Exact solution of diffusion ODEs) .
Given an initial value x s at time s > 0 , thesolution x t at time t ∈ [0 , s ] of diffusion ODEs in Eq.
(2.7) is:
[Equation]
We call the integral (cid:82) e − λ ˆ (cid:15) θ
(ˆ x λ , λ )d λ the exponentially weighted integral of ˆ (cid:15) θ , which is veryspecial and highly related to the exponential integrators in the literature of ODE solvers [25].
To thebest of our knowledge, such formulation has not been revealed in prior work of diffusion models.
Eq.
(3.4) provides a new perspective for approximating the solutions of diffusion ODEs.
Speciﬁcally,given x s at time s , According to Eq.
(3.4), approximating the solution at time t is equivalent todirectly approximating the exponentially weighted integral of ˆ (cid:15) θ from λ s to λ t , which avoids theerror of the linear terms and is well-studied in the literature of exponential integrators [25, 31].
Basedon this insight, we propose fast solvers for diffusion ODEs, as detailed in the following sections.
3.2 High-Order Solvers for Diffusion ODEs
In this section, we propose high-order solvers for diffusion ODEs with convergence order guaranteeby leveraging our proposed solution formulation Eq.
The proposed solvers and analysis arehighly motivated by the methods of exponential integrators [25, 31] in the ODE literature.
Speciﬁcally, given an initial value x T at time T and M + 1 time steps { t i } Mi =0 decreasing fromt
= T to t M = 0 .
Let ˜ x t
= x T be the initial value.
The proposed solvers use M steps to iterativelycompute a sequence { ˜ x t i } Mi =0 to approximate the true solutions at time steps { t i } Mi =0 .
In particular,the last iterate ˜ x t M approximates the true solution at time 0 .
In order to reduce the approximation error between ˜ x t M and the true solution at time 0 , we need toreduce the approximation error for each ˜ x t i at every step [30].
Starting with the previous value ˜ x t i − 1at time t i − 1 , according to Eq.
(3.4), the exact solution x t i − 1 → t i at time t i is given by
[Equation]
Therefore, to compute the value ˜ x t i for approximating x t i − 1 → t i , we need to approximate theexponentially weighted integral of ˆ (cid:15) θ from λ t i − 1 to λ t i .
Denote h
i :=
λ t i − λ t
i − 1 , and ˆ (cid:15) ( n )
θ (ˆ x λ , λ ) :
=d n
[Equation]
d λ n as the n -th order total derivative of ˆ (cid:15) θ (ˆ x λ , λ ) w.r.t.
λ .
For k ≥ 1 , the ( k − 1) -th order
Taylor expansion of ˆ (cid:15) θ (ˆ x λ , λ ) w.r.t.
λ at λ t i − 1 is
[Equation]
Substituting the above Taylor expansion into Eq.
(3.5) yields
[Equation]
where the integral (cid:82)
[Equation]
n n !
d λ can be analytically computed by repeatedly applying n timesof integration-by-parts (see Appendix B.2).
Therefore, to approximate x t i − 1 → t i , we only need toapproximate the n -th order total derivatives ˆ (cid:15) ( n ) θ ( ˆ x λ , λ ) for n ≤ k − 1 , which is a well-studiedproblem in the ODE literature [31, 32].
By dropping the O ( h k +1 i ) error term and approximating theﬁrst ( k − 1) -th total derivatives with the “stiff order conditions” [31, 32], we can derive k -th-order
ODE solvers for diffusion ODEs.
We name such solvers as DPM-Solver overall, and DPM-Solver- kfor a speciﬁc order k .
Here we take k = 1 for demonstration.
In this case, Eq.
(3.6) becomes
[Equation]
By dropping the high-order error term O ( h 2 i ) , we can obtain an approximation for x t i − 1 → t i .
As 175 k = 1 here, we call this solver DPM-Solver-1 , and the detailed algorithm is as following.
DPM-Solver-1.
Given an initial value x T and M + 1 time steps { t i } Mi =
decreasing from t 0 =
Tto t M = 0 .
Starting with ˜ x t 0 = x T , the sequence { ˜ x t i } Mi =1 is computed iteratively as follows:
[Equation]
(cid:15) θ (˜ x t
[Equation]
i
[Equation]
) , where h i = λ t i − λ t
i − 1 .
For k ≥ 2 , approximating the ﬁrst k terms of the Taylor expansion needs additional intermediatepoints between t and s [31].
The derivation is more technical so we defer it to Appendix B. Below wepropose algorithms for k = 2 , 3 and name them as DPM-Solver-2 and DPM-Solver-3 , respectively. [Table] [Table] [Table]
[Table]
Here, t λ ( · ) is the inverse function of λ ( t ) , which has an analytical formulation for the practical noiseschedule used in [2, 16], as shown in Appendix D. The chosen intermediate points are ( s i , u i ) for
DPM-Solver-2 and ( s
i − 1 , u 2 i − 1 ) and ( s 2 i , u 2 i ) for DPM-Solver-3.
As shown in the algorithm,
DPM-Solver- k requires k function evaluations per step for k = 1 , 2 , 3 .
Despite the more expensivesteps, higher-order solvers ( k = 2 , 3 ) are usually more efﬁcient since they require much fewer stepsto converge, due to their higher convergence order.
We show that DPM-Solver- k is k -th-order solver,as stated in the following theorem.
The proof is in Appendix B.
Theorem 3.2 (DPM-Solver- k as a k -th-order solver) .
Assume (cid:15) θ
( x t , t ) follows the regularity conditions detailed in Appendix B.1, then for k = 1 , 2 , 3 , DPM-Solver- k is a k -th order solver for diffusion
ODEs, i.e., for the sequence { ˜ x t i } Mi =1 computed by DPM-Solver- k , the approximation error at time 0satisﬁes ˜ x
t M − x 0 =
O ( h k max ) , where h max = max 1 ≤ i ≤ M ( λ t i − λ t
i − 1 ) .
Finally, solvers with k ≥ 4 need much more intermediate points as shown by previous work [31, 32]for exponential integrators.
Therefore, we only consider k from 1 to 3 in this work, while leaving thesolvers with higher k for future study.
3.3 Step Size Schedule
The proposed solvers in Sec. 3.2 need to specify the time steps { t i } Mi =0 in advance.
We propose twochoices of the time step schedule.
One choice is handcrafted, which is to uniformly split the interval[ λ T , λ 0 ], i.e. λ t i = λ
T + iM ( λ 0 − λ T ) , i = 0 , . .
, M .
Note that this is different from previouswork [2, 3] which chooses uniform steps for t i .
Empirically, DPM-Solver with uniform time stepsλ t i can already generate quite good samples in few steps, where results are listed in Appendix E. Asthe other choice, we propose an adaptive step size algorithm, which dynamically adjusts the step sizeby combining different orders of DPM-Solver.
The adaptive algorithm is inspired by [20] and wedefer its implementation details to Appendix C.
For few-step sampling, we need to use up all the number of function evaluations (NFE).
When the
NFE is not divisible by 3 , we ﬁrstly apply DPM-Solver-3 as much as possible, and then add a singlestep of DPM-Solver-1 or DPM-Solver-2 (dependent on the reminder of K divided by 3 ), as detailedin Appendix D. In the subsequent experiments, we use such combination of solvers with the uniformstep size schedule for NFE ≤ 20 , and otherwise the adaptive step size schedule.
3.4 Sampling from Discrete-Time DPMs
Discrete-time DPMs [2] train the noise prediction model at N ﬁxed time steps { t n } Nn =1 , and thenoise prediction model is parameterized by ˜ (cid:15) θ ( x n , n ) for n = 0 , . . .
, N − 1 , where each x n iscorresponding to the value at time t n +1 .
We can transform the discrete-time noise prediction model tothe continuous version by letting (cid:15) θ ( x , t ) :
=
˜ (cid:15) θ ( x , ( N − 1) t T ) , for all x ∈ R d , t ∈ [0 , T ] .
Note that theinput time of ˜ (cid:15) θ may not be integers, but we ﬁnd that the noise prediction model can still work well,and we hypothesize that it is because of the smooth time embeddings (e.g., position embeddings [2]).
By such reparameterization, the noise prediction model can adopt the continuous-time steps as input,and thus we can also use DPM-Solver for fast sampling.
Comparison with Existing Fast Sampling Methods
Here, we discuss the relationship and highlight the difference between DPM-Solver and existing
ODE-based fast sampling methods for DPMs.
We further brieﬂy discuss the advantage of training-freesamplers over those training-based ones.
4.1 DDIM as DPM-Solver-1
Denoising Diffusion Implicit Models (DDIM) [19] design a deterministic method for fast samplingfrom DPMs.
For two adjacent time steps t i − 1 and t i , assume that we have a solution ˜ x t i − 1 at time 226 t
i − 1 , then a single step of DDIM from time t i − 1 to time t
i is
[Equation]
Although motivated by entirely different perspectives, we show that the updates of DPM-Solver-1and Denoising Diffusion Implicit Models (DDIM) [19] are identical.
By the deﬁnition of λ , we haveσ ti − 1 α ti − 1 =
e − λ ti − 1 and σ ti α ti =
e − λ ti .
Plugging these and h i =
λ t i − λ t
i − 1 to Eq.
(4.1) results in exactlya step of DPM-Solver-1 in Eq.
However, the semi-linear ODE formulation of DPM-Solverallows for principled generalization to higher-order solvers and convergence order analysis.
Recent work [13] also show that DDIM is a ﬁrst-order discretization of diffusion ODEs by differentiating both sides of Eq.
However, they cannot explain the difference between DDIM and theﬁrst-order Euler discretization of diffusion ODEs.
In contrast, by showing that DDIM is a specialcase of DPM-Solver, we reveal that DDIM makes full use of the semi-linearity of diffusion ODEs,which explains its superiority over traditional Euler methods.
4.2 Comparison with Traditional Runge-Kutta Methods
One can obtain a high-order solver by directly applying traditional explicit Runge-Kutta (RK) methodsto the diffusion ODE in Eq.
Speciﬁcally, RK methods write the solution of Eq.
(2.7) in thefollowing integral form:
[Equation]
and use some intermediate time steps between [ t, s ] and combine the evaluations of h θ at these timesteps to approximate the whole integral.
The approximation error of explicit RK methods depends onh θ , which consists of the error corresponding to both the linear term f ( τ ) x τ and the nonlinear noiseprediction model (cid:15) θ .
However, the error of the linear term may increase exponentially because theexact solution of the linear term has an exponential coefﬁcient (as shown in Eq.
There are manyempirical evidence [25, 31] showing that directly using explicit RK methods for semi-linear ODEsmay suffer from unstable numerical issues for large step size.
We also demonstrate the empiricaldifference of the proposed DPM-Solver and the traditional explicit RK methods in Sec.
5.1, whichshows that DPM-Solver have smaller discretization errors than the RK methods with the same order.
4.3 Training-based Fast Sampling Methods for DPMs
Samplers that need extra training or optimization include knowledge distillation [13, 14], learningthe noise level or variance [15, 16, 33], and learning the noise schedule or sample trajectory [17, 18].
Although the progressive distillation method [13] can obtain a fast sampler within 4 steps, it needsfurther training costs and loses part of the information in the original DPM (e.g., after distillation, thenoise prediction model cannot predict the noise (score function) at every time step between [0 , T ] ).
In contrast, training-free samplers can keep all the information of the original model, and thereby canbe directly extended to the conditional sampling by combining the original model and an externalclassiﬁer [4] (e.g. see Appendix D for the conditional sampling with classiﬁer guidance).
Beyond directly designing fast samplers for DPMs, several works also propose novel types of
DPMs which supports faster sampling. For instance, deﬁning a low-dimensional latent variable for DPMs [34]; designing special diffusion processes with bounded score functions [35]; combining
GANs with the reverse process of DPMs [36].
The proposed DPM-Solver may also be suitable foraccelerating the sampling of these DPMs, and we leave them for future work.
[Table]
[Figure]
In this section, we show that as a training-free sampler, DPM-Solver can greatly speedup the samplingof existing pre-trained DPMs, including both continuous-time and discrete-time ones, with bothlinear noise schedule [2, 19] and cosine noise schedule [16].
We vary different number of functionevaluations (NFE) which is the number of calls to the noise prediction model (cid:15) θ ( x t , t ) , and comparethe sample quality between DPM-Solver and other methods.
For each experiment, We draw 50Ksamples and use the widely adopted FID score [37] to evaluate the sample quality, where lower FIDusually implies better sample quality.
Unless explicitly mentioned, we always use the solver combination with the uniform step sizeschedule in Sec.
3.3 if the NFE budget is less than 20, and otherwise the DPM-Solver-3 with theadaptive step size schedule in Sec.
We refer to Appendix D for other implementation details of
DPM-Solver and Appendix E for detailed settings.
5.1 Comparison with Continuous-Time Sampling Methods
We ﬁrstly compare DPM-Solver with other continuous-time sampling methods for DPMs.
Thecompared methods include the Euler-Maruyama discretization for diffusion SDEs [3], the adaptivestep size solver for diffusion SDEs [20] and the RK methods for diffusion ODEs [3, 28] in Eq.
We compare these methods for sampling from a pre-trained continuous-time “VP deep” model [3] onthe CIFAR-10 dataset [29] with the linear noise schedule.
Fig.
2a shows the efﬁciency of compared solvers.
We use uniform time steps with 50, 200, 1000 NFEfor the diffusion SDE with Euler discretization, and vary the tolerance hyperparameter [3, 20] for theadaptive step size SDE solver [20] and RK45 ODE solver [28] to control the NFE.
DPM-Solver cangenerate good sample quality within around 10 NFE, while other solvers have large discretizationerror even in 50 NFE, which shows that DPM-Solver can achieve ∼ 5 speedup of the previous bestsolver.
In particular, we achieve 4.70 FID with 10 NFE, 3.75 FID with 12 NFE, 3.24 FID with
NFE, and 2.87 FID with 20 NFE, which is the fastest sampler on CIFAR-10.
As an ablation study, we also compare the second-order and third-order DPM-Solver and RK methods,as shown in Table 1.
We compare RK methods for diffusion ODEs w.r.t.
both time t in Eq.
(2.7) andhalf-log-SNR λ by applying change-of-variable (see detailed formulations in Appendix E.1).
Theresults show that given the same NFE, the sample quality of DPM-Solver is consistently better than
RK methods with the same order.
The superior efﬁciency of DPM-Solver is particularly evident inthe few-step regime under 15 NFE, where RK methods have rather large discretization errors.
Thisis mainly because DPM-Solver analytically computes the linear term, avoiding the correspondingdiscretization error.
Besides, the higher order DPM-Solver-3 converges faster than DPM-Solver-2,which matches the order analysis in Theorem 3.2.
5.2 Comparison with Discrete-Time Sampling Methods
We use the method in Sec.
3.4 for using DPM-Solver in discrete-time DPMs, and then compare DPM-Solver with other discrete-time training-free samplers, including DDPM [2], DDIM [19],
Analytic-DDPM [21], Analytic-DDIM [21], PNDM [22], FastDPM [38] and Itô-Taylor [24].
Wealso compare with GGDM [18], which uses the same pre-trained model but needs further training forthe sampling trajectory.
We compare the sample quality by varying NFE from 10 to 1000.
Speciﬁcally, we use the discrete-time model trained by L simple in [2] on the CIFAR-10 dataset withlinear noise schedule; the discrete-time model in [19] on CelebA 64x64 [39] with linear noiseschedule; the discrete-time model trained by L hybrid in [16] on ImageNet 64x64 [26] with cosinenoise {+schedule; the discrete-time model with classiﬁer guidance in [4] on ImageNet 128x
 [26]with linear noise schedule; the discrete-time model in [4] on LSUN bedroom 256x256 [40] withlinear noise schedule. For the models trained on ImageNet, we+} only use their “mean” model andomit the “variance” model.
As shown in Fig.
2, on all datasets, DPM-Solver can obtain reasonablesamples within 12 steps (FID 4.65 on CIFAR-10, FID 3.71 on CelebA 64x64 and FID 19.97 on
ImageNet 64x64, FID 4.08 on ImageNet 128x128), which is 4 ∼ 16 × faster than the previous fastesttraining-free sampler.
DPM-Solver even outperforms GGDM, which requires additional training.
Conclusion
We tackle the problem of fast and training-free sampling from DPMs.
We propose DPM-Solver,a fast dedicated training-free solver of diffusion ODEs for fast sampling of DPMs in around 10steps of function evaluations.
DPM-Solver leverages the semi-linearity of diffusion ODEs andit directly approximates a simpliﬁed formulation of exact solutions of diffusion ODEs, whichconsists of an exponentially weighted integral of the noise prediction model.
Inspired by numericalmethods for exponential integrators, we propose ﬁrst-order, second-order and third-order DPMSolver to approximate the exponentially weighted integral of noise prediction models with theoreticalconvergence guarantee.
We propose both handcrafted and adaptive step size schedule, and apply
DPM-Solver for both continuous-time and discrete-time DPMs.
Our experimental results show that
DPM-Solver can generate high-quality samples in around 10 function evaluations on various datasets,and it can achieve 4 ∼ 16 × speedup compared with previous state-of-the-art training-free samplers.
Limitations and broader impact Despite the promising speedup performance, DPM-Solver isdesigned for fast sampling, which may be not suitable for accelerating the likelihood evaluations of
DPMs.
Besides, compared to the commonly-used GANs, diffusion models with DPM-Solver are stillnot fast enough for real-time applications.
In addition, like other deep generative models, DPMs maybe used to generate adverse fake contents, and the proposed solver may further amplify the potentialundesirable inﬂuence of deep generative models for malicious applications.
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Seesection 6.(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix B.(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Code is attached in the supplemental materials, with the appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Our method is training-free.
But we also report the hyperparameters
for evaluations used in our proposed solver.
(c) Did you report error bars (e.g., with respect to the random seed after running ex-
periments multiple times)?
[No] We observe that the standard deviation of the FID
evaluations of DPM-Solver are rather small (mainly less than 0.01) because the FID is already averaged over 50K samples, following existing work [18, 20, 21].
The small standard deviation does not change the conclusion.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] The GPU type and amount is
detailed in Appendix E.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See Appendix E
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include our code in the supplemental materials.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] All of the datasets used in the experiments are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] We mentioned the human privacy issues of the
ImageNet dataset in Appendix E.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table]
[Table]
[Table]