GENIE: Higher-Order Denoising Diffusion Solvers
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Denoising diffusion models (DDMs) have emerged as a powerful class of generativemodels.
A forward diffusion process slowly perturbs the data, while a deep modellearns to gradually denoise.
Synthesis amounts to solving a differential equation(DE) deﬁned by the learnt model.
Solving the DE requires slow iterative solversfor high-quality generation.
In this work, we propose Higher-Order
Denoising
Diffusion Solvers (GENIE): Based on truncated Taylor methods, we derive a novelhigher-order solver that signiﬁcantly accelerates synthesis.
Our solver relies onhigher-order gradients of the perturbed data distribution, that is, higher-order scorefunctions.
In practice, only Jacobian-vector products (JVPs) are required and wepropose to extract them from the ﬁrst-order score network via automatic differentiation.
We then distill the JVPs into a separate neural network that allows us to efﬁciently compute the necessary higher-order terms for our novel sampler during synthesis.
We only need to train a small additional head on top of the ﬁrst-order scorenetwork.
We validate GENIE on multiple image generation benchmarks and demonstrate that GENIE outperforms all existing solvers.
Unlike recent methods that fundamentally alter the generation process in DDMs, our GENIE solves the true generative DE and still enables applications such as encoding and guided sampling.
Introduction
Denoising diffusion models (DDMs) offer both state-of-the-art synthesis quality and sample diversityin combination with a robust and scalable learning objective.
DDMs have been used for image [1–5]and video [6, 7] synthesis, super-resolution [8, 9], deblurring [10, 11], image editing and inpainting [5,12–14], text-to-image synthesis [15–17], conditional and semantic image generation [18–22], imageto-image translation [14, 23, 24] and for inverse problems in medical imaging [25–31].
They alsoenable high-quality speech synthesis [32–37], 3D shape generation [38–42], molecular modeling [43–46], maximum likelihood training [47–50], and more [51–56].
In DDMs, a diffusion process graduallyperturbs the data towards random noise, while a deep neural network learns to denoise.
Formally, theproblem reduces to learning the score function , i.e., the gradient of the log-density of the perturbeddata.
The (approximate) inverse of the forward diffusion can be described by an ordinary or astochastic differential equation (ODE or SDE, respectively), deﬁned by the learned score function,and can therefore be used for generation when starting from random noise [47, 57].
A crucial drawback of DDMs is that the generative ODE or SDE is typically difﬁcult to solve,due to the complex score function.
Therefore, efﬁcient and tailored samplers are required forfast synthesis.
In this work, building on the generative ODE [47, 57, 58], we rigorously derive anovel second-order ODE solver using truncated Taylor methods [59].
These higher-order methodsrequire higher-order gradients of the ODE—in our case this includes higher-order gradients of the log-density of the perturbed data, i.e., higher-order score functions.
Because such higher-orderscores are usually not available, existing works typically use simple ﬁrst-order solvers or samplerswith low accuracy [1, 57, 58, 60], higher-order methods that rely on suboptimal ﬁnite difference orother approximations [61–63], or alternative approaches [64–66] for accelerated sampling.
Here,we fundamentally avoid such approximations and directly model the higher-order gradient terms:
Importantly, our novel Higher-Order Denoising Diffusion Solver (GENIE) relies on Jacobian-vectorproducts (JVPs) involving second-order scores.
We propose to calculate these JVPs by automaticdifferentiation of the regular learnt ﬁrst-order scores.
For computational efﬁciency, we then distillthe entire higher-order gradient of the ODE, including the JVPs, into a separate neural network.
Inpractice, we only need to add a small head to the ﬁrst-order score network to predict the componentsof the higher-order ODE gradient.
By directly modeling the JVPs we avoid explicitly forminghigh-dimensional higher-order scores.
Intuitively, the higher-order terms in GENIE capture the localcurvature of the ODE and enable larger steps when iteratively solving the generative ODE (Fig. 1).
Experimentally, we validate GENIE on multiple image modeling benchmarks and achieve state-ofthe-art performance in solving the generative ODE of DDMs with few synthesis steps.
In contrast torecent methods that fundamentally modify the generation process of DDMs by training conditional
GANs [67] or by distilling the full sampling trajectory [68, 69], GENIE solves the true generative
ODE.
Therefore, we also show that we can still encode images in the DDM’s latent space, as requiredfor instance for image interpolation, and use techniques such as guided sampling [4, 57, 70].
We make the following contributions: (i)
We introduce GENIE, a novel second-order ODE solver forfast DDM sampling.
(ii) We propose to extract the required higher-order terms from the ﬁrst-orderscore model by automatic differentiation.
In contrast to existing works, we explicitly work withhigher-order scores without ﬁnite difference approximations.
To the best of our knowledge, GENIE isthe ﬁrst method that explicitly uses higher-order scores for generative modeling with DDMs.
(iii) Wepropose to directly model the necessary JVPs and distill them into a small neural network.
(iv) Weoutperform all existing solvers and samplers for the generative differential equations of DDMs.
Background
We consider continuous-time DDMs [1, 57, 71] whose forward process can be described by
[Equation]
where x 0 ⇠ p 0 ( x 0 ) is drawn from the empirical data distribution and x t refers to diffused datasamples at time t 2 [0 , 1] along the diffusion process.
The functions ↵ t and  t are generally chosensuch that the logarithmic signal-to-noise ratio [48] log ↵ 2 t  2 t decreases monotonically with t and the datadiffuses towards random noise, i.e., p 1 ( x 1 ) ⇡ N ( x 1 ; 0 , I ) .
We use variance-preserving [57] diffusionprocesses for which  2 t = 1  ↵ 2 t (however, all methods introduced in this work are applicable tomore general DDMs).
The diffusion process can then be expressed by the (variance-preserving) SDE
[Equation]
where  t =  ddt log ↵ 2 t , x 0 ⇠ p 0 ( x 0 ) and w t is a standard Wiener process.
A corresponding reversediffusion process that effectively inverts the forward diffusion is given by [57, 72, 73]
[Equation]
and this reverse-time generative SDE is marginally equivalent to the generative ODE [47, 57]
[Equation]
where r x t log p t ( x t ) is the score function .
Eq.
(4) is referred to as the Probability Flow ODE [57],an instance of continuous Normalizing ﬂows [74, 75].
To generate samples from the DDM, one cansample x 1 ⇠ N ( x 1 ; 0 , I ) and numerically simulate either the Probability Flow ODE or the generative
SDE, replacing the unknown score function by a learned score model s ✓ ( x t , t ) ⇡ r x t log p t ( x t ) .
The DDIM solver [58] has been particularly popular to simulate DDMs due to its speed and simplicity.
It has been shown that DDIM is Euler’s method applied to an ODE based on a re-parameterization ofthe Probability Flow ODE [58, 69]: Deﬁning  t = q 1  ↵ 2 t ↵ 2 t and ¯ x t = x t p 1 +  2 t , we have
[Equation]
where we inserted Eq.
(4) for d x t dt and used  ( t ) dtd  t = 2  t  2 t +1 .
Letting s ✓ ( x t , t ) :
=  ✏ ✓ ( x t ,t )  t denotea parameterization of the score model, the approximate generative DDIM ODE is then given by
[Equation]
where we used  t
= p
[Equation]  [Equation] p  2 t +1 (see App.
A for a more detailed derivation of Eq.
Themodel ✏ ✓ ( x t , t ) can be learned by minimizing the score matching objective [1, 76]
[Equation]
for small 0 < t cuto ↵ ⌧ 1 .
As is standard practice, we set g ( t ) = 1 .
Other weighting functions g ( t )are possible; for example, setting g ( t ) =
t 2  2 t recovers maximum likelihood learning [47–50].
Higher-Order Denoising Diffusion Solver
As discussed in Sec. 2, the so-known DDIM solver [58] is simply Euler’s method applied to the DDIM
ODE (cf.
Eq.
In this work, we apply a higher-order method to the DDIM ODE, building onthe truncated Taylor method (TTM) [59].
The p -th TTM is simply the p -th order Taylor polynomialapplied to an ODE.
For example, for the general d y dt = f
( y , t ) , the p -th TTM reads as
[Equation]
where h n = t n +1  t n .
Note that the ﬁrst TTM is simply Euler’s method.
Applying the second TTMto the DDIM ODE results in the following scheme
[Equation]
where h n =  t n +1   t n .
Recall that  t = q [Equation]  ↵ 2 t ↵ 2 t , where the function ↵ t is a time-dependenthyperparameter of the DDM.
The total derivative d  t ✏ ✓ := d ✏ ✓ d  t can be decomposed as follows
[Equation]
[Equation]
where @ ✏ ✓ ( x t ,t ) @ x t denotes the Jacobian of ✏ ✓ ( x t , t ) and
[Equation]
If not explicitly stated otherwise, we refer to the second TTM applied to the DDIM ODE, i.e., thescheme in Eq.
(9), as Higher-Order Denoising Diffusion Solver (GENIE).
Intuitively, the higher-ordergradient terms used in the second TMM model the local curvature of the ODE.
This translates into a
Taylor formula-based extrapolation that is quadratic in time (cf.
Eqs.
(8) and (9)) and more accuratethan linear extrapolation, as in Euler’s method, thereby enabling larger time steps (see Fig.
1 for avisualization).
In App.
B, we also discuss the application of the third TTM to the DDIM ODE.
Weemphasize that TTMs are not restricted to the DDIM ODE and could just as well be applied to the
Probability Flow ODE [57] (also see App. B) or neural ODEs [74, 75] more generally.
[Figure]
[Figure]
The Beneﬁt of Higher-Order Methods: We showcase the beneﬁtof higher-order methods on a 2D toy distribution (Fig. 2a) for whichwe know the score function as well as all higher-order derivativesnecessary for GENIE analytically.
We generate 1k different accurate “ground truth” trajectories x t using DDIM with 10k steps.
Wecompare these “ground truth” trajectories to single steps of DDIMand GENIE for varying step sizes  t .
We then measure the mean
L 2 -distance of the single steps ˆ x t (  t ) to the “ground truth” trajectories x t , and we repeat this experiment for three starting pointst 2 { 0 . 1 , 0 . 2 , 0 . 5 } .
We see (Fig. 3 ( top )) that GENIE can use largerstep sizes to stay within a certain error tolerance for all startingpoints t .
We further show samples for DDIM and GENIE, using 25solver steps, in Fig. 2.
DDIM has the undesired behavior of samplinglow-density regions between modes, whereas GENIE looks like aslightly noisy version of the ground truth distribution (Fig. 2a).
Comparison to Multistep Methods: Linear multistep methods arean alternative higher-order method to solve ODEs.
Liu et al.
[63]applied the well-established Adams–Bashforth [AB, 77] method tothe DDIM ODE.
AB methods can be derived from TTMs by approximating higher-order derivatives d p y dt p using the ﬁnite difference method [78].
For example, the second AB method is obtainedfrom the second TTM by replacing d 2 y dt 2 with the ﬁrst-order forward difference approximation( f ( y t n , t n )  f ( y t n  1 , t n  1 )) /h n  1 .
In Fig.
3 ( bottom ), we visualize the mean L 2 -norm of thedifference ⇠ t (  t ) between the analytical derivative d  t ✏ ✓ and its ﬁrst-order forward difference approximation for varying step sizes  t for the 2D toy distribution.
The approximation is especially poorat small t for which the score function becomes complex
(App. E for details on all toy experiments).
3.1 Learning Higher-Order Derivatives
The above observations inspire to apply GENIE to DDMs of more complex and high-dimensional datasuch as images.
Regular DDMs learn a model ✏ ✓ for the ﬁrst-order score; however, the higher-ordergradient terms required for GENIE (cf. Eq.
(10)) are not immediately available to us, unlike in thetoy example above.
Let us insert Eq.
(11) into Eq.
(10) and analyze the required terms more closely:
[Equation]
|
[Equation]
[Equation] [Equation] | {z } JVP+ @
✏ ✓ (
[Equation]
t
[Equation]
We see that the full derivative decomposes into two JVP terms and one simpler time derivative term.
The term @ ✏ ✓ ( x t ,t ) @ x t plays a crucial role in Eq.
It can be expressed as
[Equation]
which means that GENIE relies on second-order score functions r > x t r x t log p t ( x t ) under the hood.
Given a DDM, that is, given ✏ ✓ , we could compute the derivative d  t ✏ ✓ for the GENIE schemein Eq.
(9) using automatic differentiation (AD).
This would, however, make a single step of GENIE at least twice as costly as DDIM, because we would need a forward pass through the ✏ ✓ network to compute ✏ ✓ ( x t , t ) itself, and another pass to compute the JVPs and the time derivative in Eq.
Theseforward passes cannot be parallelized, since the vector-part of JVP 1 in Eq.
(12) involves ✏ ✓ itself, andneeds to be known before computing the JVP.
To accelerate sampling, this overhead is too expensive.
Gradient Distillation: To avoid this overhead, wepropose to ﬁrst distill d  t ✏ ✓ into a separate neuralnetwork.
During distillation training, we can usethe slow AD-based calculation of d  t ✏ ✓ , but during synthesis we call the trained neural network.
We build on the observation that the internal representations of the neural network modeling ✏ ✓ (inour case a U-Net [79] architecture) can be used fordownstream tasks [80, 81]: speciﬁcally, we providethe last feature layer from the ✏ ✓ network together with its time embedding as well as x t and the output✏ ✓ ( x t , t ) to a small prediction head k ( x t , t ) that models the different terms in Eq.
(12) (see Fig. 4).
The overhead generated by k is small, for instance less than 2% for our CIFAR-10 model (alsosee Sec. 5), and we found this approach to provide excellent performance.
Note that in principle wecould also train an independent deep neural network, which does not make use of the internal representations of ✏ ✓ and could therefore theoretically be run in parallel to the ✏ ✓ model.
We justify using smallprediction heads over independent neural networks because AD-based distillation training is slow: ineach training iteration we ﬁrst need to call the ✏ ✓ network, then calculate the JVP terms, and only thencan we call the distillation model.
By modeling d  t ✏ ✓ via small prediction heads, while reusing theinternal representation of the score model, we can make training relatively fast: we only need to traink for up to 50k iterations.
In contrast, training score models from scratch takes roughly an order ofmagnitude more iterations.
We leave training of independent networks to predict d  t ✏ ✓ to future work.
Mixed Network Parameterization:
We found that learning d  t ✏ ✓ directly as single output of aneural network can be challenging.
Assuming a single data point distribution p 0 ( x 0 )
=
( x 0 = 0 ) ,for which we know the diffused score function and all higher-order derivatives analytically, we foundthat the terms in Eq.
(12) all behave very differently within the t 2 [0 , 1] interval (for instance, theprefactor of JVP 1 in Eq.
(12) approaches 1 as t ! 0 , while JVP 2 ’s prefactor vanishes).
As outlinedin detail in App.
C.2.3, this simple single data point assumption implies an effective mixed networkparameterization , an approach inspired by the “mixed score parametrizations” in Vahdat et al.
[49]and Dockhorn et al.
In particular, we model
[Equation]
where k ( i ) ( x t , t ) , i 2 { 1 , 2 , 3 } , are different output channels of the neural network (i.e. the additionalhead on top of the ✏ ✓ network).
The three terms in Eq.
(14) exactly correspond to the three termsof Eq.
(12), in the same order.
We show the superior performance of this parametrization in Sec.
Learning Objective: We would like our model k to match d  t ✏ ✓ for all t and x t .
This suggests asimple (weighted)
L 2 -loss for distillation, similar to regular score matching losses for DDMs [1, 57],
[Equation] [Equation]
where x t = ↵ t x 0 +  t ✏ . We choose the weighting function g d ( t ) =
2 t to counteract the division by t (note  0 = 0 ) in the ﬁrst and third term of our mixed network parameterization in Eq.
Wefound that this leads to a roughly constant loss over different time values t .
We emphasize that duringtraining we need to compute d  t ✏ ✓ via AD; however, at inference time we use the learned predictionhead k to approximate d  t ✏ ✓ when sampling with GENIE.
In App.
C.2.4, we provide pseudo codefor training and sampling with prediction heads k .
Alternative Learning Approaches: As shown in Eq.
(13), GENIE relies on second-order scorefunctions.
Recently, Meng et al.
[82] directly learnt such higher-order scores with higher-order scorematching objectives.
Directly applying these techniques has the downside that we would need to explicitly form the higher-order score terms r > x t ✏ ✓ ( x t , t ) , which are very high-dimensional for data such as images.
Low-rank approximations are possible, but potentially insufﬁcient for high performance.
Inour approach, we are avoiding this complication by directly modeling the lower-dimensional JVPs.
Wefound that the methods from Meng et al.
[82] can be modiﬁed to provide higher-order score matchingobjectives for the JVP terms required for GENIE and we brieﬂy explored this (see App. D).
However,our distillation approach with AD-based higher-order gradients worked much better.
Nevertheless,this is an interesting direction for future research.
To the best of our knowledge, GENIE is the ﬁrstsolver for the generative differential equations of DDMs that directly uses higher-order scores (in theform of the distilled JVPs) for generative modeling without ﬁnite difference or other approximations.
Related Work
Accelerated Sampling from DDMs.
Several previous works address the slow sampling of DDMs:
One line of work reduces and readjusts the timesteps [3, 64] used in time-discretized DDMs [1, 71].
This can be done systematically by grid search [32] or dynamic programming [83].
Bao et al.
[65]speed up sampling by deﬁning a new DDM with optimal reverse variances.
DDIM [58], discussed in
Sec. 2, was also introduced as a method to accelerate DDM synthesis.
Further works leverage modern
ODE and SDE solvers for fast synthesis from (continuous-time) DDMs: For instance, higher-order
Runge–Kutta methods [57, 84] and adaptive step size SDE solvers [62] have been used.
Thesemethods are not optimally suited for the few-step synthesis regime, in which GENIE shines; seealso Sec. 5.
Most closely related to our work is Liu et al.
[63], which simulates the DDIM ODE [58]using a higher-order linear multistep method [77].
As shown in Sec. 3, linear multistep methods can beconsidered an approximation of the TTMs used in GENIE.
Furthermore, Tachibana et al.
[61] solve thegenerative SDE via a higher-order Itô–Taylor method [59] and, in contrast to our work, approximatehigher-order score functions with the “ideal derivative trick”: they essentially assume a simplesingle-point data distribution for which higher-order score functions can be computed analytically.
Alternatively, sampling from DDMs can also be accelerated via learning: For instance, Watson et al.[66] learn parameters of a generalized family of DDMs by optimizing for perceptual output quality.
Luhman and Luhman [68] and Salimans and Ho [69] distill a DDIM sampler into a student model,which enables sampling in as few as a single step.
Xiao et al.
[67] replace DDMs’ Gaussian samplerswith expressive generative adversarial networks, similarly allowing for few-step synthesis.
GENIEcan also be considered a learning-based approach, as we distill a derivative of the generative ODE intoa separate neural network.
However, in contrast to the mentioned methods, GENIE still solves the trueunderlying generative ODE, which has major advantages: for instance, it can still be used easily forclassiﬁer-guided sampling [4, 57, 70] and to efﬁciently encode data into latent space—a prerequisitefor likelihood calculation [47, 57] and editing applications [17].
Note that the learnt sampler [66]deﬁnes a proper probabilistic generalized DDM; however, it isn’t clear how it relates to the generative
SDE or ODE and therefore how compatible the method is with applications such as classiﬁer guidance.
Other approaches to accelerate DDM sampling change the diffusion itself [60, 85, 86] or train DDMsin the latent space of a Variational Autoencoder [49].
GENIE is complementary to these methods.
Higher-Order ODE Gradients beyond DDMs.
TTMs [78] and other methods that leverage higherorder gradients are also applied outside the scope of DDMs.
For instance, higher-order derivativescan play a crucial role when developing solvers [87] and regularization techniques [88, 89] for neural
ODEs [74, 75].
Outside the ﬁeld of machine learning, higher-order TTMs have been widely studied,for example, to develop solvers for stiff [90] and non-stiff [90, 91] systems.
Experiments
Datasets: We run experiments on four datasets: CIFAR-10 [92] (resolution 32), LSUN Bedrooms [93](128), LSUN Church-Outdoor [93] (128), and (conditional)
ImageNet [94] (64).
Architectures: Except for CIFAR-10 (we use a checkpoint by Song et al.
[57]), we train ourown score models using architectures introduced by previous works [1, 4].
The architecture of ourprediction heads is based on (modiﬁed) BigGAN residual blocks [57, 95].
To minimize computationaloverhead, we only use a single residual block.
See App.
C for training and architecture details.
Evaluation: We measure sample quality via Fréchet Inception Distance [FID, 96] (see App. F.1).
Synthesis Strategy: We simulate the DDIM ODE from t =1 up to t =10  3 using evaluation timesfollowing a quadratic function ( quadratic striding [58]).
For variance-preserving DDMs, it can bebeneﬁcial to denoise the ODE solver output at the cutoff t =10  3 , i.e., x 0 = x t   t ✏ ✓ ( x t ,t ) ↵ t [57, 97].
Note that the denoising step involves a score model evaluation, and therefore “loses” a functionevaluation that could otherwise be used as an additional step in the ODE solver.
To this end,denoising the output of the ODE solver is left as a hyperparameter of our synthesis strategy.
Analytical First Step (AFS): Every additional neural network call becomes crucial in the low numberof function evaluations (NFEs) regime.
We found that we can improve the performance of GENIEand all other methods evaluated on our checkpoints by replacing the learned score with the (analytical)score of N ( 0 , I ) ⇡ p t =1 ( x t ) in the ﬁrst step of the ODE solver.
The “gained” function evaluationcan then be used as an additional step in the ODE solver.
Similarly to the denoising step mentionedabove, AFS is treated as a hyperparameter of our Synthesis Strategy .
AFS details in App.
F.2.
Accounting for Computational Overhead: GENIE has a slightly increased computational overheadcompared to other solvers due to the prediction head k .
The computational overhead is increasedby 1.47%, 2.83%, 14.0%, and 14.4% on CIFAR-10, ImageNet, LSUN Bedrooms, and LSUN ChurchOutdoor, respectively (see also App. C.2.5).
This additional overhead is always accounted forimplicitly: we divide the NFEs by the computational overhead and round to the nearest integer.
Forexample, on LSUN Bedrooms, we compare baselines with 10/15 NFEs to GENIE with 9/13 NFEs.
5.1 Image Generation
In Fig.
5 we compare our method to the most competitive baselines.
In particular, on the same scoremodel checkpoints, we compare GENIE with DDIM [58], S-PNDM [63], and F-PNDM [63].
Forthese four methods, we only include the best result over the two hyperparameters discussed above,namely, the denoising step and AFS (see App.
F.7 for tables with all results).
We also include threecompetitive results from the literature [64–66] that use different checkpoints and sampling strategies:for each method, we include the best result for their respective set of hyperparameters.
We do notcompare in this ﬁgure with Knowledge Distillation [KD, 68], Progressive Distillation [PG, 69] and
Denoising Diffusion GANs [DDGAN, 67] as they do not solve the generative ODE/SDE and usefundamentally different sampling approaches with drawbacks discussed in Sec. 4.
For NFEs 2 { 10 , 15 , 20 , 25 } , GENIE outperforms all baselines (on the same checkpoint) on all fourdatasets (GENIE image samples shown in App. F.8).
On CIFAR-10 and (conditional) ImageNet,
GENIE also outperforms these baselines for NFEs=5, whereas DDIM outperforms GENIE slightlyon the LSUN datasets (see tables in App. F.7).
GENIE performs better than the three additionalbaselines from the literature (which use different checkpoints and sampling strategies) with theexception of the Learned Sampler [LS, 66] on CIFAR-10 for NFEs=5 and on LSUN Bedrooms for
NFEs=20.
The LS’s advantage can be potentially due to the different checkpoint, LS learning anoptimal striding schedule (whereas GENIE simply uses quadratic striding), or LS learning additionaldegrees of freedom within a more general class of DDMs.
For CIFAR-10, we show additionalbaseline results in Tab. 1.
In particular, we compare to a baseline where we replace the learnedderivatives in GENIE with the “ideal derivative trick” proposed in Tachibana et al.
We callthis method “2nd TTM (ID)” and refer to App.
B.1 for details of the method.
We do not includecommonly-used adaptive step size solvers in Fig.
5, as they are arguably not well-suited for this low
NFE regime: for example, on the same CIFAR-10 checkpoint we use for GENIE, the adaptive SDEsolver introduced in Jolicoeur-Martineau et al.
[62] obtains an FID of 82.4 at 48 NFEs.
Also on thesame checkpoint, the Runge–Kutta 4(5)
[84] method applied to the ProbabilityFlow ODE obtains an
FID of 13.1 at 38 NFEs (solver tolerances set to 10  2 ).
The results in Fig.
5 suggest that higher-order gradient information, as used in GENIE, can beefﬁciently leveraged for image synthesis.
Despite using small prediction heads our distillation seemsto be sufﬁciently accurate: for reference, replacing the distillation heads with the derivatives computedvia AD, we obtain FIDs of 9.22, 4.11, 3.54, 3.46 using 10, 20, 30, and 40 NFEs, respectively (NFEsadjusted assuming an additional computational overhead of 100%).
Moreover, our distillation
[Figure]
CIFARImageNet
[Figure]
GENIE (ours)
DDIM [58]
S-PNDM [63] F-PNDM [63]
Learned Sampler [66] (†)
FastDDIM [64] (†)
Analytic DDIM [65] (†)
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
FastDDIM [64] (†) -
[Figure]
[Figure]
[Figure]
Sampler [66] (†)
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] .25 Analytic DDIM (LS) [65] (†) - 14. - - 5.71 CLD-SGM [60]    106 VESDE-PC [57]     approach is signiﬁcantly better than approximating higher-order scores using methods such as in the“ideal derivative trick” [61]. As discussed in Sec. 3, linear multistep methods such as S-PNDM [63]and F-PNDM [63] can be considered (ﬁnite difference) approximations to TTMs as used in GENIE.
These approximations are inaccurate for large timesteps, which potentially explains their inferiorperformance when compared to GENIE.
When compared to DDIM, the superior performance of
GENIE seems to become less signiﬁcant for large NFE: this is in line with the theory, as higher-ordergradients contribute less for smaller step sizes (see the GENIE scheme in Eq.
Approaches suchas FastDDIM [64] and AnalyticDDIM [65], which adapt variances and discretizations of discrete-time
DDMs, are useful; however, GENIE suggests that rigorous higher-order ODE solvers leveragingthe continuous-time DDM formalism are still more powerful.
To the best of our knowledge, theonly methods that outperform GENIE abandon this ODE or SDE formulation entirely and train
NFE-speciﬁc models [67, 69] which are optimized for the single use-case of image synthesis.
5.2 Guidance and Encoding
[Figure]
As discussed in Sec. 4, one major drawback of approaches suchas KD [68], PG [69] and DDGAN [67] is that they abandon the
ODE/SDE formalism, and cannot easily use methods such asclassiﬁer(-free) guidance [57, 70] or perform image encoding.
However, these techniques can play an important role in synthesizing photorealistic images from DDMs [3, 4, 15, 17], as well as forimage editing tasks [12, 17] and to calculate likelihoods [47, 57].
Classiﬁer-Free Guidance [70]: We replace the unconditionalmodel ✏ ✓ ( x t , t ) with ˆ ✏ ✓ ( x t , t, c, w ) =
(1 + w ) ✏ ✓ ( x t , t, c ) w ✏ ✓ ( x t , t ) in the DDIM ODE (cf Eq.
(6)), where ✏ ✓ ( x t , t, c )is a conditional model and w > 1 . 0 is the “guidance scale”.
GENIE then requires the derivative
[Equation]
(16) for guidance.
Hence, we need to distill d  t ✏ ✓ ( x t , t, c ) and d  t ✏ ✓ ( x t , t ) , for which we could also shareparameters [70].
We compare GENIE with DDIM on ImageNet in Fig.
GENIE clearly outperforms
DDIM, in particular for few NFEs, and GENIE also synthesizes high-quality images (see Fig. 7).
[Figure]
Image Encoding: We can use GENIE also to solve the generative ODE in reverse to encodegiven images.
Therefore, we compare GENIE to DDIM on the “encode-decode” task, analyzingreconstructions for different NFEs (used twice for encoding and decoding):
We ﬁnd that GENIEreconstructs images much more accurately (see Fig.
For more details on this experiment as wellas the guidance experiment above, see App. F.5 and App.
F.4, respectively.
We also show latentspace interpolations for both GENIE and DDIM in App. F.6.
5.3 Ablation Studies
[Figure]
[Table]
We perform ablation studies over architecture and training objective for the predictionheads used in GENIE:
In Tab. 2, “No mixed”refers to learning d  t ✏ ✓ directly as singlenetwork output without mixed network parameterization; “No weighting” refers to setting g d ( t ) = 1in Eq.
(15); “Standard” uses both the mixed network parameterization and the weighting function
[Figure] g d ( t ) =  2 t .
We can see that having both the mixed network parametrization and the weighting function is clearly beneﬁcial.
We also tested deeper networks in the prediction heads: for “Bigger model”we increased the number of residual blocks from one to two.
The performance is roughly on par with“Standard”, and we therefore opted for the smaller head due to the lower computational overhead.
Conclusions
We introduced GENIE, a higher-order ODE solver for DDMs.
GENIE improves upon the commonlyused DDIM solver by capturing the local curvature of its ODE’s gradient ﬁeld, which allows for largerstep sizes when solving the ODE.
We further propose to distill the required higher-order derivativesinto a small prediction head—which we can efﬁciently call during inference—on top of the ﬁrst-orderscore network.
A limitation of GENIE is that it is still slightly slower than approaches that abandonthe differential equation framework of DDMs altogether, which, however, comes at the considerablecost of preventing applications such as guided sampling.
To overcome this limitation, in future workwe plan to leverage even higher-order gradients to accelerate sampling from DDMs even further.
Broader Impact.
Fast synthesis from DDMs, the goal of GENIE, can potentially make DDMs anattractive method for promising interactive generative modeling applications, such as digital contentcreation or real-time audio synthesis, and also reduce DDMs’ environmental footprint by decreasingthe computational load during inference.
Although we validate GENIE on image synthesis, it couldalso be utilized for other tasks, which makes its broader societal impact application-dependent.
Inthat context, it is important that practitioners apply an abundance of caution to mitigate impacts givengenerative modeling can also be used for malicious purposes, discussed for instance in Vaccari and
Chadwick [99], Nguyen et al.
[100], Mirsky and Lee [101].
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] On CIFAR10, we use a checkpoint provided by Song et al.
For baseline comparisons,we run publicly available code: in particular, we make use of the repositorieshttps://github.com/yang-song/score_sde_pytorch and https://github.com/nv-tlabs/CLD-SGM from previous, publicly available papers, which we cite(Song et al.
[57] and Dockhorn et al.
[60], respectively).
We also cite all used datasets: CIFAR-10 [92], LSUN Bedrooms [93], LSUN Church-Outdoor [93], andImageNet [94].
(b) Did you mention the license of the assets?
[Yes] In the Appendix, we mentionthe licenses of all codes we are running for baseline comparisons and evaluation.
See App. F.1.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] We are only using publicly availabledatasets, which have been widely used in the generative modeling literature as standardbenchmarks (CIFAR-10, ImageNet, LSUN Church-Outdoor, LSUN Bedrooms) andavoid using face image datasets.
If you used crowdsourcing or conducted research with human subjects...
[Table]
) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable? [N/A [Table] [Table]
) Did you describe any potential participant risks, with links to Institutional Review
[Table]
IRB) approvals, if applicable?
[N/A [Table]
c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]