Original: DPM-S OLVER ++: F AST S OLVER FOR G UIDED S AMPLING OF D IFFUSION P ROBABILISTIC M ODELS
Revised: DPM-S OLVER ++: F AST S OLVER FOR G UIDED S AMPLING OF D IFFUSION P ROBABILISTIC M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Diffusion probabilistic models (DPMs) have achieved impressive success in highresolution image synthesis, especially in recent large-scale text-to-image generation applications.
Revised: Diffusion probabilistic models (DPMs) have achieved impressive success in highresolution image synthesis, especially in recent large-scale text-to-image generation applications.

Original: An essential technique for improving the sample quality of DPMs is guided sampling , which usually needs a large guidance scale to obtain the best sample quality.
Revised: An essential technique for improving the sample quality of DPMs is guided sampling , which usually needs a large guidance scale to obtain the best sample quality.

Original: The commonly-used fast sampler for guided sampling is DDIM, a first-order diffusion ODE solver that generally needs 100 to 250 steps for highquality samples.
Revised: The commonly-used fast sampler for guided sampling is DDIM, a first-order diffusion ODE solver that generally needs 100 to 250 steps for highquality samples.

Original: Although recent works propose dedicated high-order solvers and achieve a further speedup for sampling without guidance, their effectiveness for guided sampling has not been well-tested before.
Revised: Although recent works propose dedicated high-order solvers and achieve a further speedup for sampling without guidance, their effectiveness for guided sampling has not been well-tested before.

Original: In this work, we demonstrate that previous high-order fast samplers suffer from instability issues, and they even become slower than DDIM when the guidance scale grows large.
Revised: In this work, we demonstrate that previous high-order fast samplers suffer from instability issues, and they even become slower than DDIM when the guidance scale grows large.

Original: To further speed up guided sampling, we propose DPM-Solver++ , a high-order solver for the guided sampling of DPMs.
Revised: To further speed up guided sampling, we propose DPM-Solver++ , a high-order solver for the guided sampling of DPMs.

Original: DPM-Solver++ solves the diffusion ODE with the data prediction model and adopts thresholding methods to keep the solution matches training data distribution.
Revised: DPM-Solver++ solves the diffusion ODE with the data prediction model and adopts thresholding methods to keep the solution matches training data distribution.

Original: We further propose a multistep variant of DPM-Solver++ to address the instability issue by reducing the effective step size.
Revised: We further propose a multistep variant of DPM-Solver++ to address the instability issue by reducing the effective step size.

Original: Experiments show that DPM-Solver++ can generate high-quality samples within only 15 tosteps for guided sampling by pixel-space and latent-space DPMs.
Revised: Experiments show that DPM-Solver++ can generate high-quality samples within only 15 tosteps for guided sampling by pixel-space and latent-space DPMs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Diffusion probabilistic models (DPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021b) have achieved impressive success on various tasks, such as high-resolution image synthesis (Dhariwal & Nichol, 2021;
Revised: Diffusion probabilistic models (DPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021b) have achieved impressive success on various tasks, such as high-resolution image synthesis (Dhariwal & Nichol, 2021;

Original: Ho et al., 2022; Rombach et al., 2022), image editing (Meng et al., 2022; Saharia et al., 2022a; Zhao et al., 2022), text-to-image generation (Nichol et al., 2021; Saharia et al., 2022b; Ramesh et al., 2022; Rombach et al., 2022; Gu et al., 2022), voice synthesis (Liu et al., 2022a; Chen et al., 2021a;b), molecule generation (Xu et al., 2022; Hoogeboom et al., 2022; Wu et al., 2022) and data compression (Theis et al., 2022; Kingma et al., 2021).
Revised: Ho et al., 2022; Rombach et al., 2022), image editing (Meng et al., 2022; Saharia et al., 2022a; Zhao et al., 2022), text-to-image generation (Nichol et al., 2021; Saharia et al., 2022b; Ramesh et al., 2022; Rombach et al., 2022; Gu et al., 2022), voice synthesis (Liu et al., 2022a; Chen et al., 2021a;b), molecule generation (Xu et al., 2022; Hoogeboom et al., 2022; Wu et al., 2022) and data compression (Theis et al., 2022; Kingma et al., 2021).

Original: Compared with other deep generative models such as GANs (Goodfellow et al., 2014) and VAEs (Kingma & Welling, 2014),
Revised: Compared with other deep generative models such as GANs (Goodfellow et al., 2014) and VAEs (Kingma & Welling, 2014),

Original: DPMs can even achieve better sample quality by leveraging an essential technique called guided sampling (Dhariwal & Nichol, 2021; Ho & Salimans, 2021), which uses additional guidance models to improve the sample fidelity and the condition-sample alignment.
Revised: DPMs can even achieve better sample quality by leveraging an essential technique called guided sampling (Dhariwal & Nichol, 2021; Ho & Salimans, 2021), which uses additional guidance models to improve the sample fidelity and the condition-sample alignment.

Original: Through it, DPMs in text-to-image and image-to-image tasks can generate high-resolution photorealistic and artistic images which are highly correlated to the given condition, bringing a new trend in artificial intelligence art painting.
Revised: Through it, DPMs in text-to-image and image-to-image tasks can generate high-resolution photorealistic and artistic images which are highly correlated to the given condition, bringing a new trend in artificial intelligence art painting.

Original: The sampling procedure of DPMs gradually removes the noise from pure Gaussian random variables to obtain clear data, which can be viewed as discretizing either the diffusion SDEs (Ho et al., 2020;
Revised: The sampling procedure of DPMs gradually removes the noise from pure Gaussian random variables to obtain clear data, which can be viewed as discretizing either the diffusion SDEs (Ho et al., 2020;

Original: Song et al., 2021b) or the diffusion ODEs (Song et al., 2021b;a) defined by a parameterized noise prediction model or data prediction model (Ho et al., 2020; Kingma et al., 2021).
Revised: Song et al., 2021b) or the diffusion ODEs (Song et al., 2021b;a) defined by a parameterized noise prediction model or data prediction model (Ho et al., 2020; Kingma et al., 2021).

Original: Guided sampling of DPMs can also be formalized with such discretizations by combining an unconditional model with a guidance model, where a hyperparameter controls the scale of the guidance model (i.e. guidance scale ).
Revised: Guided sampling of DPMs can also be formalized with such discretizations by combining an unconditional model with a guidance model, where a hyperparameter controls the scale of the guidance model (i.e. guidance scale ).

Original: The commonly-used method for guided sampling is DDIM (Song et al., 2021a), which is proven as a first-order diffusion ODE solver (Salimans & Ho, 2022;
Revised: The commonly-used method for guided sampling is DDIM (Song et al., 2021a), which is proven as a first-order diffusion ODE solver (Salimans & Ho, 2022;

Original: Lu et al., 2022)
Revised: Lu et al., 2022)

Original: and it generally needs 100 to 250 steps of large neural network evaluations to converge, which is time-consuming.
Revised: and it generally needs 100 to 250 steps of large neural network evaluations to converge, which is time-consuming.

Original: Dedicated high-order diffusion ODE solvers (Lu et al., 2022; Zhang & Chen, 2022) can generate high-quality samples in 10 to 20 steps for sampling without guidance.
Revised: Dedicated high-order diffusion ODE solvers (Lu et al., 2022; Zhang & Chen, 2022) can generate high-quality samples in 10 to 20 steps for sampling without guidance.

Original: However, their effectiveness
Revised: However, their effectiveness

Original: [Figure]
Revised: [Figure]

Original: for guided sampling has not been carefully examined before.
Revised: for guided sampling has not been carefully examined before.

Original: In this work, we demonstrate that previous high-order solvers for DPMs generate unsatisfactory samples for guided sampling, even worse than the simple first-order solver DDIM.
Revised: In this work, we demonstrate that previous high-order solvers for DPMs generate unsatisfactory samples for guided sampling, even worse than the simple first-order solver DDIM.

Original: We identify two challenges of applying high-order solvers to guided sampling: (1) the large guidance scale narrows the convergence radius of high-order solvers, making them unstable; and (2) the converged solution does not fall into the same range with the original data (a.k.a.
Revised: We identify two challenges of applying high-order solvers to guided sampling: (1) the large guidance scale narrows the convergence radius of high-order solvers, making them unstable; and (2) the converged solution does not fall into the same range with the original data (a.k.a.

Original: “train-test mismatch” (Saharia et al., 2022b)).
Revised: “train-test mismatch” (Saharia et al., 2022b)).

Original: Based on the observations, we propose DPM-Solver++ , a training-free fast diffusion ODE solver for guided sampling.
Revised: Based on the observations, we propose DPM-Solver++ , a training-free fast diffusion ODE solver for guided sampling.

Original: We find that the parameterization of the DPM critically impacts the solution quality.
Revised: We find that the parameterization of the DPM critically impacts the solution quality.

Original: Subsequently, we solve the diffusion ODE defined by the data prediction model, which predicts the clean data given the noisy ones.
Revised: Subsequently, we solve the diffusion ODE defined by the data prediction model, which predicts the clean data given the noisy ones.

Original: We derive a high-order solver for solving the ODE with the data prediction parameterization, and adopt dynamic thresholding methods (Saharia et al., 2022b) to mitigate the train-test mismatch problem.
Revised: We derive a high-order solver for solving the ODE with the data prediction parameterization, and adopt dynamic thresholding methods (Saharia et al., 2022b) to mitigate the train-test mismatch problem.

Original: Furthermore, we develop a multistep solver which uses smaller step sizes to address instability.
Revised: Furthermore, we develop a multistep solver which uses smaller step sizes to address instability.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, DPM-Solver++ can generate high-quality samples in only 15 steps, which is much faster than all the previous training-free samplers for guided sampling.
Revised: 1, DPM-Solver++ can generate high-quality samples in only 15 steps, which is much faster than all the previous training-free samplers for guided sampling.

Original: Our additional experimental results show that DPM-Solver++ can generate high-fidelity samples and almost converge within only 15 to 20 steps, for a wide variety of guided sampling applications, including both pixel-space DPMs and latent-space DPMs.
Revised: Our additional experimental results show that DPM-Solver++ can generate high-fidelity samples and almost converge within only 15 to 20 steps, for a wide variety of guided sampling applications, including both pixel-space DPMs and latent-space DPMs.

Original:  D IFFUSION P ROBABILISTIC M ODELS
Revised:  D IFFUSION P ROBABILISTIC M ODELS

Original: In this section, we review diffusion probabilistic models (DPMs) and their sampling methods.
Revised: In this section, we review diffusion probabilistic models (DPMs) and their sampling methods.

Original: 2. F AST S AMPLING FOR DPM S BY D IFFUSION ODE S
Revised: 2. F AST S AMPLING FOR DPM S BY D IFFUSION ODE S

Original: Diffusion Probabilistic Models (DPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021b) gradually add Gaussian noise to a D -dimensional random variable x 0 ∈ R D to perturb the corresponding unknown data distribution q 0 ( x 0 ) at time 0 to a simple normal distribution q T ( x T )
Revised: Diffusion Probabilistic Models (DPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021b) gradually add Gaussian noise to a D -dimensional random variable x 0 ∈ R D to perturb the corresponding unknown data distribution q 0 ( x 0 ) at time 0 to a simple normal distribution q T ( x T )

Original: ≈ N ( x T | 0 , ˜
Revised: ≈ N ( x T | 0 , ˜

Original: σ 2 I ) at time T > 

Revised: σ 2 I ) at time T >

Original: for some ˜ σ > 0 .
Revised: for some ˜ σ > 0 .

Original: The transition distribution q t 0 ( x t | x 0 ) at each time t ∈ [0 , T ] satisfies
Revised: The transition distribution q t 0 ( x t | x 0 ) at each time t ∈ [0 , T ] satisfies

Original: [Equation]
Revised: [Equation]

Original: where α t , σ t > 0 and the signal-to-noise-ratio (SNR) α 2 t /σ 2 t is strictly decreasing w.r.t.
Revised: where α t , σ t > 0 and the signal-to-noise-ratio (SNR) α 2 t /σ 2 t is strictly decreasing w.r.t.

Original: t (Kingma et al., 2021).
Revised: t (Kingma et al., 2021).

Original: Eq.
Revised: Eq.

Original: (1) can be written as x t = α t x 0 + σ t ϵ , where ϵ ∼ N ( 0 , I ) .
Revised: (1) can be written as x t = α t x 0 + σ t ϵ , where ϵ ∼ N ( 0 , I ) .

Original: Parameterization: noise prediction and data prediction DPMs learn to recover the data xbased on the noisy input x T with a sequential denoising procedure.
Revised: Parameterization: noise prediction and data prediction DPMs learn to recover the data xbased on the noisy input x T with a sequential denoising procedure.

Original: There are two alternative ways to define the model.
Revised: There are two alternative ways to define the model.

Original: The noise prediction model ϵ θ ( x t , t ) attempts to predict the noise ϵ from the data x t , which optimizes the parameter θ by the following objective (Ho et al., 2020; Song et al., 2021b):
Revised: The noise prediction model ϵ θ ( x t , t ) attempts to predict the noise ϵ from the data x t , which optimizes the parameter θ by the following objective (Ho et al., 2020; Song et al., 2021b):

Original: [Equation]
Revised: [Equation]

Original: where x 0 ∼ q 0 ( x 0 ) , ϵ ∼ N ( 0 , I ) , t ∼ U ([0 , 1]) , and ω ( t ) > 0 is a weighting function.
Revised: where x 0 ∼ q 0 ( x 0 ) , ϵ ∼ N ( 0 , I ) , t ∼ U ([0 , 1]) , and ω ( t ) > 0 is a weighting function.

Original: Alternatively, the data prediction model x θ ( x t , t ) predicts the original data x 0 based on the noisy x t , and its relationship with ϵ θ ( x t , t ) is given by x θ ( x t , t ) :
Revised: Alternatively, the data prediction model x θ ( x t , t ) predicts the original data x 0 based on the noisy x t , and its relationship with ϵ θ ( x t , t ) is given by x θ ( x t , t ) :

Original: = ( x t − σ t ϵ θ ( x t , t )) /α
Revised: = ( x t − σ t ϵ θ ( x t , t )) /α

Original: t (Kingma et al., 2021).
Revised: t (Kingma et al., 2021).

Original: Diffusion ODEs Sampling by DPMs can be implemented by solving the diffusion ODEs (Song et al., 2021b;a;
Revised: Diffusion ODEs Sampling by DPMs can be implemented by solving the diffusion ODEs (Song et al., 2021b;a;

Original: Liu et al., 2022b; Zhang & Chen, 2022;
Revised: Liu et al., 2022b; Zhang & Chen, 2022;

Original: Lu et al., 2022), which is generally faster than other sampling methods.
Revised: Lu et al., 2022), which is generally faster than other sampling methods.

Original: Specifically, sampling by diffusion ODEs need to discretize the following
Revised: Specifically, sampling by diffusion ODEs need to discretize the following

Original: ODE (Song et al., 2021b) with t changing from T to 0 :
Revised: ODE (Song et al., 2021b) with t changing from T to 0 :

Original: [Equation]
Revised: [Equation]

Original: and the equivalent diffusion ODE w.r.t.
Revised: and the equivalent diffusion ODE w.r.t.

Original: the data prediction model x θ is
Revised: the data prediction model x θ is

Original: [Equation]
Revised: [Equation]

Original: where the coefficients f ( t ) = dlog
Revised: where the coefficients f ( t ) = dlog

Original: [Equation]
Revised: [Equation]

Original: t d
Revised: t d

Original: [Equation]
Revised: [Equation]

Original: 2. G UIDED S AMPLING FOR DPM S
Revised: 2. G UIDED S AMPLING FOR DPM S

Original: Guided sampling (Dhariwal & Nichol, 2021; Ho & Salimans, 2021) is a widely-used technique to apply DPMs for conditional sampling, which is useful in text-to-image, image-to-image, and class-to-image applications (Dhariwal & Nichol, 2021;
Revised: Guided sampling (Dhariwal & Nichol, 2021; Ho & Salimans, 2021) is a widely-used technique to apply DPMs for conditional sampling, which is useful in text-to-image, image-to-image, and class-to-image applications (Dhariwal & Nichol, 2021;

Original: Saharia et al., 2022b; Rombach et al., 2022;
Revised: Saharia et al., 2022b; Rombach et al., 2022;

Original: Nichol et al., 2021; Ramesh et al., 2022).
Revised: Nichol et al., 2021; Ramesh et al., 2022).

Original: Given a condition variable c , guided sampling defines a conditional noise prediction model ˜
Revised: Given a condition variable c , guided sampling defines a conditional noise prediction model ˜

Original: ϵ
Revised: ϵ

Original: θ ( x t , t, c ) .
Revised: θ ( x t , t, c ) .

Original: There are two types of guided sampling methods, depending on whether they require a classifier model.
Revised: There are two types of guided sampling methods, depending on whether they require a classifier model.

Original: Classifier guidance (Dhariwal & Nichol, 2021) leverages a pretrained classifier p ϕ
Revised: Classifier guidance (Dhariwal & Nichol, 2021) leverages a pretrained classifier p ϕ

Original: ( c | x t , t ) to define the conditional noise prediction model by:
Revised: ( c | x t , t ) to define the conditional noise prediction model by:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where s > 0 is the guidance scale .
Revised: where s > 0 is the guidance scale .

Original: In practice, a large s is usually preferred for improving the condition-sample alignment (Rombach et al., 2022; Saharia et al., 2022b) for guided sampling.
Revised: In practice, a large s is usually preferred for improving the condition-sample alignment (Rombach et al., 2022; Saharia et al., 2022b) for guided sampling.

Original: Classifier-free guidance (Ho & Salimans, 2021) shares the same parameterized model
Revised: Classifier-free guidance (Ho & Salimans, 2021) shares the same parameterized model

Original: ϵ θ ( x t , t, c ) for the unconditional and conditional noise prediction models, where the input c for the unconditional model is a special placeholder ∅ .
Revised: ϵ θ ( x t , t, c ) for the unconditional and conditional noise prediction models, where the input c for the unconditional model is a special placeholder ∅ .

Original: The corresponding conditional model is defined by:
Revised: The corresponding conditional model is defined by:

Original: [Equation]
Revised: [Equation]

Original: Then, samples can be drawn by solving the ODE (3) with ϵ θ ( x t , t, c ) in place of ϵ θ ( x t , t ) .
Revised: Then, samples can be drawn by solving the ODE (3) with ϵ θ ( x t , t, c ) in place of ϵ θ ( x t , t ) .

Original: DDIM (Song et al., 2021a) is a typical solver for guided sampling, which generates samples in a few hundreds of steps.
Revised: DDIM (Song et al., 2021a) is a typical solver for guided sampling, which generates samples in a few hundreds of steps.

Original: 2. E XPONENTIAL I NTEGRATORS AND H IGH -O RDER ODE S OLVERS
Revised: 2. E XPONENTIAL I NTEGRATORS AND H IGH -O RDER ODE S OLVERS

Original: It is shown in recent works (Lu et al., 2022; Zhang & Chen, 2022) that ODE solvers based on exponential integrators (Hochbruck & Ostermann, 2010) converge much faster than the traditional solvers for solving the unconditional diffusion ODE (3).
Revised: It is shown in recent works (Lu et al., 2022; Zhang & Chen, 2022) that ODE solvers based on exponential integrators (Hochbruck & Ostermann, 2010) converge much faster than the traditional solvers for solving the unconditional diffusion ODE (3).

Original: Given an initial value x s at time s > 0 , Lu et al.
Revised: Given an initial value x s at time s > 0 , Lu et al.

Original: (2022) derive the solution x t of the diffusion ODE (3) at time t as:
Revised: (2022) derive the solution x t of the diffusion ODE (3) at time t as:

Original: [Equation]
Revised: [Equation]

Original: where the ODE is changed from the time ( t ) domain to the log-SNR ( λ ) domain by the change-ofvariables formula.
Revised: where the ODE is changed from the time ( t ) domain to the log-SNR ( λ ) domain by the change-ofvariables formula.

Original: Here, the log-SNR λ t := log( α t /σ t ) is a strictly decreasing function of t with the inverse function t λ ( · ) , and ˆ x λ := x t λ ( λ ) , ˆ ϵ θ (ˆ x λ , λ ) :
Revised: Here, the log-SNR λ t := log( α t /σ t ) is a strictly decreasing function of t with the inverse function t λ ( · ) , and ˆ x λ := x t λ ( λ ) , ˆ ϵ θ (ˆ x λ , λ ) :

Original: = ϵ θ ( x t λ ( λ ) , t λ ( λ )) are the corresponding change-of-variable forms for λ .
Revised: = ϵ θ ( x t λ ( λ ) , t λ ( λ )) are the corresponding change-of-variable forms for λ .

Original: Lu et al.
Revised: Lu et al.

Original: (2022) showed that DDIM is a first-order solver for Eq.
Revised: (2022) showed that DDIM is a first-order solver for Eq.

Original: They further proposed a high-order solver named “DPM-Solver”, which can generate realistic samples for the unconditional model in only 10-20 steps.
Revised: They further proposed a high-order solver named “DPM-Solver”, which can generate realistic samples for the unconditional model in only 10-20 steps.

Original: Unfortunately, the outstanding efficiency of existing high-order solvers does not transfer to guided sampling, which we shall discuss soon.
Revised: Unfortunately, the outstanding efficiency of existing high-order solvers does not transfer to guided sampling, which we shall discuss soon.

Original:  C HALLENGES OF H IGH -O RDER S OLVERS FOR G UIDED S AMPLING
Revised:  C HALLENGES OF H IGH -O RDER S OLVERS FOR G UIDED S AMPLING

Original: Before developing new fast solvers, we first examine the performance of existing high-order diffusion
Revised: Before developing new fast solvers, we first examine the performance of existing high-order diffusion

Original: ODE solvers and highlight the challenges.
Revised: ODE solvers and highlight the challenges.

Original: The first challenge is the large guidance scale causes high-order solvers to be instable .
Revised: The first challenge is the large guidance scale causes high-order solvers to be instable .

Original: As shown in
Revised: As shown in

Original: Fig.
Revised: Fig.

Original: 1, for a large guidance scale s = 8 . 0 and 15 function evaluations, previous high-order diffusion
Revised: 1, for a large guidance scale s = 8 . 0 and 15 function evaluations, previous high-order diffusion

Original: ODE solvers (Lu et al., 2022; Zhang & Chen, 2022; Liu et al., 2022b) produce low-quality images.
Revised: ODE solvers (Lu et al., 2022; Zhang & Chen, 2022; Liu et al., 2022b) produce low-quality images.

Original: Their sample quality is even worse than the first-order DDIM.
Revised: Their sample quality is even worse than the first-order DDIM.

Original: Moreover, the sample quality becomes even worse as the order of the solver gets higher.
Revised: Moreover, the sample quality becomes even worse as the order of the solver gets higher.

Original: Intuitively, large guidance scales may amplify both the output and the derivatives of the model ˜
Revised: Intuitively, large guidance scales may amplify both the output and the derivatives of the model ˜

Original: ϵ θ in Eq.
Revised: ϵ θ in Eq.

Original: The derivatives of the model affect the convergence range of ODE solvers, and the amplification may cause high-order ODE solvers to need much smaller step sizes to converge, and thus the higher-order solvers may perform worse than the first-order solver.
Revised: The derivatives of the model affect the convergence range of ODE solvers, and the amplification may cause high-order ODE solvers to need much smaller step sizes to converge, and thus the higher-order solvers may perform worse than the first-order solver.

Original: Moreover, high-order solvers require high-order derivatives, which are generally more sensitive to the amplifications.
Revised: Moreover, high-order solvers require high-order derivatives, which are generally more sensitive to the amplifications.

Original: This further narrows the convergence radius.
Revised: This further narrows the convergence radius.

Original: The second challenge is the “train-test mismatch” problem (Saharia et al., 2022b).
Revised: The second challenge is the “train-test mismatch” problem (Saharia et al., 2022b).

Original: The data lie in a bounded interval (e.g. [ − 1 , 1] for image data).
Revised: The data lie in a bounded interval (e.g. [ − 1 , 1] for image data).

Original: However, the large guidance scale pushes the conditional noise prediction model ˜
Revised: However, the large guidance scale pushes the conditional noise prediction model ˜

Original: ϵ
Revised: ϵ

Original: θ ( x t , t, c ) away from the true noise, which in turns make the sample (i.e. the converged solution x 0 of diffusion ODEs) to fall out of the bound.
Revised: θ ( x t , t, c ) away from the true noise, which in turns make the sample (i.e. the converged solution x 0 of diffusion ODEs) to fall out of the bound.

Original: In this case, the generated images are saturated and unnatural (Saharia et al., 2022b).
Revised: In this case, the generated images are saturated and unnatural (Saharia et al., 2022b).

Original:  D ESIGNING T RAINING -F REE F AST S AMPLERS FOR G UIDED S AMPLING
Revised:  D ESIGNING T RAINING -F REE F AST S AMPLERS FOR G UIDED S AMPLING

Original: In this section, we design novel high-order diffusion ODE solvers for faster guided sampling.
Revised: In this section, we design novel high-order diffusion ODE solvers for faster guided sampling.

Original: As discussed in Sec. 3, previous high-order solvers have instability and “train-test mismatch” issues for large guidance scales.
Revised: As discussed in Sec. 3, previous high-order solvers have instability and “train-test mismatch” issues for large guidance scales.

Original: The “train-test mismatch” issue arises from the ODE itself, and we find the parameterization of the ODE is critical for the converged solution to be bounded.
Revised: The “train-test mismatch” issue arises from the ODE itself, and we find the parameterization of the ODE is critical for the converged solution to be bounded.

Original: While previous high-order solvers are designed for the noise prediction model ˜
Revised: While previous high-order solvers are designed for the noise prediction model ˜

Original: ϵ θ , we solve the ODE (4) for the data prediction model x θ , which itself has some advantages and thresholding methods are further available to keep the samples bounded (Ho et al., 2020; Saharia et al., 2022b).
Revised: ϵ θ , we solve the ODE (4) for the data prediction model x θ , which itself has some advantages and thresholding methods are further available to keep the samples bounded (Ho et al., 2020; Saharia et al., 2022b).

Original: We also propose a multistep solver to address the instability issue.
Revised: We also propose a multistep solver to address the instability issue.

Original: 4. D ESIGNING S OLVERS BY D ATA P REDICTION M ODEL
Revised: 4. D ESIGNING S OLVERS BY D ATA P REDICTION M ODEL

Original: We follow the notations in Lu et al.
Revised: We follow the notations in Lu et al.

Original: Given a sequence { t i } M i =

Revised: Given a sequence { t i } M i =

Original: decreasing from t 

Revised: decreasing from t

Original: = T to t M = 0 and an initial value x t 0 ∼ N ( 0 | ˜ σ 2 I ) , the solver aims to iteratively compute a sequence { ˜ x t i } Mi =0 to approximate the exact solution at each time t i , and the final value ˜ x
Revised: = T to t M = 0 and an initial value x t 0 ∼ N ( 0 | ˜ σ 2 I ) , the solver aims to iteratively compute a sequence { ˜ x t i } Mi =0 to approximate the exact solution at each time t i , and the final value ˜ x

Original: t M is the approximated sample by the diffusion ODE.
Revised: t M is the approximated sample by the diffusion ODE.

Original: Denote h
Revised: Denote h

Original: i :=
Revised: i :=

Original: λ t i − λ t
Revised: λ t i − λ t

Original: i − 1 for i = 1 , . . .
Revised: i − 1 for i = 1 , . . .

Original: , M .
Revised: , M .

Original: For solving the diffusion ODE w.r.t.
Revised: For solving the diffusion ODE w.r.t.

Original: x θ in Eq.
Revised: x θ in Eq.

Original: (4), we firstly propose a simplified formulation of the exact solution of diffusion ODEs w.r.t.
Revised: (4), we firstly propose a simplified formulation of the exact solution of diffusion ODEs w.r.t.

Original: x θ below.
Revised: x θ below.

Original: Such formulation exactly computes the linear term in Eq.
Revised: Such formulation exactly computes the linear term in Eq.

Original: (4) and only remains an exponentially-weighted integral of x θ .
Revised: (4) and only remains an exponentially-weighted integral of x θ .

Original: Denote ˆ x θ (ˆ x λ , λ ) :
Revised: Denote ˆ x θ (ˆ x λ , λ ) :

Original: = x θ ( x t λ ( λ ) , t λ ( λ )) as the change-of-variable form of x θ for λ , we have:
Revised: = x θ ( x t λ ( λ ) , t λ ( λ )) as the change-of-variable form of x θ for λ , we have:

Original: Proposition 4.1 (Exact solution of diffusion ODEs of x θ , proof in Appendix A) .
Revised: Proposition 4.1 (Exact solution of diffusion ODEs of x θ , proof in Appendix A) .

Original: Given an initial value x s at time s > 0 , the solution x t at time t ∈ [0 , s ] of diffusion ODEs in Eq.
Revised: Given an initial value x s at time s > 0 , the solution x t at time t ∈ [0 , s ] of diffusion ODEs in Eq.

Original: (4) is:
Revised: (4) is:

Original: [Equation]
Revised: [Equation]

Original: As the diffusion ODEs in Eq.
Revised: As the diffusion ODEs in Eq.

Original: (3) and Eq.(4) are equivalent, the exact solution formulations in Eq.
Revised: (3) and Eq.(4) are equivalent, the exact solution formulations in Eq.

Original: (7) and Eq.
Revised: (7) and Eq.

Original: (8) are also equivalent.
Revised: (8) are also equivalent.

Original: However, from the prespective of designing ODE solvers, these two formulations are different.
Revised: However, from the prespective of designing ODE solvers, these two formulations are different.

Original: Firstly, Eq.
Revised: Firstly, Eq.

Original: (7) exactly computes the linear term α t α s x s , while Eq.
Revised: (7) exactly computes the linear term α t α s x s , while Eq.

Original: (8) exactly computes another the linear term σ t σ s x s .
Revised: (8) exactly computes another the linear term σ t σ s x s .

Original: Moreover, to design ODE solvers, Eq.
Revised: Moreover, to design ODE solvers, Eq.

Original: (7) needs to approximate the integral (cid:82) e − λ
Revised: (7) needs to approximate the integral (cid:82) e − λ

Original: ϵ θ d λ , while Eq.
Revised: ϵ θ d λ , while Eq.

Original: (8) needs to approximate (cid:82)
Revised: (8) needs to approximate (cid:82)

Original: e
Revised: e

Original: λ x θ d λ , and these two integrals are different (recall that x θ :=
Revised: λ x θ d λ , and these two integrals are different (recall that x θ :=

Original: ( x t − σ t ϵ θ ) /α t ).
Revised: ( x t − σ t ϵ θ ) /α t ).

Original: Therefore, the high-order solvers based on Eq.
Revised: Therefore, the high-order solvers based on Eq.

Original: (7) and Eq.
Revised: (7) and Eq.

Original: (8) are essentially different.
Revised: (8) are essentially different.

Original: We further propose the general manner for design high-order ODE solvers based on Eq.
Revised: We further propose the general manner for design high-order ODE solvers based on Eq.

Original: (8) below.
Revised: (8) below.

Original: Given the previous value ˜ x t i − 1 at time t i − 1 , the aim of our solver is to approximate the exact solution at time t i .
Revised: Given the previous value ˜ x t i − 1 at time t i − 1 , the aim of our solver is to approximate the exact solution at time t i .

Original: Denote x ( n ) θ ( λ ) :
Revised: Denote x ( n ) θ ( λ ) :

Original: =
Revised: =

Original: d n ˆ x θ ( x λ ,λ ) d λ n as the n -th order total derivatives of x θ w.r.t.
Revised: d n ˆ x θ ( x λ ,λ ) d λ n as the n -th order total derivatives of x θ w.r.t.

Original: logSNR λ .
Revised: logSNR λ .

Original: For k ≥ 1 , taking the ( k − 1) -th
Revised: For k ≥ 1 , taking the ( k − 1) -th

Original: Taylor expansion at λ t i − 1 for x θ w.r.t.
Revised: Taylor expansion at λ t i − 1 for x θ w.r.t.

Original: λ ∈ [ λ t i − 1 , λ t i ] and substitute it into Eq.
Revised: λ ∈ [ λ t i − 1 , λ t i ] and substitute it into Eq.

Original: (8) with s =
Revised: (8) with s =

Original: t
Revised: t

Original: i − 1 and t =
Revised: i − 1 and t =

Original: t i , we have
Revised: t i , we have

Original: [Equation]
Revised: [Equation]

Original: analytically computed (Appendix A)
Revised: analytically computed (Appendix A

Original: [Equation]
Revised: [Equation]

Original: where the integral (cid:82) e λ ( λ − λ ti − 1 ) n n !
Revised: where the integral (cid:82) e λ ( λ − λ ti − 1 ) n n !

Original: d λ can be analytically computed by integral-by-parts (detailed in
Revised: d λ can be analytically computed by integral-by-parts (detailed in

Original: Appendix A).
Revised: Appendix A).

Original: Therefore, to design a k -th order ODE solver, we only need to estimate the n -th order derivatives x ( n ) θ ( λ t i − 1 ) for n ≤ k − 1 after omitting the O (
Revised: Therefore, to design a k -th order ODE solver, we only need to estimate the n -th order derivatives x ( n ) θ ( λ t i − 1 ) for n ≤ k − 1 after omitting the O (

Original: h k +1 i ) high-order error terms, which are well-studied techniques
Revised: h k +1 i ) high-order error terms, which are well-studied techniques

Original: and we discussed in details in Sec.
Revised: and we discussed in details in Sec.

Original: A special case is k = 1 , where the solver is the same as DDIM (Song et al., 2021a), and we discuss in Sec. 5.1.
Revised: A special case is k = 1 , where the solver is the same as DDIM (Song et al., 2021a), and we discuss in Sec. 5.1.

Original: For k = 2 , we use a similar technique as DPM-Solver-

Revised: For k = 2 , we use a similar technique as DPM-Solver-

Original: (Lu et al., 2022) to estimate the derivative x (1)
Revised: (Lu et al., 2022) to estimate the derivative x (1)

Original: θ
Revised: θ

Original: (ˆ x λ ti − 1 , λ t i − 1 ) .
Revised: (ˆ x λ ti − 1 , λ t i − 1 ) .

Original: Specifically, we introduce an additional intermediate time step s i between t
Revised: Specifically, we introduce an additional intermediate time step s i between t

Original: i −and
Revised: i −and

Original: t i and combine the function values at s i and t i − 1 to approximate the derivative, which is the standard manner for singlestep ODE solvers (Atkinson et al., 2011).
Revised: t i and combine the function values at s i and t i − 1 to approximate the derivative, which is the standard manner for singlestep ODE solvers (Atkinson et al., 2011).

Original: Overall, we need 2 M + 1 time steps ( { t i } Mi =0 and { s i } Mi =1 ) which satisfies t 0 > s 1 > t 1 > · · · > t M − 1 > s M > t M .
Revised: Overall, we need 2 M + 1 time steps ( { t i } Mi =0 and { s i } Mi =1 ) which satisfies t 0 > s 1 > t 1 > · · · > t M − 1 > s M > t M .

Original: The detailed algorithm is proposed in Algorithm 1, where we combine the previous value ˜ x t i − 1 at time t i − 1 with the intermediate value
Revised: The detailed algorithm is proposed in Algorithm 1, where we combine the previous value ˜ x t i − 1 at time t i − 1 with the intermediate value

Original: u i at time s i to compute the value ˜ x t i at time t i .
Revised: u i at time s i to compute the value ˜ x t i at time t i .

Original: [Table]
Revised: [Table]

Original: − α t 1 (cid:0) e − h 1 − 1 (cid:1) x θ (˜ x t 0 ,t 0 ) 5: Q buffer ←−−− x θ (˜ x t 1 ,t 1 ) 6: for i ← 2 to M do 7: r i ← h
Revised: − α t 1 (cid:0) e − h 1 − 1 (cid:1) x θ (˜ x t 0 ,t 0 ) 5: Q buffer ←−−− x θ (˜ x t 1 ,t 1 ) 6: for i ← 2 to M do 7: r i ← h

Original: i − 1 h i 8:
Revised: i − 1 h i 8:

Original: D i ← (cid:16) 1 + 12 r
Revised: D i ← (cid:16) 1 + 12 r

Original: i (cid:17) x θ (˜ x t i − 1 ,t i − 1 ) − 12 r i x θ (˜ x t i − 2 ,t i − 2 ) 9: ˜ x t i ← σ ti σ ti − 1 ˜ x t i − 1 − α t
Revised: i (cid:17) x θ (˜ x t i − 1 ,t i − 1 ) − 12 r i x θ (˜ x t i − 2 ,t i − 2 ) 9: ˜ x t i ← σ t i σ ti − 1 ˜ x t i − 1 − α t

Original: i (cid:0) e − h
Revised: i (cid:0) e − h

Original: i − 1 (cid:1)
Revised: i − 1 (cid:1)

Original: D i 10:
Revised: D i 10:

Original: If i < M , then Q buffer ←−−− x θ (˜ x t i ,t i ) 11: end for 12: return ˜ x t M
Revised: If i < M , then Q buffer ←−−− x θ (˜ x t i ,t i ) 11: end for 12: return ˜ x t M

Original: We name the algorithm as DPM-Solver++(2S), which means that the proposed solver is a second-order singlestep method.
Revised: We name the algorithm as DPM-Solver++(2S), which means that the proposed solver is a second-order singlestep method.

Original: We present the theoretical guarantee of the convergence order in Appendix A. Fork ≥ 3 , as discussed in Sec. 3, high-order solvers may be unsuitable for large guidance scales, thus we mainly consider k = 2 in this work, and leave the solvers for higher orders for future study.
Revised: We present the theoretical guarantee of the convergence order in Appendix A. For

Original: [Table]
Revised: [Table]

Revised: k ≥ 3 , as discussed in Sec. 3, high-order solvers may be unsuitable for large guidance scales, thus we mainly consider k = 2 in this work, and leave the solvers for higher orders for future study.

Revised: Moreover, we provide a theoretical comparison between DPM-Solver-2 (Lu et al., 2022) and DPMSolver++(2S) in Appendix B. We find that DPM-Solver++(2S) has a smaller constant before the high-order error terms, thus generally has a smaller discretization error than DPM-Solver-2.

Original: 4. F ROM S INGLESTEP TO M ULTISTEP
Revised: 4. F ROM S INGLESTEP TO M ULTISTEP

Original: At each step (from t i − 1 to t i ) [Table] the proposed singlestep solver needs two sequential function evaluations of the neural network x θ .
Revised: At each step (from t i − 1 to t i ), the proposed singlestep solver needs two sequential function evaluations of the neural network x θ .

Original: Moreover, the intermediate values u i are only used once and then discarded.
Revised: Moreover, the intermediate values u i are only used once and then discarded.

Original: Such method loses the previous information and may be inefficient.
Revised: Such method loses the previous information and may be inefficient.

Original: In this section [Table] we propose another second-order diffusion ODE solver which uses the previous information at each step.
Revised: In this section, we propose another second-order diffusion ODE solver which uses the previous information at each step.

Original: In general, to approximate the derivatives x ( n ) θ in Eq.
Revised: In general, to approximate the derivatives x ( n ) θ in Eq.

Original: (9) for n ≥ [Table] there is another mainstream approach (Atkinson et al., 2011): multistep methods (such as Adams–Bashforth methods).
Revised: (9) for n ≥ 1 , there is another mainstream approach (Atkinson et al., 2011): multistep methods (such as Adams–Bashforth methods).

Original: Given the previous values { ˜ x t j } i − 1 j =0 at time t i − 1 , multistep methods just reuse the previous values to approximate the high-order derivatives.
Revised: Given the previous values { ˜ x t j } i − 1 j =0 at time t i − 1 , multistep methods just reuse the previous values to approximate the high-order derivatives.

Original: Multistep methods are empirically more efficient than singlestep methods, especially for limited number of function evaluations.
Revised: Multistep methods are empirically more efficient than singlestep methods, especially for limited number of function evaluations.

Original: (Atkinson et al., 2011)
Revised: (Atkinson et al., 2011)

Original: We combine the techniques for designing multistep solvers with the Taylor expansions in Eq.
Revised: We combine the techniques for designing multistep solvers with the Taylor expansions in Eq.

Original: (9) and further propose a multistep second-order solver for diffusion ODEs with x θ .
Revised: (9) and further propose a multistep second-order solver for diffusion ODEs with x θ .

Original: The detailed algorithm is proposed in Algorithm 2, where we combine the previous values ˜ x t i − 1 and ˜ x t i − 2 to compute the value ˜ x t i without additional intermediate values.
Revised: The detailed algorithm is proposed in Algorithm 2, where we combine the previous values ˜ x t i − 1 and ˜ x t i − 2 to compute the value ˜ x t i without additional intermediate values.

Original: We name the algorithm as DPM-Solver++(2M), which means that the proposed solver is a second-order multistep solver.
Revised: We name the algorithm as DPM-Solver++(2M), which means that the proposed solver is a second-order multistep solver.

Original: We also present a detailed theoretical guarantee of the convergence order, which is stated in Appendix A.
Revised: We also present a detailed theoretical guarantee of the convergence order, which is stated in Appendix A.

Original: For a fixed budget N of the total number of function evaluations, multistep methods can use M = N steps, while k -th order singlestep methods can only use no more than M = N/k steps.
Revised: For a fixed budget N of the total number of function evaluations, multistep methods can use M = N steps, while k -th order singlestep methods can only use no more than M = N/k steps.

Original: Therefore, each step size h i of multistep methods is around 1 /k of that of singlestep methods, so the high-order error terms O ( h ki ) in Eq.
Revised: Therefore, each step size h i of multistep methods is around 1 /k of that of singlestep methods, so the high-order error terms O ( h k i ) in Eq.

Original: (9) of multistep methods may also be smaller than those of singlestep methods.
Revised: (9) of multistep methods may also be smaller than those of singlestep methods.

Original: We show in Sec.
Revised: We show in Sec.

Original: 6.1 that the multistep methods are slightly better than singlestep methods.
Revised: 6.1 that the multistep methods are slightly better than singlestep methods.

Original: 4. C OMBINING T HRESHOLDING WITH DPM-S OLVER ++
Revised: 4. C OMBINING T HRESHOLDING WITH DPM-S OLVER ++

Original: For distributions of bounded data (such as the image data), thresholding methods (Ho et al., 2020;
Revised: For distributions of bounded data (such as the image data), thresholding methods (Ho et al., 2020;

Original: Saharia et al., 2022b) can push out-of-bound samples inwards and somehow reduce the adverse impact of the large guidance scale.
Revised: Saharia et al., 2022b) can push out-of-bound samples inwards and somehow reduce the adverse impact of the large guidance scale.

Original: Specifically, thresholding methods define a clipped data prediction model ˆ x θ ( x t , t, c ) by elementwise clipping the original model x θ :=
Revised: Specifically, thresholding methods define a clipped data prediction model ˆ x θ ( x t , t, c ) by elementwise clipping the original model x θ :=

Original: ( x t − σ t ϵ θ ) /α
Revised: ( x t − σ t ϵ θ ) /α

Original: t within the data bound, which results in better sample quality for large guidance scales (Saharia et al., 2022b).
Revised: t within the data bound, which results in better sample quality for large guidance scales (Saharia et al., 2022b).

Original: As our proposed DPM-Solver++ is designed for the x θ model, we can straightforwardly combine thresholding methods with DPM-Solver++ to accelerate the sampling speed of diffusion ODEs with the clipped model. To the best of our knowledge, DPM-Solver++ is the first high-order diffusion ODE solver which can directly use the thresholding methods.
Revised: As our proposed DPM-Solver++ is designed for the x θ model, we can straightforwardly combine thresholding methods with DPM-Solver++.

Original:  R ELATIONSHIP WITH O THER F AST S AMPLING M ETHODS
Revised:  R ELATIONSHIP WITH O THER F AST S AMPLING M ETHODS

Original: In essence, all training-free sampling methods for DPMs can be understood as either discretizing diffusion SDEs (Ho et al., 2020; Song et al., 2021b;
Revised: In essence, all training-free sampling methods for DPMs can be understood as either discretizing diffusion SDEs (Ho et al., 2020; Song et al., 2021b;

Original: Jolicoeur-Martineau et al., 2021; Tachibana et al., 2021; Kong & Ping, 2021; Bao et al., 2022b; Zhang et al., 2022) or discretizing diffusion ODEs (Song et al., 2021b;a; Liu et al., 2022b; Zhang & Chen, 2022;
Revised: Jolicoeur-Martineau et al., 2021; Tachibana et al., 2021; Kong & Ping, 2021; Bao et al., 2022b; Zhang et al., 2022) or discretizing diffusion ODEs (Song et al., 2021b;a; Liu et al., 2022b; Zhang & Chen, 2022;

Original: Lu et al., 2022).
Revised: Lu et al., 2022).

Original: As DPM-Solver++ is designed for solving diffusion ODEs, in this section, we discuss the relationship between DPM-Solver++ and other diffusion ODE solvers.
Revised: As DPM-Solver++ is designed for solving diffusion ODEs, in this section, we discuss the relationship between DPM-Solver++ and other diffusion ODE solvers.

Original: We further briefly discuss other fast sampling methods for DPMs.
Revised: We further briefly discuss other fast sampling methods for DPMs.

Original: 5. C OMPARISON WITH S OLVERS BASED ON E XPONENTIAL I NTEGRATORS
Revised: 5. C OMPARISON WITH S OLVERS BASED ON E XPONENTIAL I NTEGRATORS

Original: Previous state-of-the-art fast diffusion ODE solvers (Lu et al., 2022; Zhang & Chen, 2022) leverages exponential integrators to solve diffusion ODEs with noise prediction models ϵ θ .
Revised: Previous state-of-the-art fast diffusion ODE solvers (Lu et al., 2022; Zhang & Chen, 2022) leverages exponential integrators to solve diffusion ODEs with noise prediction models ϵ θ .

Original: In short, these solvers approximate the exact solution in Eq.
Revised: In short, these solvers approximate the exact solution in Eq.

Original: (7) and include DDIM (Song et al., 2021a)
Revised: (7) and include DDIM (Song et al., 2021a)

Original: as the first-order case.
Revised: as the first-order case.

Original: Below we show that the first-order case for DPM-Solver++ is also DDIM.
Revised: Below we show that the first-order case for DPM-Solver++ is also DDIM.

Original: For k = 1 , Eq.
Revised: For k = 1 , Eq.

Original: (9) becomes (after omitting the O ( h k +1 i ) terms)
Revised: (9) becomes (after omitting the O ( h k +1 i ) terms)

Original: [Equation]
Revised: [Equation]

Original: Therefore, our proposed DPM-Solver++ is the high-order generalization of DDIM w.r.t.
Revised: Therefore, our proposed DPM-Solver++ is the high-order generalization of DDIM w.r.t.

Original: the data prediction model x θ .
Revised: the data prediction model x θ .

Original: To the best of our knowledge, such generalization has not been proposed before.
Revised: To the best of our knowledge, such generalization has not been proposed before.

Original: We list the detailed difference between previous high-order solvers based on exponential integrators and DPM-Solver++ in Table 1.
Revised: We list the detailed difference between previous high-order solvers based on exponential integrators and DPM-Solver++ in Table 1.

Original: We emphasize that although the first-order version of these solvers are equivalent, the high-order versions of these solvers are rather different.
Revised: We emphasize that although the first-order version of these solvers are equivalent, the high-order versions of these solvers are rather different.

Original: Moreover, previous solvers are designed for the noise prediction model ˜

Original: ϵ θ , which cannot directly apply the thresholding method.

Original: 5. O THER F AST S AMPLING M ETHODS
Revised: 5. O THER F AST S AMPLING M ETHODS

Original: Samplers based on diffusion SDEs (Ho et al., 2020; Song et al., 2021b; Jolicoeur-Martineau et al., 2021; Tachibana et al., 2021; Kong & Ping, 2021; Bao et al., 2022b; Zhang et al., 2022) generally needs more steps to converge than those based on diffusion ODEs (Lu et al., 2022), because SDEs introduce more randomness and make denoising more difficult.
Revised: Samplers based on diffusion SDEs (Ho et al., 2020; Song et al., 2021b; Jolicoeur-Martineau et al., 2021; Tachibana et al., 2021; Kong & Ping, 2021; Bao et al., 2022b; Zhang et al., 2022) generally needs more steps to converge than those based on diffusion ODEs (Lu et al., 2022), because SDEs introduce more randomness and make denoising more difficult.

Original: Samplers based on extra training include model distillation (Salimans & Ho, 2022; Luhman & Luhman, 2021), learning reverse process variances (San-Roman et al., 2021; Nichol & Dhariwal, 2021; Bao et al., 2022a), and learning sampling steps (Lam et al., 2021; Watson et al., 2022).
Revised: Samplers based on extra training include model distillation (Salimans & Ho, 2022; Luhman & Luhman, 2021), learning reverse process variances (San-Roman et al., 2021; Nichol & Dhariwal, 2021; Bao et al., 2022a), and learning sampling steps (Lam et al., 2021; Watson et al., 2022).

Original: However, training-based samplers are hard to scale-up to pre-trained large DPMs (Saharia et al., 2022b; Rombach et al., 2022; Ramesh et al., 2022).
Revised: However, training-based samplers are hard to scale-up to pre-trained large DPMs (Saharia et al., 2022b; Rombach et al., 2022; Ramesh et al., 2022).

Original: There are other fast sampling methods by modifying the original DPMs to a latent space (Vahdat et al., 2021) or with momentum (Dockhorn et al., 2022).
Revised: There are other fast sampling methods by modifying the original DPMs to a latent space (Vahdat et al., 2021) or with momentum (Dockhorn et al., 2022).

Original: In addition, combining DPMs with GANs (Xiao et al., 2022; Wang et al., 2022) improves the sample quality of GANs and sampling speed of DPMs.
Revised: In addition, combining DPMs with GANs (Xiao et al., 2022; Wang et al., 2022) improves the sample quality of GANs and sampling speed of DPMs.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we show that DPM-Solver++ can speed up both the pixel-space DPMs and the latent-space DPMs for guided sampling.
Revised: In this section, we show that DPM-Solver++ can speed up both the pixel-space DPMs and the latent-space DPMs for guided sampling.

Original: We vary different number of function evaluations (NFE) which is the numebr of calls to the model ϵ θ ( x t , t, c ) or x θ ( x t , t, c ) , and compare DPM-Solver++ with the previous state-of-the-art fast samplers for DPMs including DPM-Solver (Lu et al., 2022),
Revised: We vary different number of function evaluations (NFE) which is the numebr of calls to the model ϵ θ ( x t , t, c ) or x θ ( x t , t, c ) , and compare DPM-Solver++ with the previous state-of-the-art fast samplers for DPMs including DPM-Solver (Lu et al., 2022),

Original: DEIS (Zhang & Chen, 2022), PNDM (Liu et al., 2022b) and DDIM (Song et al., 2021a).
Revised: DEIS (Zhang & Chen, 2022), PNDM (Liu et al., 2022b) and DDIM (Song et al., 2021a).

Original: We also convert the discrete-time DPMs to the continuous-time and use these continuous-time solvers.
Revised: We also convert the discrete-time DPMs to the continuous-time and use these continuous-time solvers.

Original: We refer to Appendix C for the detailed implementations and experiment settings.
Revised: We refer to Appendix C for the detailed implementations and experiment settings.

Original: As previous solvers did not test the performance in guided sampling, we also carefully tune the baseline samplers by ablating the step size schedule (i.e. the choice for the time steps { t i } Mi =0 ) and the solver order.
Revised: As previous solvers did not test the performance in guided sampling, we also carefully tune the baseline samplers by ablating the step size schedule (i.e. the choice for the time steps { t i } Mi =0 ) and the solver order.

Original: We find that
Revised: We find that

Original: • For the step size schedule, we search the time steps in the following choices: uniform t (a widely-used setting in high-resolution image synthesis), uniform λ (used in (Lu et al., 2022)), uniform split of the power functions of t (used in (Zhang & Chen, 2022), detailed in Appendix C), and we find that the best choice is uniform t .
Revised: • For the step size schedule, we search the time steps in the following choices: uniform t (a widely-used setting in high-resolution image synthesis), uniform λ (used in (Lu et al., 2022)), uniform split of the power functions of t (used in (Zhang & Chen, 2022), detailed in Appendix C), and we find that the best choice is uniform t .

Original: Thus, we use uniform t for the time steps in all of our experiments for all of the solvers.
Revised: Thus, we use uniform t for the time steps in all of our experiments for all of the solvers.

Original: • We find that for a large guidance scale, the best choice for all the previous solvers is the second-order (
Revised: • We find that for a large guidance scale, the best choice for all the previous solvers is the second-order (

Original: i
Revised: i

Original: e. DPM-Solver-2 and DEIS-1).
Revised: e. DPM-Solver-2 and DEIS-1).

Original: However, for a comprehensive comparison,
Revised: However, for a comprehensive comparison,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: we run all the orders of previous solvers, including DPM-Solver-2 and DPM-Solver-3;
Revised: we run all the orders of previous solvers, including DPM-Solver-2 and DPM-Solver-3;

Original: DEIS-1, DEIS-2 and DEIS-3 and choose their best result for each NFE in our comparison.
Revised: DEIS-1, DEIS-2 and DEIS-3 and choose their best result for each NFE in our comparison.

Original: We run both DPM-Solver++(2S) and DPM-Solver++(2M), and we find that for large guidance scales, the multistep DPM-Solver++(2M) performs better; and for a slightly small guidance scales, the singlestep DPM-Solver++(2S) performs better.
Revised: We run both DPM-Solver++(2S) and DPM-Solver++(2M), and we find that for large guidance scales, the multistep DPM-Solver++(2M) performs better; and for a slightly small guidance scales, the singlestep DPM-Solver++(2S) performs better.

Original: We report the best results of DPM-Solver++ and all of the previous samplers in the following sections, the detailed values are listed in Appendix D.
Revised: We report the best results of DPM-Solver++ and all of the previous samplers in the following sections, the detailed values are listed in Appendix D.

Original: 6. P IXEL -S PACE DPM S WITH G UIDANCE
Revised: 6. P IXEL -S PACE DPM S WITH G UIDANCE

Original: We firstly compare DPM-Solver++ with other samplers for the guided sampling with classifierguidance on ImageNet 256x256 dataset by the pretrained DPMsDhariwal & Nichol (2021).
Revised: We firstly compare DPM-Solver++ with other samplers for the guided sampling with classifierguidance on ImageNet 256x256 dataset by the pretrained DPMs(Dhariwal & Nichol, 2021).

Original: We measure the sample quality by drawing 10K samples and computing the widely-used FID score (Heusel et al., 2017), where lower FID usually implies better sample quality.
Revised: We measure the sample quality by drawing 10K samples and computing the widely-used FID score (Heusel et al., 2017), where lower FID usually implies better sample quality.

Original: We also adopt the dynamic thresholding method (Saharia et al., 2022b) for both DDIM and DPM-Solver++, while other samplers cannot directly use the thresholding technique.
Revised: We also adopt the dynamic thresholding method (Saharia et al., 2022b) for both DDIM and DPM-Solver++.

Original: We vary the guidance scale s in 8.0, 4.0 and 2.0, the results are shown in Fig.
Revised: We vary the guidance scale s in 8.0, 4.0 and 2.0, the results are shown in Fig.

Original: 3(a-c).
Revised: 3(a-c).

Original: We find that for large guidance scales, all the previous high-order samplers (DEIS, PNDM and DPM-Solver) converge slower than the first-order DDIM, which shows that previous high-order samplers are unstable.
Revised: We find that for large guidance scales, all the previous high-order samplers (DEIS, PNDM, DPM-Solver) converge slower than the first-order DDIM, which shows that previous high-order samplers are unstable.

Original: Instead, DPM-Solver++ achieve the best speedup performance for both large guidance scales and small guidance scales.
Revised: Instead, DPM-Solver++ achieve the best speedup performance for both large guidance scales and small guidance scales.

Original: Especially for large guidance scales, DPM-Solver++ can almost converge within only 15 NFE,
Revised: Especially for large guidance scales, DPM-Solver++ can almost converge within only 15 NFE.

Original: As an ablation, we also compare the singlestep DPM-Solver-2, the singlestep DPM-Solver++(2S) and the multistep DPM-Solver++(2M) to demonstrate the effectiveness of our method.
Revised: As an ablation, we also compare the singlestep DPM-Solver-2, the singlestep DPM-Solver++(2S) and the multistep DPM-Solver++(2M) to demonstrate the effectiveness of our method.

Original: We use a large guidance scale s = 8 . 0 and conduct the following ablations:
Revised: We use a large guidance scale s = 8 . 0 and conduct the following ablations:

Original: • From ϵ θ to x θ : As shown in Fig.
Revised: • From ϵ θ to x θ : As shown in Fig.

Original: 2a, by simply changing the solver from ϵ θ to x θ (i
Revised: 2a, by simply changing the solver from ϵ θ to x θ (i

Original: e. from DPM-Solver-2 to DPM-Solver++(2S)), the solver can achieve a stable acceleration performance which is faster than the first-order DDIM.
Revised: e. from DPM-Solver-2 to DPM-Solver++(2S)), the solver can achieve a stable acceleration performance which is faster than the first-order DDIM.

Original: Such result indicates that for guided sampling, high-order solvers w.r.t.
Revised: Such result indicates that for guided sampling, high-order solvers w.r.t.

Original: x θ may be more preferred than those w.r.t.
Revised: x θ may be more preferred than those w.r.t.

Original: ϵ θ .
Revised: ϵ θ .

Original: • From singlestep to multistep: As show in Fig.
Revised: • From singlestep to multistep: As show in Fig.

Original: 2b, the multistep DPM-Solver++(2M) converges slightly faster than the singlestep DPM-Solver++(2S), which almost converges in 15 NFE.
Revised: 2b, the multistep DPM-Solver++(2M) converges slightly faster than the singlestep DPM-Solver++(2S), which almost converges in 15 NFE.

Original: Such result indicates that for guided sampling with a large guidance scale, multistep methods may be faster than singlestep methods.
Revised: Such result indicates that for guided sampling with a large guidance scale, multistep methods may be faster than singlestep methods.

Original: • With or without thresholding: We compare the performance of DDIM and DPM-Solver++ with / without thresholding methods in Fig.
Revised: • With or without thresholding: We compare the performance of DDIM and DPM-Solver++ with / without thresholding methods in Fig.

Original: 2c.
Revised: 2c.

Original: Note that the thresholding method changes the model x θ and thus also changes the converged solutions of diffusion ODEs.
Revised: Note that the thresholding method changes the model x θ and thus also changes the converged solutions of diffusion ODEs.

Original: Firstly, we find that after using the thresholding method, the diffusion ODE can generate higher quality samples, which is consistent with the conclusion in (Saharia et al., 2022b).
Revised: Firstly, we find that after using the thresholding method, the diffusion ODE can generate higher quality samples, which is consistent with the conclusion in (Saharia et al., 2022b).

Original: Secondly, the sample quality of DPM-Solver++ with thresholding outperforms DPM-Solver++ without thresholding under the same NFE.
Revised: Secondly, the sample quality of DPM-Solver++ with thresholding outperforms DPM-Solver++ without thresholding under the same NFE.

Original: Moreover, when combined with thresholding, DPMSolver++ is faster than the first-order DDIM, which shows that DPM-Solver++ can also speed up guided sampling by DPMs with thresholding methods.
Revised: Moreover, when combined with thresholding, DPMSolver++ is faster than the first-order DDIM, which shows that DPM-Solver++ can also speed up guided sampling by DPMs with thresholding methods.

Original: 6. L ATENT -S PACE DPM S WITH G UIDANCE
Revised: 6. L ATENT -S PACE DPM S WITH G UIDANCE

Original: We also evaluate DPM-Solver++ on the latent-space DPMs (Rombach et al., 2022), which is recently popular among the community due to their official code “stable-diffusion”.
Revised: We also evaluate DPM-Solver++ on the latent-space DPMs (Rombach et al., 2022), which is recently popular among the community due to their official code “stable-diffusion”.

Original: We use the default guidance scale s = 7 . 5 in their official code.
Revised: We use the default guidance scale s = 7 . 5 in their official code.

Original: The latent-space DPMs map the image data with a latent code by training a pair of encoder and decoder, and then train a DPM for the latent code.
Revised: The latent-space DPMs map the image data with a latent code by training a pair of encoder and decoder, and then train a DPM for the latent code.

Original: As the latent code is unbounded, we do not apply the thresholding method.
Revised: As the latent code is unbounded, we do not apply the thresholding method.

Original: Specifically, we randomly sample 10K caption-image pairs from the MS-COCO2014 validation dataset and use the captions as conditions to draw 10K images from the pretrained “stable-diffusion” model, and we only draw a single image sample of each caption, following the standard evaluation procedures in (Nichol et al., 2021; Rombach et al., 2022).
Revised: Specifically, we randomly sample 10K caption-image pairs from the MS-COCO2014 validation dataset and use the captions as conditions to draw 10K images from the pretrained “stable-diffusion” model, and we only draw a single image sample of each caption, following the standard evaluation procedures in (Nichol et al., 2021; Rombach et al., 2022).

Original: We find that all the solvers can achieve a FID around 15.0 to 16.0 even within only 10 steps, which is very close to the FID computed by the converged samples reported in the official page of “stable-diffusion”.
Revised: We find that all the solvers can achieve a FID around 15.0 to 16.0 even within only 10 steps, which is very close to the FID computed by the converged samples reported in the official page of “stable-diffusion”.

Original: We believe it is due to the powerful pretrained decoder, which can map a non-converged latent code to a good image sample.
Revised: We believe it is due to the powerful pretrained decoder, which can map a non-converged latent code to a good image sample.

Original: For latent-space DPMs, different diffusion ODE solvers directly affect the convergence speed on the latent space.
Revised: For latent-space DPMs, different diffusion ODE solvers directly affect the convergence speed on the latent space.

Original: To further compare different samplers for latent-space DPMs, we directly compare different solvers according to the convergence error on the latent space by the L2-norm between the sampled x 0 and the true solution x ∗ 0 (and the error between them is ∥ x 0 − x ∗ 

Revised: To further compare different samplers for latent-space DPMs, we directly compare different solvers according to the convergence error on the latent space by the L2-norm between the sampled x 0 and the true solution x ∗ 0 (and the error between them is ∥ x 0 − x ∗

Original: ∥ 2 / √ D ).
Revised: ∥ 2 / √ D ).

Original: Specifically, we firstly sample 10K noise variables from the standard normal distribution and fix them.
Revised: Specifically, we firstly sample 10K noise variables from the standard normal distribution and fix them.

Original: Then we sample 10K latent codes by different DPM samplers, starting from the 10K fixed noise variables.
Revised: Then we sample 10K latent codes by different DPM samplers, starting from the 10K fixed noise variables.

Original: As all these solvers can be understood as discretizing diffusion ODEs, we compare the sampled latent codes by the true solution x ∗ 0 from a 999-step DDIM with samples x 0 by different samplers within different NFE, and the results are shown in Fig. 3(d).
Revised: As all these solvers can be understood as discretizing diffusion ODEs, we compare the sampled latent codes by the true solution x ∗ 0 from a 999-step DDIM with samples x 0 by different samplers within different NFE, and the results are shown in Fig. 3(d).

Original: We find that the supported fast samplers (DDIM and PNDM) in “stable-diffusion” converge much slower than DPM-Solver++ and DEIS, and we find that the second-order multistep DPM-Solver++ and DEIS achieve a quite close speedup on the latent space.
Revised: We find that the supported fast samplers (DDIM and PNDM) in “stable-diffusion” converge much slower than DPM-Solver++ and DEIS, and we find that the second-order multistep DPM-Solver++ and DEIS achieve a quite close speedup on the latent space.

Original: Moreover, as “stable-diffusion” by default use PNDM with 50 steps, we find that
Revised: Moreover, as “stable-diffusion” by default use PNDM with 50 steps, we find that

Original: DPM-Solver++ can achieve a similar convergence error with only 15 to 20 steps.
Revised: DPM-Solver++ can achieve a similar convergence error with only 15 to 20 steps.

Original: We also present an empirical comparison of the sampled images between different solvers in Appendix D, and we find that DPM-Solver++ can indeed generate quite good image samples within only 15 to 20 steps.
Revised: We also present an empirical comparison of the sampled images between different solvers in Appendix D, and we find that DPM-Solver++ can indeed generate quite good image samples within only 15 to 20 steps.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We study the problem of accelerating guided sampling of DPMs.
Revised: We study the problem of accelerating guided sampling of DPMs.

Original: We demonstrate that previous high-order solvers based on the noise prediction models are abnormally unstable and generate worsequality samples than the first-order solver DDIM for guided sampling with large guidance scales.
Revised: We demonstrate that previous high-order solvers based on the noise prediction models are abnormally unstable and generate worsequality samples than the first-order solver DDIM for guided sampling with large guidance scales.

Original: To address this issue and speed up guided sampling, we propose DPM-Solver++, a training-free fast diffusion ODE solver for guided sampling.
Revised: To address this issue and speed up guided sampling, we propose DPM-Solver++, a training-free fast diffusion ODE solver for guided sampling.

Original: DPM-Solver++ is based on the diffusion ODE with the data prediction models, which can directly adopt the thresholding methods to stabilize the sampling procedure further.
Revised: DPM-Solver++ is based on the diffusion ODE with the data prediction models, which can directly adopt the thresholding methods to stabilize the sampling procedure further.

Original: We propose both singlestep and multistep variants of DPM-Solver++.
Revised: We propose both singlestep and multistep variants of DPM-Solver++.

Original: Experiment results show that DPM-Solver++ can generate high-fidelity samples and almost converge within only 15 to 20 steps, applicable for pixel-space and latent-space DPMs.
Revised: Experiment results show that DPM-Solver++ can generate high-fidelity samples and almost converge within only 15 to 20 steps, applicable for pixel-space and latent-space DPMs.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Like other deep generative models such as GANs, DPMs may also be used to generate adverse fake contents (images).
Revised: Like other deep generative models such as GANs, DPMs may also be used to generate adverse fake contents (images).

Original: The proposed solver can accelerate the guided sampling by DPMs which can further be used for image editing and generate photorealistic fake images.
Revised: The proposed solver can accelerate the guided sampling by DPMs which can further be used for image editing and generate photorealistic fake images.

Original: Such influence may further amplify the potential undesirable affects of DPMs for malicious applications.
Revised: Such influence may further amplify the potential undesirable affects of DPMs for malicious applications.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: Our code is based on the official code of DPM-Solver (Lu et al., 2022) and the pretrained checkpoints in Dhariwal & Nichol (2021) and Stable-Diffusion (Rombach et al., 2022).
Revised: Our code is based on the official code of DPM-Solver (Lu et al., 2022) and the pretrained checkpoints in Dhariwal & Nichol (2021) and Stable-Diffusion (Rombach et al., 2022).

Original: We will release it after the blind review.
Revised: We will release it after the blind review.

Original: In addition, datasets used in experiments are publicly available.
Revised: In addition, datasets used in experiments are publicly available.

Original: Our detailed experiment settings and implementations are listed in Appendix C, and the proof of the solver convergence guarantee are presented in Appendix A
Revised: Our detailed experiment settings and implementations are listed in Appendix C, and the proof of the solver convergence guarantee are presented in Appendix A

Original: A A DDITIONAL P ROOFS A. P ROOF OF P ROPOSITION 4. 
Revised: A A DDITIONAL P ROOFS A. P ROOF OF P ROPOSITION 4.

Original: Taking derivative w.r.t.
Revised: Taking derivative w.r.t.

Original: t in Eq.
Revised: t in Eq.

Original: (8) yields d x t d
Revised: (8) yields d x t d

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from the definitions f ( t )
Revised: where the last inequality follows from the definitions f ( t )

Original: = dlog α t d t
Revised: = dlog α t d t

Original: [Equation]
Revised: [Equation]

Original: A. C ONVERGENCE OF A LGORITHMS
Revised: A. C ONVERGENCE OF A LGORITHMS

Original: We make the following assumptions as in Lu et al.
Revised: We make the following assumptions as in Lu et al.

Original: for x θ , i.e.,
Revised: for x θ , i.e.,

Original: • x (0) θ , x (1) θ and x (2) θ exist and are continuous (and hence are bounded).
Revised: • x (0) θ , x (1) θ and x (2) θ exist and are continuous (and hence are bounded).

Original: • The map x
Revised: • The map x

Original: (cid:55)→ x θ ( x , t ) is L -Lipschitz.
Revised: (cid:55)→ x θ ( x , t ) is L -Lipschitz.

Original: • h max := max 1 ≤ j ≤ M h
Revised: • h max := max 1 ≤ j ≤ M h

Original: j
Revised: j

Original: =
Revised: =

Original: O
Revised: O

Original: (1 /M ) .
Revised: (1 /M ) .

Original: We also assume further
Revised: We also assume further

Original: • r i > c > 0 for all i
Revised: • r i > c > 0 for all i

Original: = 1 , · · · , M .
Revised: = 1 , · · · , M .

Original: Then, both algorithms are second-order:
Revised: Then, both algorithms are second-order:

Original: Proposition A.1.
Revised: Proposition A.1.

Original: Under the above assumptions, when h max is sufficiently small, we have for both
Revised: Under the above assumptions, when h max is sufficiently small, we have for both

Original: Algorithms 1 and 2, ∥ x t M − ˜ x t M ∥ =
Revised: Algorithms 1 and 2, ∥ x t M − ˜ x t M ∥ =

Original: O ( h 2 max ) .
Revised: O ( h 2 max ) .

Original: A.2. C ONVERGENCE OF A LGORITHM
Revised: A.2. C ONVERGENCE OF A LGORITHM

Original: The convergence proof of this algorithm is similar to that in DPM-Solver-

Revised: The convergence proof of this algorithm is similar to that in DPM-Solver-

Original: (Lu et al., 2022).
Revised: (Lu et al., 2022).

Original: We give it in this section for completeness.
Revised: We give it in this section for completeness.

Original: First, Taylor’s expansion gives
Revised: First, Taylor’s expansion gives

Original: [Equation]
Revised: [Equation]

Original: Let ∆ i := ∥
Revised: Let ∆ i := ∥

Original: ˜ x t i − x t i ∥ , then ∥ u i − x s i ∥ ≤ C ∆ i − 1 + CLh i ∆ i − 1 + Ch 2 i .
Revised: ˜ x t i − x t i ∥ , then ∥ u i − x s i ∥ ≤ C ∆ i − 1 + CLh i ∆ i − 1 + Ch 2 i .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: ≤ Ch i .
Revised: ≤ Ch i .

Original: Since r i is bounded away from zero, and e − h i = 1 − h
Revised: Since r i is bounded away from zero, and e − h i = 1 − h

Original: i + h 2 i / 2 + O ( h 3 i ) , we know (cid:13)(cid:13)(cid:13)(cid:13)
Revised: i + h 2 i / 2 + O ( h 3 i ) , we know (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: ) −
Revised: ) −

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: − x θ
Revised: − x θ

Original: [Equation]
Revised: [Equation]

Original: ˜ x t i −
Revised: ˜ x t i −

Original: [Equation]
Revised: [Equation]

Original: t i − 1 ) (cid:1)(cid:13)(cid:13)(cid:13)(cid:13) ≤ CLh i (∆ i − 1 + ∥ u
Revised: t i − 1 ) (cid:1)(cid:13)(cid:13)(cid:13)(cid:13) ≤ CLh i (∆ i − 1 + ∥ u

Original: i − x s i ∥ )
Revised: i − x s i ∥ )

Original: + Ch 3 i
Revised: + Ch 3 i

Original: +r i (cid:12)(cid:12)(cid:12)(cid:12)
Revised: +r i (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: where M i = 1 + sup t
Revised: where M i = 1 + sup t

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: 1 ≤ t ≤ t i ∥ x (1) θ
Revised: 1 ≤ t ≤ t i ∥ x (1) θ

Original: [Equation]
Revised: [Equation]

Original: x t , t ) ∥ .
Revised: x t , t ) ∥ .

Original: Then, ∆ i could be estimated as follows.
Revised: Then, ∆ i could be estimated as follows.

Original: [Equation]
Revised: [Equation]

Original: Thus, ∆ i =
Revised: Thus, ∆ i =

Original: O ( h 2 max ) as long as h max is sufficiently small.
Revised: O ( h 2 max ) as long as h max is sufficiently small.

Original: A. C ONVERGENCE OF A LGORITHM
Revised: A. C ONVERGENCE OF A LGORITHM

Original: Following the same line of argument of the convergence proof of Algorithm 1, we can prove the convergence of Algorithm 2.
Revised: Following the same line of argument of the convergence proof of Algorithm 1, we can prove the convergence of Algorithm 2.

Original: Let ∆ i := ∥
Revised: Let ∆ i := ∥

Original: ˜ x t i − x t i ∥ .
Revised: ˜ x t i − x t i ∥ .

Original: Taylor’s expansion yields
Revised: Taylor’s expansion yields

Original: [Equation]
Revised: [Equation]

Original: where C is a constant depends on x (2) θ .
Revised: where C is a constant depends on x (2) θ .

Original: Also note that
Revised: Also note that

Original: [Equation]
Revised: [Equation]

Original: Since r i is bounded away from zero, and e − h i = 1 − h
Revised: Since r i is bounded away from zero, and e − h i = 1 − h

Original: i + h 2 i / 2 + O ( h 3 i ) , we know
Revised: i + h 2 i / 2 + O ( h 3 i ) , we know

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation]
Revised: [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] ≤ CLh i (∆ i − 1 + ∆ i − 2 )
Revised: [Table] ≤ CLh i (∆ i − 1 + ∆ i − 2 )

Original: + CM i h 3 i , where M i = 1 + sup t
Revised: + CM i h 3 i , where M i = 1 + sup t

Original: i − 1 ≤ t ≤ t i ∥ x (1) θ ( x t , t ) ∥ .
Revised: i − 1 ≤ t ≤ t i ∥ x (1) θ ( x t , t ) ∥ .

Original: Then, ∆ i could be estimated as follows.
Revised: Then, ∆ i could be estimated as follows.

Original: [Equation]
Revised: [Equation]

Original: Thus, ∆ i =
Revised: Thus, ∆ i =

Original: O ( h 2 max ) as long as h max is sufficiently small and ∆ 

Revised: O ( h 2 max ) as long as h max is sufficiently small and ∆

Original: + ∆ 1 = O ( h 2 max ) , which can be verified by the Taylor’s expansion.
Revised: + ∆ 1 = O ( h 2 max ) , which can be verified by the Taylor’s expansion.

Original: B C OMPARISON BETWEEN DPM-S OLVER AND DPM-S OLVER ++
Revised: B C OMPARISON BETWEEN DPM-S OLVER AND DPM-S OLVER ++

Original: In this section, we convert DPM-Solver++(2S) to the formulation w.r.t.
Revised: In this section, we convert DPM-Solver++(2S) to the formulation w.r.t.

Original: the noise prediction model and compare it with the second-order DPM-Solver (Lu et al., 2022)
Revised: the noise prediction model and compare it with the second-order DPM-Solver (Lu et al., 2022)

Original: [Equation]
Revised: [Equation]

Original: At each step, the second-order DPM-Solver (DPM-Solver-2 (Lu et al., 2022)) has the following updating:
Revised: At each step, the second-order DPM-Solver (DPM-Solver-2 (Lu et al., 2022)) has the following updating:

Original: [Equation]
Revised: [Equation]

Original: while DPM-Solver++(2S) has the following updating:
Revised: while DPM-Solver++(2S) has the following updating:

Original: [Equation]
Revised: [Equation]

Original: Because
Revised: Because

Original: [Equation]
Revised: [Equation]

Original: we can rewrite DPM-Solver++(2S) w.r.t.
Revised: we can rewrite DPM-Solver++(2S) w.r.t.

Original: the noise prediction model (see Appendix B.1 for details):
Revised: the noise prediction model (see Appendix B.1 for details):

Original: [Equation]
Revised: [Equation]

Original: Comparing with Eq.
Revised: Comparing with Eq.

Original: 10, we can find that the only difference between DPM-Solver-2 and DPMSolver++(2S) is that DPM-Solver++(2S) has an additional coefficient e − r
Revised: 10, we can find that the only difference between DPM-Solver-2 and DPMSolver++(2S) is that DPM-Solver++(2S) has an additional coefficient e − r

Original: i h
Revised: i h

Original: i < 1 at the second term (which is corresponding to approximating the first-order total derivativeof ϵ  θ ).
Revised: i < 1 at the second term (which is corresponding to approximating the first-order total derivative ϵ (1) θ ).

Revised: Specifically, we have

Original:  As they both are equivalent to a second-order discretization of the diffusion ODE, a smaller second-order approximation may reduce the numerical instabilities  for large guidance scales, which may explain why DPM-Solver++(2S) is more stable than DPM-Solver-2.
Revised: [Equation] As DPM-Solver++(2S) multiplies a smaller coefficient into the O ( h i ) error term, the constant before the high-order error term of DPM-Solver++(2S) is smaller than that of DPM-Solver-2. As they both are equivalent to a second-order discretization of the diffusion ODE, a smaller constant before the error term can result in a smaller discretization error and reducing the numerical instabilities (especially for large guidance scales). Therefore, using the data prediction model is a key for stabilizing the sampling, and DPM-Solver++(2S) is more stable than DPM-Solver-2.

Original: B. D ETAILED D ERIVATION
Revised: B. D ETAILED D ERIVATION

Original: We can rewrite DPM-Solver++(2S) by:
Revised: We can rewrite DPM-Solver++(2S) by:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]

Original: and

Original: [Equation]
Revised: [Equation]

Original: so we have
Revised: so we have

Original: [Equation]
Revised: [Equation]

Original: C I MPLEMENTATION D ETAILS C. C ONVERTING D ISCRETE -T IME DPM S TO C ONTINUOUS -T IME
Revised: C I MPLEMENTATION D ETAILS C. C ONVERTING D ISCRETE -T IME DPM S TO C ONTINUOUS -T IME

Original: Discrete-time DPMs (Ho et al., 2020) train the noise prediction model ϵ θ at N fixed time steps { t n } Nn =1 and the noise prediction model is parameterized by ˜ ϵ θ ( x n , 1000 n N ) for n = 1 , . .
Revised: Discrete-time DPMs (Ho et al., 2020) train the noise prediction model ϵ θ at N fixed time steps { t n } Nn =1 and the noise prediction model is parameterized by ˜ ϵ θ ( x n , 1000 n N ) for n = 1 , . .

Original: N , where each x n is corresponding to the value at time t n .
Revised: N , where each x n is corresponding to the value at time t n .

Original: In practice, these discrete-time DPMs usually choose uniform time steps between [0 , T ] , thus t n = nTN , for n = 1 , . . .
Revised: In practice, these discrete-time DPMs usually choose uniform time steps between [0 , T ] , thus t n = nTN , for n = 1 , . . .

Original: , N .
Revised: , N .

Original: The smallest time is TN .
Revised: The smallest time is TN .

Original: Moreover, for the widely-used DDPM (Ho et al., 2020), we usually choose a sequence { β n } Nn = which is defined by either linear schedule (Ho et al., 2020) or cosine schedule (Nichol & Dhariwal, 2021).
Revised: Moreover, for the widely-used DDPM (Ho et al., 2020), we usually choose a sequence { β n } Nn = which is defined by either linear schedule (Ho et al., 2020) or cosine schedule (Nichol & Dhariwal, 2021).

Original: After obtained the β n sequence, the noise schedule α n is defined by
Revised: After obtained the β n sequence, the noise schedule α n is defined by

Original: [Equation]
Revised: [Equation]

Original: where each α n is corresponding to the continuous-time t n = nTN , i.e. α t n = α n .
Revised: where each α n is corresponding to the continuous-time t n = nTN , i.e. α t n = α n .

Original: To generalize the discrete α n to the continuous version, we use a linear interpolation for the function log α n .
Revised: To generalize the discrete α n to the continuous version, we use a linear interpolation for the function log α n .

Original: Specifically, for each t ∈ [ t n , t n +1 ] , we define
Revised: Specifically, for each t ∈ [ t n , t n +1 ] , we define

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can obtain a continuous-time noise schedule α t defined for all t ∈ [ TN , T ] , and the std σ t = (cid:112) 1 − α 2 t and the logSNR λ t = log α t − log σ t .
Revised: Therefore, we can obtain a continuous-time noise schedule α t defined for all t ∈ [ TN , T ] , and the std σ t = (cid:112) 1 − α 2 t and the logSNR λ t = log α t − log σ t .

Original: Moreover, the logSNR λ t is strictly decreasing for t , thus the change-of-variable for λ is still valid.
Revised: Moreover, the logSNR λ t is strictly decreasing for t , thus the change-of-variable for λ is still valid.

Original: In practice, we usually have T = 1 and N = 1000 , thus the smallest time is 10 − 3 .
Revised: In practice, we usually have T = 1 and N = 1000 , thus the smallest time is 10 − 3 .

Original: Therefore, we solve the diffusion ODEs from time t = 1 to time t = 10 − 3 to get our final sample.
Revised: Therefore, we solve the diffusion ODEs from time t = 1 to time t = 10 − 3 to get our final sample.

Original: Such sampling can reduce the first-order discrete-time DDIM solver when using a uniform time step.
Revised: Such sampling can reduce the first-order discrete-time DDIM solver when using a uniform time step.

Original: C. A BLATING T IME S TEPS
Revised: C. A BLATING T IME S TEPS

Original: Previous DEIS only tuned on low-resolutional data CIFAR-10, which may be not suitable for highresolutional data such as ImageNet 256x256 and large guidance scales for guided sampling.
Revised: Previous DEIS only tuned on low-resolutional data CIFAR-10, which may be not suitable for highresolutional data such as ImageNet 256x256 and large guidance scales for guided sampling.

Original: For a fair comparison with the baseline samplers, we firstly do ablation study for the time steps with the pretrained DPMs (Dhariwal & Nichol, 2021) on ImageNet 256x256 and vary the classifier guidance scale.
Revised: For a fair comparison with the baseline samplers, we firstly do ablation study for the time steps with the pretrained DPMs (Dhariwal & Nichol, 2021) on ImageNet 256x256 and vary the classifier guidance scale.

Original: In our experiments, we tune the time step schedule according to their power function choices.
Revised: In our experiments, we tune the time step schedule according to their power function choices.

Original: Specifically, let t M = 10 − 3 and t 

Revised: Specifically, let t M = 10 − 3 and t

Original: = 1 , the time steps { t i } Mi =

Revised: = 1 , the time steps { t i } Mi =

Original: satisfies
Revised: satisfies

Original: [Equation]
Revised: [Equation]

Original: where κ is a hyperparameter.
Revised: where κ is a hyperparameter.

Original: Following Zhang & Chen (2022), we search κ in 1 , 2 , 3 by DEIS, and the results are shown in Table 2.
Revised: Following Zhang & Chen (2022), we search κ in 1 , 2 , 3 by DEIS, and the results are shown in Table 2.

Original: We find that for all guidance scales, the best setting is κ = 1 , i.e. the uniform t for time steps.
Revised: We find that for all guidance scales, the best setting is κ = 1 , i.e. the uniform t for time steps.

Original: We further compare uniform t and uniform λ and find that the uniform t time step schedule is still the best choice.
Revised: We further compare uniform t and uniform λ and find that the uniform t time step schedule is still the best choice.

Original: Therefore, in all of our experiments, we use the uniform t for evaluations.
Revised: Therefore, in all of our experiments, we use the uniform t for evaluations.

Original: C. E XPERIMENT S ETTINGS
Revised: C. E XPERIMENT S ETTINGS

Original: We use uniform time step schedule for all experiments.
Revised: We use uniform time step schedule for all experiments.

Original: Particularly, as DPM-Solver (Lu et al., 2022) is designed for uniform λ (the intermediate time steps are a half of the step size w.r.t.
Revised: Particularly, as DPM-Solver (Lu et al., 2022) is designed for uniform λ (the intermediate time steps are a half of the step size w.r.t.

Original: λ ), we also convert the intermediate time steps to ensure all the time steps are uniform t .
Revised: λ ), we also convert the intermediate time steps to ensure all the time steps are uniform t .

Original: We find that such conversion can improve the sample quality of both the singlestep DPM-Solver the singlestep
Revised: We find that such conversion can improve the sample quality of both the singlestep DPM-Solver the singlestep

Original: DPM-Solver++.
Revised: DPM-Solver++.

Original: We run NFE in 10, 15, 20, 25 for the high-order solvers and additional 50, 100, 250 for DDIM.
Revised: We run NFE in 10, 15, 20, 25 for the high-order solvers and additional 50, 100, 250 for DDIM.

Original: For all experiments, we solver diffusion ODEs from t = 1 to t = 10 − 3 with the interpolation of noise schedule detailed in Appendix C.1.
Revised: For all experiments, we solver diffusion ODEs from t = 1 to t = 10 − 3 with the interpolation of noise schedule detailed in Appendix C.1.

Original: For DEIS, we use the “t-AB- k ” methods for k = 1 , 2 , 3 , which is the fastest method in their original paper, and we name them as DEIS- k , respectively.
Revised: For DEIS, we use the “t-AB- k ” methods for k = 1 , 2 , 3 , which is the fastest method in their original paper, and we name them as DEIS- k , respectively.

Original: For the sampled image in Fig.
Revised: For the sampled image in Fig.

Original: 5, we use the prompt “A beautiful castle beside a waterfall in the woods, by Josef Thoma, matte painting, trending on artstation HQ”.
Revised: 5, we use the prompt “A beautiful castle beside a waterfall in the woods, by Josef Thoma, matte painting, trending on artstation HQ”.

Original: [Table]
Revised: [Table]

Original: D E XPERIMENT D ETAILS
Revised: D E XPERIMENT D ETAILS

Original: We list all the detailed experimental results in this section.
Revised: We list all the detailed experimental results in this section.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: \ \ \ DEIS-1 (Zhang & Chen, 2022) 0.84 0.72 0.64 0.

Original: [Table]
Revised: \ \ \ DEIS-2 (Zhang & Chen, 2022) 0.87 0.76 0.68 0. \ \ \ DEIS-3 (Zhang & Chen, 2022) 1.06 0.88 0.78 0. DPM-Solver++(S) ( ours ) 0.88 0.75 0.68 0. \ \ \ DPM-Solver++(M) ( ours ) 0.84 0.72 0.64 0.

Original: [Figure]
Revised: [Figure]

Original: DDIM ( N = 999 , converged)
Revised: DDIM ( N = 999 , converged)

Original: (Song et al., 2021a)
Revised: (Song et al., 2021a)

Original: DDIM ( N = 15 ) (Song et al., 2021a)
Revised: DDIM ( N = 15 ) (Song et al., 2021a)

Original: PNDM ( N = 15 ) (Liu et al., 2022b)
Revised: PNDM ( N = 15 ) (Liu et al., 2022b)

Original: DPM-Solver-2 ( N = 15 ) (Lu et al., 2022)
Revised: DPM-Solver-2 ( N = 15 ) (Lu et al., 2022)

Original: DEIS-1 ( N = 15 ) (Zhang & Chen, 2022)
Revised: DEIS-1 ( N = 15 ) (Zhang & Chen, 2022)

Original: DPM-Solver++(2M) ( N = 15 ) (ours)
Revised: DPM-Solver++(2M) ( N = 15 ) (ours)

Original: DDIM ( N = 20 ) (Song et al., 2021a)
Revised: DDIM ( N = 20 ) (Song et al., 2021a)

Original: PNDM ( N = 20 ) (Liu et al., 2022b)
Revised: PNDM ( N = 20 ) (Liu et al., 2022b)

Original: DPM-Solver-2 ( N = 20 ) (Lu et al., 2022)
Revised: DPM-Solver-2 ( N = 20 ) (Lu et al., 2022)

Original: DEIS-1 ( N = 20 ) (Zhang & Chen, 2022)
Revised: DEIS-1 ( N = 20 ) (Zhang & Chen, 2022)

Original: DPM-Solver++(2M) ( N = 20 ) (ours)
Revised: DPM-Solver++(2M) ( N = 20 ) (ours)

Original: DDIM ( N = 50 ) (Song et al., 2021a)
Revised: DDIM ( N = 50 ) (Song et al., 2021a)

Original: PNDM ( N = 50 ) (Liu et al., 2022b)
Revised: PNDM ( N = 50 ) (Liu et al., 2022b)

Original: DPM-Solver-2 ( N = 50 ) (Lu et al., 2022)
Revised: DPM-Solver-2 ( N = 50 ) (Lu et al., 2022)

Original: DEIS-1 ( N = 50 ) (Zhang & Chen, 2022)
Revised: DEIS-1 ( N = 50 ) (Zhang & Chen, 2022)

Original: DPM-Solver++(2M) (
Revised: DPM-Solver++(2M) (

Original: N = 50 ) (ours)
Revised: N = 50 ) (ours)