Original: On Fast Sampling of Diffusion Probabilistic Models
Revised: On Fast Sampling of Diffusion Probabilistic Models

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: In this work, we propose FastDPM, a uniﬁed framework for fast sampling in diffusion probabilistic models.
Revised: In this work, we propose FastDPM, a uniﬁed framework for fast sampling in diffusion probabilistic models.

Original: FastDPM generalizes previous methods and gives rise to new algorithms with improved sample quality.
Revised: FastDPM generalizes previous methods and gives rise to new algorithms with improved sample quality.

Original: We systematically investigate the fast sampling methods under this framework across different domains, on different datasets, and with different amount of conditional information provided for generation.
Revised: We systematically investigate the fast sampling methods under this framework across different domains, on different datasets, and with different amount of conditional information provided for generation.

Original: We ﬁnd the performance of a particular method depends on data domains (e.g., image or audio), the trade-off between sampling speed and sample quality, and the amount of conditional information.
Revised: We ﬁnd the performance of a particular method depends on data domains (e.g., image or audio), the trade-off between sampling speed and sample quality, and the amount of conditional information.

Original: We further provide insights and recipes on the choice of methods for practitioners.
Revised: We further provide insights and recipes on the choice of methods for practitioners.

Original: 1. Introduction
Revised: 1. Introduction

Original: Diffusion probabilistic models are a class of deep generative models that use Markov chains to gradually transform between a simple distribution (e.g., isotropic Gaussian) and the complex data distribution (Sohl-Dickstein et al., 2015;
Revised: Diffusion probabilistic models are a class of deep generative models that use Markov chains to gradually transform between a simple distribution (e.g., isotropic Gaussian) and the complex data distribution (Sohl-Dickstein et al., 2015;

Original: Ho et al., 2020).
Revised: Ho et al., 2020).

Original: Most recently, these models have obtained the state-of-the-art results in several important domains, including image synthesis (Ho et al., 2020; Song et al., 2020b;
Revised: Most recently, these models have obtained the state-of-the-art results in several important domains, including image synthesis (Ho et al., 2020; Song et al., 2020b;

Original: Dhariwal & Nichol, 2021), audio synthesis (Kong et al., 2020b; Chen et al., 2020), and 3-D point cloud generation (Luo & Hu, 2021; Zhou et al., 2021).
Revised: Dhariwal & Nichol, 2021), audio synthesis (Kong et al., 2020b; Chen et al., 2020), and 3-D point cloud generation (Luo & Hu, 2021; Zhou et al., 2021).

Original: We will use “diffusion models” as shorthand to refer to these models.
Revised: We will use “diffusion models” as shorthand to refer to these models.

Original: Diffusion models usually comprise: i) a parameter-free Tstep Markov chain named the diffusion process , which gradually adds random noise into the data, and ii) a parameterized T -step
Revised: Diffusion models usually comprise: i) a parameter-free Tstep Markov chain named the diffusion process , which gradually adds random noise into the data, and ii) a parameterized T -step

Original: Markov chain called the reverse or denoising process , which removes the added noise as a denoising function.
Revised: Markov chain called the reverse or denoising process , which removes the added noise as a denoising function.

Original: The likelihood in diffusion models is intractable, but they can be efﬁciently trained by optimizing a variant of the variational lower bound.
Revised: The likelihood in diffusion models is intractable, but they can be efﬁciently trained by optimizing a variant of the variational lower bound.

Original: In particular, Ho et al.
Revised: In particular, Ho et al.

Original: (2020) propose a certain parameterization called the denoising diffusion probabilistic model (DDPM) and show its connection with denoising score matching (Song & Ermon, 2019), so the reverse process can be viewed as sampling from a scorebased model using Langevin dynamics.
Revised: (2020) propose a certain parameterization called the denoising diffusion probabilistic model (DDPM) and show its connection with denoising score matching (Song & Ermon, 2019), so the reverse process can be viewed as sampling from a scorebased model using Langevin dynamics.

Original: DDPM can produce high-ﬁdelity samples reliably with large model capacity and outperforms the state-of-the-art models in image and audio domains (Dhariwal & Nichol, 2021; Kong et al., 2020b).
Revised: DDPM can produce high-ﬁdelity samples reliably with large model capacity and outperforms the state-of-the-art models in image and audio domains (Dhariwal & Nichol, 2021; Kong et al., 2020b).

Original: However, a noticeable limitation of diffusion models is their expensive denoising or sampling process.
Revised: However, a noticeable limitation of diffusion models is their expensive denoising or sampling process.

Original: For example,
Revised: For example,

Original: DDPM requires a Markov chain with T = 1000 steps to generate high quality image samples (Ho et al., 2020), and
Revised: DDPM requires a Markov chain with T = 1000 steps to generate high quality image samples (Ho et al., 2020), and

Original: DiffWave requires T = 200 to obtain high-ﬁdelity audio synthesis (Kong et al., 2020b).
Revised: DiffWave requires T = 200 to obtain high-ﬁdelity audio synthesis (Kong et al., 2020b).

Original: In other words, one has to run the forward-pass of the neural network T times to generate a sample, which is much slower than the state-of-the-art
Revised: In other words, one has to run the forward-pass of the neural network T times to generate a sample, which is much slower than the state-of-the-art

Original: GANs or ﬂow-based models for image and audio synthesis (e.g., Karras et al., 2020; Kingma & Dhariwal, 2018;
Revised: GANs or ﬂow-based models for image and audio synthesis (e.g., Karras et al., 2020; Kingma & Dhariwal, 2018;

Original: Kong et al., 2020a; Ping et al., 2020).
Revised: Kong et al., 2020a; Ping et al., 2020).

Original: To deal with this limitation, several methods have been proposed to reduce the length of the reverse process to S (cid:28) T steps.
Revised: To deal with this limitation, several methods have been proposed to reduce the length of the reverse process to S (cid:28) T steps.

Original: One class of methods compute continuous noise levels based on discrete diffusion steps and retrain a new model conditioned on these continuous noise levels (Song &
Revised: One class of methods compute continuous noise levels based on discrete diffusion steps and retrain a new model conditioned on these continuous noise levels (Song &

Original: Ermon, 2019; Chen et al., 2020; Okamoto et al., 2021; SanRoman et al., 2021).
Revised: Ermon, 2019; Chen et al., 2020; Okamoto et al., 2021; SanRoman et al., 2021).

Original: Then, a shorter reverse process can be obtained by carefully choosing a small set (size S ) of noise levels.
Revised: Then, a shorter reverse process can be obtained by carefully choosing a small set (size S ) of noise levels.

Original: However, these methods cannot reuse the pretrained diffusion models, because the state-of-the-art DDPM models are conditioned on discrete diffusion steps (Ho et al., 2020; Dhariwal & Nichol, 2021).
Revised: However, these methods cannot reuse the pretrained diffusion models, because the state-of-the-art DDPM models are conditioned on discrete diffusion steps (Ho et al., 2020; Dhariwal & Nichol, 2021).

Original: It is also unclear the diffusion models conditioned on continuous noise levels can achieve comparable sample quality as the state-of-the-art
Revised: It is also unclear the diffusion models conditioned on continuous noise levels can achieve comparable sample quality as the state-of-the-art

Original: DDPMs on challenging unconditional image and audio synthesis tasks (Dhariwal & Nichol, 2021; Kong et al., 2020b).
Revised: DDPMs on challenging unconditional image and audio synthesis tasks (Dhariwal & Nichol, 2021; Kong et al., 2020b).

Original: Another class of methods directly approximate the original reverse process of DDPM models with shorter ones (of length S ), which are conditioned on discrete diffusion steps (Song et al., 2020a; Kong et al., 2020b).
Revised: Another class of methods directly approximate the original reverse process of DDPM models with shorter ones (of length S ), which are conditioned on discrete diffusion steps (Song et al., 2020a; Kong et al., 2020b).

Original: Although both classes of methods have shown the trade-off between sampling speed and sample quality (i.e., larger S lead to higher sample quality), the fast sampling methods without retraining are more advantageous for fast iteration and deployment, while still keeping high-ﬁdelity synthesis with small number of steps in the reverse process (e.g., S = 6 in
Revised: Although both classes of methods have shown the trade-off between sampling speed and sample quality (i.e., larger S lead to higher sample quality), the fast sampling methods without retraining are more advantageous for fast iteration and de- ployment, while still keeping high-ﬁdelity synthesis with small number of steps in the reverse process (e.g., S = 6 in

Original: Kong et al.
Revised: Kong et al.

Original: (2020b)).
Revised: (2020b)).

Original: In this work, we propose FastDPM, a uniﬁed framework of fast sampling methods for diffusion models without retraining.
Revised: In this work, we propose FastDPM, a uniﬁed framework of fast sampling methods for diffusion models without retraining.

Original: The core idea of FastDPM is to i) generalize discrete diffusion steps to continuous diffusion steps, and ii) design a bijective mapping between continuous diffusion steps and continuous noise levels.
Revised: The core idea of FastDPM is to i) generalize discrete diffusion steps to continuous diffusion steps, and ii) design a bijective mapping between continuous diffusion steps and continuous noise levels.

Original: Then, we use this bijection to construct an approximate diffusion process and an approximate reverse process, both of which have length S (cid:28) T .
Revised: Then, we use this bijection to construct an approximate diffusion process and an approximate reverse process, both of which have length S (cid:28) T .

Original: FastDPM includes and generalizes the fast sampling algorithms from denoising diffusion implicit models (DDIM)
Revised: FastDPM includes and generalizes the fast sampling algorithms from denoising diffusion implicit models (DDIM)

Original: (Song et al., 2020a) and DiffWave (Kong et al., 2020b).
Revised: (Song et al., 2020a) and DiffWave (Kong et al., 2020b).

Original: In detail, FastDPM offers two ways to construct the approximate diffusion process: selecting S steps in the original diffusion process, or more ﬂexibly, choosing S variances.
Revised: In detail, FastDPM offers two ways to construct the approximate diffusion process: selecting S steps in the original diffusion process, or more ﬂexibly, choosing S variances.

Original: FastDPM also offers ways to construct the approximate reverse process: using the stochastic DDPM reverse process (DDPM-rev), or using the implicit (deterministic)
Revised: FastDPM also offers ways to construct the approximate reverse process: using the stochastic DDPM reverse process (DDPM-rev), or using the implicit (deterministic)

Original: DDIM reverse process (DDIM-rev).
Revised: DDIM reverse process (DDIM-rev).

Original: We can control the amount of stochasticity in the reverse process of FastDPM as in Song et al.
Revised: We can control the amount of stochasticity in the reverse process of FastDPM as in Song et al.

Original: (2020a).
Revised: (2020a).

Original: FastDPM gives rise to new algorithms with improved sample quality than previous methods when the length of the approximate reverse process S is small.
Revised: FastDPM gives rise to new algorithms with improved sample quality than previous methods when the length of the approximate reverse process S is small.

Original: We then extensively evaluate the family of FastDPM methods across image and audio domains.
Revised: We then extensively evaluate the family of FastDPM methods across image and audio domains.

Original: We ﬁnd the deterministic DDIM-rev signiﬁcantly outperforms the stochastic DDPM-rev in image generation tasks, but DDPM-rev signiﬁcantly outperforms
Revised: We ﬁnd the deterministic DDIM-rev signiﬁcantly outperforms the stochastic DDPM-rev in image generation tasks, but DDPM-rev signiﬁcantly outperforms

Original: DDIM-rev in audio synthesis tasks.
Revised: DDIM-rev in audio synthesis tasks.

Original: Finally, we investigate the performance of different methods by varying the amount of conditional information.
Revised: Finally, we investigate the performance of different methods by varying the amount of conditional information.

Original: We ﬁnd with different amount of conditional information, we need different amount of stochasticity in the reverse process of FastDPM.
Revised: We ﬁnd with different amount of conditional information, we need different amount of stochasticity in the reverse process of FastDPM.

Original: We discuss related work in Appendix A.
Revised: We discuss related work in Appendix A.

Original: 2. Diffusion Models
Revised: 2. Diffusion Models

Original: Let d be the data dimension.
Revised: Let d be the data dimension.

Original: Let p data be the data distribution and p latent =
Revised: Let p data be the data distribution and p latent =

Original: N (0 , I d × d ) be the latent distribution.
Revised: N (0 , I d × d ) be the latent distribution.

Original: Then, the denoising diffusion probabilistic model (DDPM,
Revised: Then, the denoising diffusion probabilistic model (DDPM,

Original: Sohl-Dickstein et al., 2015; Ho et al., 2020) is a deep generative model consisting two Markov chains called diffusion and reverse processes, respectively.
Revised: Sohl-Dickstein et al., 2015; Ho et al., 2020) is a deep generative model consisting two Markov chains called diffusion and reverse processes, respectively.

Original: The length of each
Revised: The length of each

Original: Markov chain is T , which is called the number of diffusion or reverse steps.
Revised: Markov chain is T , which is called the number of diffusion or reverse steps.

Original: The diffusion process gradually adds Gaussian noise to the data distribution until the noisy data distribution is close to the latent distribution.
Revised: The diffusion process gradually adds Gaussian noise to the data distribution until the noisy data distribution is close to the latent distribution.

Original: Formally, the diffusion process from data x 0 ∼ p data to the latent variable x T is deﬁned as q ( x 1 , · · · , x T | x 0 )
Revised: Formally, the diffusion process from data x 0 ∼ p data to the latent variable x T is deﬁned as q ( x 1 , · · · , x T | x 0 )

Original: = (cid:81)
Revised: = (cid:81)

Original: Tt =
Revised: Tt =

Original: 1 q ( x t | x t − 1 ) , where each of q ( x t | x t − 1 ) =
Revised: 1 q ( x t | x t − 1 ) , where each of q ( x t | x t − 1 ) =

Original: N ( x t ; √ 1 − β t x t − 1 , β t I ) for some small constant β t > 0 .
Revised: N ( x t ; √ 1 − β t x t − 1 , β t I ) for some small constant β t > 0 .

Original: The hyperparameters β 1 , · · · , β T are called the variance schedule .
Revised: The hyperparameters β 1 , · · · , β T are called the variance schedule .

Original: The reverse process aims to eliminate the noise added in each diffusion step.
Revised: The reverse process aims to eliminate the noise added in each diffusion step.

Original: Formally, the reverse process from x T ∼ p latent to x 0 is deﬁned as p θ ( x 0 , · · · , x T − 1 | x T ) =
Revised: Formally, the reverse process from x T ∼ p latent to x 0 is deﬁned as p θ ( x 0 , · · · , x T − 1 | x T ) =

Original: (cid:81)
Revised: (cid:81)

Original: Tt =1 p θ ( x t − 1 | x t ) , where each of p θ ( x t − 1 | x t ) is deﬁned as N ( x t − 1 ; µ θ ( x t , t ) , σ 2 t I ) ; the mean µ θ ( x t , t ) is parameterized through a neural network and the variance σ t is time-step dependent constant.
Revised: Tt =1 p θ ( x t − 1 | x t ) , where each of p θ ( x t − 1 | x t ) is deﬁned as N ( x t − 1 ; µ θ ( x t , t ) , σ 2 t I ) ; the mean µ θ ( x t , t ) is parameterized through a neural network and the variance σ t is time-step dependent constant.

Original: Based on the reverse process, the sampling process is to ﬁrst draw x T ∼ N (0 , I ) , then draw x t − 1 ∼ p θ ( x t − 1 | x t ) for t =
Revised: Based on the reverse process, the sampling process is to ﬁrst draw x T ∼ N (0 , I ) , then draw x t − 1 ∼ p θ ( x t − 1 | x t ) for t =

Original: T, T − 1 , · · · , 1 , and ﬁnally outputs x 0 .
Revised: T, T − 1 , · · · , 1 , and ﬁnally outputs x 0 .

Original: The training objective of DDPM is based on the variational evidence lower bound (ELBO).
Revised: The training objective of DDPM is based on the variational evidence lower bound (ELBO).

Original: Under a certain parameterization introduced by Ho et al.
Revised: Under a certain parameterization introduced by Ho et al.

Original: (2020), the objective can be largely simpliﬁed.
Revised: (2020), the objective can be largely simpliﬁed.

Original: One may ﬁrst deﬁne constants
Revised: One may ﬁrst deﬁne constants

Original: [Equation]
Revised: [Equation]

Original: for t

Original: [Equation]
Revised: [Equation]

Original: 1 and ˜

Original: [Equation]
Revised: [Equation]

Revised: t for t > 1 and ˜ β 1 = β 1 .

Original: Then, a noticeable property of diffusion model is
Revised: Then, a noticeable property of diffusion model is

Original: [Equation]
Revised: [Equation]

Original: thus one can directly sample x t given x 0 (see Appendix
Revised: thus one can directly sample x t given x 0 (see Appendix

Original: B.1 for derivation).
Revised: B.1 for derivation).

Original: Furthermore, one may parameterize
Revised: Furthermore, one may parameterize

Original: [Equation]
Revised: [Equation]

Original: , where (cid:15) θ is a neural network taking x t and the diffusion-step t as inputs.
Revised: , where (cid:15) θ is a neural network taking x t and the diffusion-step t as inputs.

Original: In addition, σ t is simply parameterized as ˜ β
Revised: In addition, σ t is simply parameterized as ˜ β

Original: 12 t .
Revised: 12 t .

Original: Ho et al.
Revised: Ho et al.

Original: (2020) show that minimizing the following unweighted variant of the ELBO leads to higher generation quality:
Revised: (2020) show that minimizing the following unweighted variant of the ELBO leads to higher generation quality:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) ∼ N (0 , I ) , x 0 ∼ q data , t is uniformly taken from 1 , · · · , T , and x t =
Revised: where (cid:15) ∼ N (0 , I ) , x 0 ∼ q data , t is uniformly taken from 1 , · · · , T , and x t =

Original: √ ¯ α t · x 0 + √ 1 − ¯ α t · (cid:15) from Eq.
Revised: √ ¯ α t · x 0 + √ 1 − ¯ α t · (cid:15) from Eq.

Original: 3. FastDPM: A Uniﬁed Framework for Fast Sampling in Diffusion Models
Revised: 3. FastDPM: A Uniﬁed Framework for Fast Sampling in Diffusion Models

Original: In order to achieve high-ﬁdelity synthesis, the number of diffusion steps T in DDPM is set to be very large so that q ( x T | x 0 ) is close to p latent .
Revised: In order to achieve high-ﬁdelity synthesis, the number of diffusion steps T in DDPM is set to be very large so that q ( x T | x 0 ) is close to p latent .

Original: For example, T = 1000 in image synthesis (Ho et al., 2020) and T = 200 in audio synthesis (Kong et al., 2020b).
Revised: For example, T = 1000 in image synthesis (Ho et al., 2020) and T = 200 in audio synthesis (Kong et al., 2020b).

Original: Then, sampling from DDPM needs running through the network (cid:15) θ for as many as T times, which can be very slow.
Revised: Then, sampling from DDPM needs running through the network (cid:15) θ for as many as T times, which can be very slow.

Original: In this section, we propose FastDPM, which approximates the pretrained DDPM via much shorter diffusion and reverse processes of length
Revised: In this section, we propose FastDPM, which approximates the pretrained DDPM via much shorter diffusion and reverse processes of length

Original: S (cid:28) T , thus it can generate a sample by only running the network S times.
Revised: S (cid:28) T , thus it can generate a sample by only running the network S times.

Original: The core idea of FastDPM is to: i) generalize discrete diffusion steps to continuous diffusion steps and, then ii) design a bijective mapping between continuous diffusion steps and continuous noise levels, where these noise levels indicate the amount of noise in data.
Revised: The core idea of FastDPM is to: i) generalize discrete diffusion steps to continuous diffusion steps and, then ii) design a bijective mapping between continuous diffusion steps and continuous noise levels, where these noise levels indicate the amount of noise in data.

Original: Finally, we use this bijective mapping to construct an approximate diffusion process and an approximate reverse process, respectively.
Revised: Finally, we use this bijective mapping to construct an approximate diffusion process and an approximate reverse process, respectively.

Original: 3.1. Bijective mapping between Continuous Diffusion Steps and Noise Levels
Revised: 3.1. Bijective mapping between Continuous Diffusion Steps and Noise Levels

Original: In this section, we generalize discrete (integer) diffusion steps to continuous (real-valued) diffusion steps.
Revised: In this section, we generalize discrete (integer) diffusion steps to continuous (real-valued) diffusion steps.

Original: Then, we introduce a bijective mapping R and T
Revised: Then, we introduce a bijective mapping R and T

Original: =
Revised: =

Original: R − 1 between continuous diffusion steps t and noise levels r .
Revised: R − 1 between continuous diffusion steps t and noise levels r .

Original: Deﬁne R .
Revised: Deﬁne R .

Original: We start with an integer diffusion step t .
Revised: We start with an integer diffusion step t .

Original: From
Revised: From

Original: Eq.
Revised: Eq.

Original: (1), one can observe x 
Revised: (1), one can observe x t = √ ¯ α t · x

Original: [Equation]
Revised: [Equation]

Original:  where (cid:15) ∼ N (0 , I ) , thus sampling x t given x 0 is equivalent to adding a Gaussian noise to x 0 .
Revised: + [Equation] (cid:15) where (cid:15) ∼ N (0 , I ) , thus sampling x t given x 0 is equivalent to adding a Gaussian noise to x 0 .

Original: Based on this observation, we deﬁne the noise level at step t as R ( t ) =
Revised: Based on this observation, we deﬁne the noise level at step t as R ( t ) =

Original: √ ¯ α t , which means x t is composed of R ( t ) fraction of the data x 0 and (1 − R ( t )) fraction of white noise.
Revised: √ ¯ α t , which means x t is composed of R ( t ) fraction of the data x 0 and (1 − R ( t )) fraction of white noise.

Original: For example, R ( t ) = 0 means no noise and R ( t ) = 1 means pure white noise.
Revised: For example, R ( t ) = 0 means no noise and R ( t ) = 1 means pure white noise.

Original: Next, we extend the domain of R to real values.
Revised: Next, we extend the domain of R to real values.

Original: Assume that the variance schedule { β t } Tt =1 is linear: β i =
Revised: Assume that the variance schedule { β t } Tt = is linear: β i =

Original: β 1 + ( i − 1)∆ β , where ∆ β = β T − β 1 T − 1 (Ho et al., 2020).
Revised: β 1 + ( i − 1)∆ β , where ∆ β = β T − β 1 T −(Ho et al., 2020).

Original: We further deﬁne an auxiliary constant ˆ β = 1 − β 1 ∆ β , which is (cid:29) T assuming that β T (cid:28) 1 . 0 .
Revised: We further deﬁne an auxiliary constant ˆ β = 1 − β 1 ∆ β , which is (cid:29) T assuming that β T (cid:28) 1 . 0 .

Original: Then, we have ¯ α t =
Revised: Then, we have ¯ α t =

Original: (∆ β ) t Γ (cid:16) ˆ β + 1 (cid:17) / Γ (cid:16) ˆ β − t + 1 (cid:17) .
Revised: (∆ β ) t Γ (cid:16) ˆ β + 1 (cid:17) / Γ (cid:16) ˆ β − t + 1 (cid:17) (see Appendix B.2).

Original: Because the Gamma function Γ is well-deﬁned on (0 , ∞ ) , it gives rise to a natural extension of ¯ α t for continuous diffusion steps t .
Revised: Because the Gamma function Γ is well-deﬁned on (0 , ∞ ) , it gives rise to a natural extension of ¯ α t for continuous diffusion steps t .

Original: As a result, for t ∈ [0 , ˆ β ) , we deﬁne the noise level at t as:
Revised: As a result, for t ∈ [0 , ˆ β ) , we deﬁne the noise level at t as:

Original: [Equation]
Revised: [Equation]

Original: Deﬁne T .
Revised: Deﬁne T .

Original: For any noise level r ∈ (0 , 1) , its corresponding (continuous) diffusion step, T ( r ) , is deﬁned by inverting 
Revised: For any noise level r ∈ (0 , 1) , its corresponding (continuous) diffusion step, T ( r ) , is deﬁned by inverting R :

Original: R : T ( r ) =
Revised: T ( r ) =

Original: R − 1 ( r ) .
Revised: R − 1 ( r ) .

Original: Given a noise level r = R ( t ) , we numerically solve t = T ( r ) by applying a binary search based on Eq.
Revised: Given a noise level r = R ( t ) , we numerically solve t = T ( r ) by applying a binary search based on Eq.

Original: We have T ( r ) ∈ [ t, t +1] for r ∈ [ √ ¯ α t [-+
-] , √ ¯ α t ] , and this provides a good initialization to the binary search algorithm.
Revised: We have T ( r ) ∈ [ t, t + 1] for r ∈ [ √ ¯ α t [-+
-]+1 , √ ¯ α t ] , and this provides a good initialization to the binary search algorithm.

Original: Experimentally, we ﬁnd the binary search algorithm converges in no more than 20 iterations.
Revised: Experimentally, we ﬁnd the binary search algorithm converges in no more than 20 iterations.

Original: 3.2. Approximate the Diffusion Process
Revised: 3.2. Approximate the Diffusion Process

Original: Let ˆ x 0 ∼ p data .
Revised: Let ˆ x 0 ∼ p data .

Original: Given a sequence of noise levels 1 > r 1 > r 2 > · · · > r S > 0 , we aim to construct each step in the approximate diffusion process as ˆ x s ∼ N (ˆ x s ; r s ˆ x 0 , (1 − r 2 s ) I ) .
Revised: Given a sequence of noise levels 1 > r 1 > r 2 > · · · > r S > 0 , we aim to construct each step in the approximate diffusion process as ˆ x s ∼ N (ˆ x s ; r s ˆ x 0 , (1 − r 2 s ) I ) .

Original: To achieve this goal, we deﬁne γ s =
Revised: To achieve this goal, we deﬁne γ s =

Original: r 2 s /r 2 s − 1 , compute the corresponding variances as η s = 1 − γ s = 1 − r 2 s /r 2 s − 1 , and then deﬁne the transition probability in the approximate diffusion process as[Equation] 1 − [Equation]
Revised: r 2 s /r 2 s − 1 , compute the corresponding variances as η s = 1 − γ s = 1 − r 2 s /r 2 s − 1 , and then deﬁne the transition probability in the approximate diffusion process as

Original: s r

Original: [Equation]
Revised: [Equation]

Original: s

Original: [Equation]

Original: (cid:19)

Original: I (cid:19) .

Original: One can see this by rewriting Eq.
Revised: One can see this by rewriting Eq.

Original: (1): η s corresponds to  t = 1 − α t , γ s corresponds to α t , and r s corresponds to 
Revised: (1): η s corresponds to β t = 1 − α t , γ s corresponds to α t , and r s corresponds to E.g., β T = 0 . 02 in Ho

Original: √ ¯ α t .
Revised: √ ¯ α t .

Original: We then propose the following two ways to schedule the noise levels { r s } Ss =1 .
Revised: We then propose the following two ways to schedule the noise levels { r s } Ss =1 .

Original: Noise levels from variances (VAR).
Revised: Noise levels from variances (VAR).

Original: We start from the variance schedule { η s } Ss =1 .
Revised: We start from the variance schedule { η s } Ss =1 .

Original: Next, we compute γ s = 1 − η s and ¯ γ s = (cid:81) si =1 γ i .
Revised: Next, we compute γ s = 1 − η s and ¯ γ s = (cid:81) si =1 γ i .

Original: The noise level at step s is r s = √ ¯ γ s .
Revised: The noise level at step s is r s = √ ¯ γ s .

Original: Noise levels from steps (STEP).
Revised: Noise levels from steps (STEP).

Original: We start from a subset of diffusion steps { τ s } Ss − 1 in { 1 , · · · , T } .
Revised: We start from a subset of diffusion steps { τ s } Ss − 1 in { 1 , · · · , T } .

Original: Then, the noise level at step s is r s =
Revised: Then, the noise level at step s is r s =

Original: R ( τ s ) =
Revised: R ( τ s ) =

Original: √ ¯ α τ s .
Revised: √ ¯ α τ s .

Original: When η s = 1 − ¯ α τ s / ¯ α τ s − 1 , we have ¯ γ s =
Revised: When η s = 1 − ¯ α τ s / ¯ α τ s − 1 , we have ¯ γ s =

Original: ¯ α τ s .
Revised: ¯ α τ s .

Original: Therefore, noise levels from steps can be regarded as a special case of noise levels from variances.
Revised: Therefore, noise levels from steps can be regarded as a special case of noise levels from variances.

Original: 3.3. Approximate the Reverse Process
Revised: 3.3. Approximate the Reverse Process

Original: Given the same sequence of noise levels in Section 3.2, we aim to approximate the reverse process in the original
Revised: Given the same sequence of noise levels in Section 3.2, we aim to approximate the reverse process in the original

Original: DDPM.
Revised: DDPM.

Original: To achieve this goal, we regard the model (cid:15) θ as being trained on variances { η s } Ss =1 instead of the original { β t } Tt =1 .
Revised: To achieve this goal, we regard the model (cid:15) θ as being trained on variances { η s } Ss =1 instead of the original { β t } Tt =1 .

Original: Then, the transition probability in the approximate reverse process is
Revised: Then, the transition probability in the approximate reverse process is

Original: [Equation]
Revised: [Equation]

Original: where ˆ 
Revised: where ˆ µ (ˆ x s , s ) =

Original: [Equation]
Revised: [Equation]

Original: ˆ x
Revised: − η

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: =
Revised: 1 −

Original: [Equation]
Revised: [Equation]

Original: s  − 1 1 − ¯ γ s η s for s > 1 and ˜ η 1 = η 1 .
Revised: s (cid:15) θ (ˆ x s , T ( r s )) (cid:17) , ˜ η s = 1 − ¯ γ s − 1 1 − ¯ γ s η s for s > 1 and ˜ η 1 = η 1 .

Original: ˜
Revised: ˜

Original: η s corresponds to the ˜ β t = σ 2 t term.
Revised: η s corresponds to the ˜ β t = σ 2 t term.

Original: There are two ways to sample from the approximate reverse process in Eq.
Revised: There are two ways to sample from the approximate reverse process in Eq.

Original: Let every ˆ (cid:15) s be i.i.d.
Revised: Let every ˆ (cid:15) s be i.i.d.

Original: standard Gaussians for 1 ≤ s ≤ S .
Revised: standard Gaussians for 1 ≤ s ≤ S .

Original: DDPM reverse process (DDPM-rev).
Revised: DDPM reverse process (DDPM-rev).

Original: The sampling procedure based on the DDPM reverse process is based on
Revised: The sampling procedure based on the DDPM reverse process is based on

Original: Eq.
Revised: Eq.

Original: (5): that is, to ﬁrst sample ˆ x S ∼ p latent and then sampleˆ x s − 1 = ˆ µ (ˆ x s , s ) + √ ˜
Revised: (5): that is, to ﬁrst sample ˆ x S ∼ p latent and then sample

Original: η s ˆ (cid:15) s .
Revised: [Equation]

Original: DDIM reverse process (DDIM-rev).
Revised: DDIM reverse process (DDIM-rev).

Original: Let κ ∈ [0 , 1] be a hyperparameter.
Revised: Let κ ∈ [0 , 1] be a hyperparameter.

Original: Then, the sampling procedure based on
Revised: Then, the sampling procedure based on

Original: DDIM (Song et al., 2020a) is to ﬁrst sample ˆ x S ∼ p latent
Revised: DDIM (Song et al., 2020a) is to ﬁrst sample ˆ x S ∼ p latent

Original: and then sample ˆ x s− 1 = √ ¯ γ s − 1 (cid:16) ˆ x s −√ 1 − ¯ [Equation]
Revised: and then sample ˆ x s

Original: s (cid:15) θ (ˆ x s , T ( r s

Original: [Equation]
Revised: [Equation]

Original: s (cid:17) + (cid:112) [Equation] s , T [Equation]
Revised: s .

Original: s )) + κ √ ˜ η s ˆ (cid:15) s . When κ = 1 , it is exactly DDPM-rev (see Appendix B.2 for derivation).
Revised: When κ = 1 , it is exactly DDPM-rev (see Appendix B.3 for derivation).

Original: 3.4. Connections with Previous Methods
Revised: 3.4. Connections with Previous Methods

Original: The DDIM (Song et al., 2020a) method is equivalent to
Revised: The DDIM (Song et al., 2020a) method is equivalent to

Original: STEP + DDIM-rev in FastDPM.
Revised: STEP + DDIM-rev in FastDPM.

Original: The fast sampling algorithm by DiffWave (Kong et al., 2020b) is related to VAR + DDPM-rev in FastDPM.
Revised: The fast sampling algorithm by DiffWave (Kong et al., 2020b) is related to VAR + DDPM-rev in FastDPM.

Original: Compared with DiffWave, FastDPM offers an automatic way to select variances in different settings and a more natural way to compute noise levels.
Revised: Compared with DiffWave, FastDPM offers an automatic way to select variances in different settings and a more natural way to compute noise levels.

Original: 4. Experiments
Revised: 4. Experiments

Original: In this section, we aim to answer the following two questions for FastDPM.
Revised: In this section, we aim to answer the following two questions for FastDPM.

Original: (1) Which approximate diffusion process,
Revised: (1) Which approximate diffusion process,

Original: VAR or STEP, is better?
Revised: VAR or STEP, is better?

Original: And (2) which approximate reverse process, DDPM-rev or DDIM-rev, is better?
Revised: And (2) which approximate reverse process, DDPM-rev or DDIM-rev, is better?

Original: We investigate these questions by conducting extensive experiments in both image and audio domains.
Revised: We investigate these questions by conducting extensive experiments in both image and audio domains.

Original: We conduct unconditional image generation experiments on CIFAR-10 (Krizhevsky et al., 2009), CelebA (Liu et al., 2015), and LSUN-bedroom (Yu et al., 2015), unconditional and class-conditional audio synthesis experiments on the
Revised: We conduct unconditional image generation experiments on CIFAR-10 (Krizhevsky et al., 2009), CelebA (Liu et al., 2015), and LSUN-bedroom (Yu et al., 2015), unconditional and class-conditional audio synthesis experiments on the

Original: Speech Commands 0-9 (SC09) dataset (Warden, 2018), and neural vocoding experiments (audio synthesis conditioned on mel spectrogram) on the LJSpeech dataset (Ito, 2017).
Revised: Speech Commands 0-9 (SC09) dataset (Warden, 2018), and neural vocoding experiments (audio synthesis conditioned on mel spectrogram) on the LJSpeech dataset (Ito, 2017).

Original: We use pretrained models in all experiments (Ho et al., 2020;
Revised: We use pretrained models in all experiments (Ho et al., 2020;

Original: Esser, 2020; Song et al., 2020a; Kong et al., 2020b).
Revised: Esser, 2020; Song et al., 2020a; Kong et al., 2020b).

Original: We use Fréchet Inception Distance (FID) (Heusel et al., 2017;
Revised: We use Fréchet Inception Distance (FID) (Heusel et al., 2017;

Original: Lang, 2020), Inception Score (IS) (Salimans et al., 2016), and the crowdMOS tookit (Ribeiro et al., 2011) to evaluate generated samples.
Revised: Lang, 2020), Inception Score (IS) (Salimans et al., 2016), and the crowdMOS tookit (Ribeiro et al., 2011) to evaluate generated samples.

Original: Details of experimental setup can be found in Appendix C. Results can be found in Appendix
Revised: Details of experimental setup can be found in Appendix C. Results can be found in Appendix

Original: D. Generated samples can be found in Appendix E and the demo website.
Revised: D. Generated samples can be found in Appendix E and the demo website.

Original: 4.1. Observations and Insights
Revised: 4.1. Observations and Insights

Original: We have the following observations and insights according to the above experimental results.
Revised: We have the following observations and insights according to the above experimental results.

Original: VAR marginally outperforms STEP for small S .
Revised: VAR marginally outperforms STEP for small S .

Original: In the above experiments, the two approximate diffusion processes (STEP and VAR) generally match performances of each other.
Revised: In the above experiments, the two approximate diffusion processes (STEP and VAR) generally match performances of each other.

Original: On CIFAR-10, VAR outperforms STEP when S = 10 , and STEP slightly outperforms VAR when S ≥ 20 .
Revised: On CIFAR-10, VAR outperforms STEP when S = 10 , and STEP slightly outperforms VAR when S ≥ 20 .

Original: On
Revised: On

Original: CelebA, VAR slightly outperforms STEP when S ≤ 20 , and they have similar results when S ≥ 50 .
Revised: CelebA, VAR slightly outperforms STEP when S ≤ 20 , and they have similar results when S ≥ 50 .

Original: On LSUNbedroom, VAR slightly outperforms STEP when S ≤ 50 , and STEP slightly outperforms VAR when S = 100 .
Revised: On LSUNbedroom, VAR slightly outperforms STEP when S ≤ 50 , and STEP slightly outperforms VAR when S = 100 .

Original: On
Revised: On

Original: SC09, VAR slightly outperforms STEP in most cases.
Revised: SC09, VAR slightly outperforms STEP in most cases.

Original: On
Revised: On

Original: LJSpeech, VAR slightly outperforms STEP when S = 5 .
Revised: LJSpeech, VAR slightly outperforms STEP when S = 5 .

Original: Based on these results, we conclude that VAR marginally outperforms STEP for small S .
Revised: Based on these results, we conclude that VAR marginally outperforms STEP for small S .

Original: Different reverse processes dominate in different domains.
Revised: Different reverse processes dominate in different domains.

Original: In the above experiments, the difference between
Revised: In the above experiments, the difference between

Original: DDPM and DDIM reverse processes is very clear.
Revised: DDPM and DDIM reverse processes is very clear.

Original: In image generation tasks, DDIM-rev signiﬁcantly outperforms
Revised: In image generation tasks, DDIM-rev signiﬁcantly outperforms

Original: DDPM-rev except for the S = 100 case in the LSUNbedroom experiment.
Revised: DDPM-rev except for the S = 100 case in the LSUNbedroom experiment.

Original: When we reduce κ from 1 . 0 to 0 .(see
Revised: When we reduce κ from 1 . 0 to 0 .(see

Original: Table 1), the quality of generated samples consistently improves.
Revised: Table 1), the quality of generated samples consistently improves.

Original: In contrast, in audio synthesis tasks, DDPM-rev signiﬁcantly outperforms DDIM-rev.
Revised: In contrast, in audio synthesis tasks, DDPM-rev signiﬁcantly outperforms DDIM-rev.

Original: When we increase κ from 0 . 0 to 1 . 

Revised: When we increase κ from 0 . 0 to 1 .

Original: (see Table 4), the quality of generated samples consistently improves.
Revised: (see Table 4), the quality of generated samples consistently improves.

Original: This can also be observed from Figure 8: DDIM produces very noisy utterances while
Revised: This can also be observed from Figure 8: DDIM produces very noisy utterances while

Original: DDPM produces very clean utterances.
Revised: DDPM produces very clean utterances.

Original: The results indicate that in the image domain, DDIMrev produces better quality whereas in the audio domain,
Revised: The results indicate that in the image domain, DDIMrev produces better quality whereas in the audio domain,

Original: DDPM-rev produces better quality.
Revised: DDPM-rev produces better quality.

Original: We speculate the reason behind the difference is that in the audio domain, waveforms naturally exhibit signiﬁcant amount of stochasticity.
Revised: We speculate the reason behind the difference is that in the audio domain, waveforms naturally exhibit signiﬁcant amount of stochasticity.

Original: The
Revised: The

Original: DDPM reverse process offers much stochasticity because at each reverse step s , ˆ x s − 1 is sampled from a Gaussian distribution.
Revised: DDPM reverse process offers much stochasticity because at each reverse step s , ˆ x s − 1 is sampled from a Gaussian distribution.

Original: However, the DDIM reverse process ( κ = 0 . 0 ) is a deterministic mapping from latents to data, so it leads to degrade quality in the audio domain.
Revised: However, the DDIM reverse process ( κ = 0 . 0 ) is a deterministic mapping from latents to data, so it leads to degrade quality in the audio domain.

Original: This hypothesis is also aligned with previous result that the ﬂow-based model with deterministic mapping was unable to generate intelligible speech unconditionally on SC09 (Ping, 2021).
Revised: This hypothesis is also aligned with previous result that the ﬂow-based model with deterministic mapping was unable to generate intelligible speech unconditionally on SC09 (Ping, 2021).

Original: The amount of conditional information affects the choice of reverse processes.
Revised: The amount of conditional information affects the choice of reverse processes.

Original: In audio synthesis experiments, we ﬁnd the amount of conditional information affects the generation quality of methods with different reverse processes.
Revised: In audio synthesis experiments, we ﬁnd the amount of conditional information affects the generation quality of FastDPM with different reverse processes.

Original: In the unconditional generation experiment on SC09, DDPM-rev (which corresponds to κ = 1 . 0 ) has the best results.
Revised: In the unconditional generation experiment on SC09, DDPM-rev (i.e. κ = 1 . 0 ) has the best results.

Original: When there is slightly more conditional information in the class-conditional generation experiment on SC09, DDIM-rev with κ = 0 .
Revised: When there is slightly more conditional information in the class-conditional generation experiment on SC09, DDIMrev with κ = 0 .

Original: 5 has the best results and slightly outperforms DDPM-rev.
Revised: 5 has the best results and slightly outperforms DDPM-rev.

Original: In both experiments DDIM-rev with κ = 0 . 0 has much worse results.
Revised: In both experiments DDIM-rev with κ = 0 . 0 has much worse results.

Original: When there is much more conditional information (mel spectrogram) in the neural vocoding experiments on LJSpeech, DDPM-rev is still better than DDIM-rev, but the difference between these two methods is reduced.
Revised: When there is much more conditional information (mel spectrogram) in the neural vocoding experiments on LJSpeech, DDPM-rev is still better than DDIM-rev, but the difference between these two methods is reduced.

Original: We speculate that adding conditional information reduces the amount of stochasticity required.
Revised: We speculate that adding conditional information reduces the amount of stochasticity required.

Original: When there is no conditional information, we need a large amount of stochasticity ( κ = 1 . 0 ); when there is weak class information, we need moderate stochasticity ( κ = 0 . 5 ); and when there is strong mel-spectrogram information, even having no stochasticity ( κ = 0 . 0 ) is able to generate reasonable samples.
Revised: When there is no conditional information, we need maximum stochasticity ( κ = 1 . 0 ); with weak class information, we need moderate stochasticity ( κ = 0 . 5 ); and with strong mel-spectrogram information, even having no stochasticity ( κ = 0 . 0 ) is able to generate reasonable samples.

Original: 5. Conclusion
Revised: 5. Conclusion

Original: Diffusion models are a class of powerful deep generative models that produce superior quality samples on various generation tasks.
Revised: Diffusion models are a class of powerful deep generative models that produce superior quality samples on various generation tasks.

Original: In this paper, we introduce FastDPM, a uniﬁed framework for fast sampling in diffusion models without retraining.
Revised: In this paper, we introduce FastDPM, a uniﬁed framework for fast sampling in diffusion models without retraining.

Original: FastDPM generalizes prior methods and provides more ﬂexibility.
Revised: FastDPM generalizes prior methods and provides more ﬂexibility.

Original: We extensively evaluate and analyze FastDPM in image and audio generation tasks.
Revised: We extensively evaluate and analyze FastDPM in image and audio generation tasks.

Original: One limitation of FastDPM is that when S is small, there is still quality degradation compared to the original DDPM.
Revised: One limitation of FastDPM is that when S is small, there is still quality degradation compared to the original DDPM.

Original: We plan to study algorithms offering higher quality for extremely small S in future.
Revised: We plan to study algorithms offering higher quality for extremely small S in future.

Original: A. Related Work
Revised: A. Related Work

Original: Diffusion models are a class of powerful deep generative models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Goyal et al., 2017), which have received a lot of attention recently.
Revised: Diffusion models are a class of powerful deep generative models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Goyal et al., 2017), which have received a lot of attention recently.

Original: These models have been applied to various domains, including image generation (Ho et al., 2020; Dhariwal & Nichol, 2021), audio synthesis (Kong et al., 2020b;
Revised: These models have been applied to various domains, including image generation (Ho et al., 2020; Dhariwal & Nichol, 2021), audio synthesis (Kong et al., 2020b;

Original: Chen et al., 2020; Okamoto et al., 2021), image or audio super-resolution (Li et al., 2021; Lee & Han, 2021), text-to-speech (Jeong et al., 2021; Popov et al., 2021), music synthesis (Liu et al., 2021; Mittal et al., 2021), 3-D point cloud generation (Luo & Hu, 2021;
Revised: Chen et al., 2020; Okamoto et al., 2021), image or audio super-resolution (Li et al., 2021; Lee & Han, 2021), text-to-speech (Jeong et al., 2021; Popov et al., 2021), music synthesis (Liu et al., 2021; Mittal et al., 2021), 3-D point cloud generation (Luo & Hu, 2021;

Original: Zhou et al., 2021), and language models (Hoogeboom et al., 2021).
Revised: Zhou et al., 2021), and language models (Hoogeboom et al., 2021).

Original: Diffusion models are connected with scored-based models (Song & Ermon, 2019; 2020; Song et al., 2020b), and there have been a series of research extending and improving diffusion models (Song et al., 2020b;
Revised: Diffusion models are connected with scored-based models (Song & Ermon, 2019; 2020; Song et al., 2020b), and there have been a series of research extending and improving diffusion models (Song et al., 2020b;

Original: Gao et al., 2020; Dhariwal & Nichol, 2021; San-Roman et al., 2021;
Revised: Gao et al., 2020; Dhariwal & Nichol, 2021; San-Roman et al., 2021;

Original: Meng et al., 2021).
Revised: Meng et al., 2021).

Original: There are two families of methods aiming for accelerating diffusion models at synthesis, which reduce the length of the reverse process from T to a much smaller S .
Revised: There are two families of methods aiming for accelerating diffusion models at synthesis, which reduce the length of the reverse process from T to a much smaller S .

Original: One family of methods tackle this problem at training.
Revised: One family of methods tackle this problem at training.

Original: They retrain the network conditioned on continuous noise levels instead of discrete diffusion steps (Song & Ermon, 2019; Chen et al., 2020;
Revised: They retrain the network conditioned on continuous noise levels instead of discrete diffusion steps (Song & Ermon, 2019; Chen et al., 2020;

Original: Okamoto et al., 2021; San-Roman et al., 2021).
Revised: Okamoto et al., 2021; San-Roman et al., 2021).

Original: Assuming that the corresponding network is able to predict added noise at any noise level, we can carefully choose only S (cid:28) T noise levels and construct a short reverse process just based on them.
Revised: Assuming that the corresponding network is able to predict added noise at any noise level, we can carefully choose only S (cid:28) T noise levels and construct a short reverse process just based on them.

Original: San-Roman et al.
Revised: San-Roman et al.

Original: present a learning scheme that can step-by-step adjust those noise level parameters, for any given number of steps S .
Revised: present a learning scheme that can step-by-step adjust those noise level parameters, for any given number of steps S .

Original: Another family of methods aim to directly approximate the original reverse process within the pretrained DDPM conditioned on discrete steps.
Revised: Another family of methods aim to directly approximate the original reverse process within the pretrained DDPM conditioned on discrete steps.

Original: In other words, no retraining is needed.
Revised: In other words, no retraining is needed.

Original: Song et al.
Revised: Song et al.

Original: (2020a) introduce denoising diffusion implicit models (DDIM), which contain non-Markovian processes that lead to an equivalent training objective as DDPM.
Revised: (2020a) introduce denoising diffusion implicit models (DDIM), which contain non-Markovian processes that lead to an equivalent training objective as DDPM.

Original: These non-Markovian processes naturally permit "jumping steps", or formally, using a subset of steps to form a short reverse process.
Revised: These non-Markovian processes naturally permit "jumping steps", or formally, using a subset of steps to form a short reverse process.

Original: However, compared to using continuous noise levels, selecting discrete steps offers less ﬂexibility.
Revised: However, compared to using continuous noise levels, selecting discrete steps offers less ﬂexibility.

Original: Kong et al.
Revised: Kong et al.

Original: (2020b) introduce a fast sampling algorithm by interpolating steps according to corresponding noise levels.
Revised: (2020b) introduce a fast sampling algorithm by interpolating steps according to corresponding noise levels.

Original: This can be seen as an attempt to map continuous noise levels to discrete diffusion steps.
Revised: This can be seen as an attempt to map continuous noise levels to discrete diffusion steps.

Original: However, it lacks both theoretical justiﬁcation for the interpolation and extensive empirical studies.
Revised: However, it lacks both theoretical justiﬁcation for the interpolation and extensive empirical studies.

Original: In this paper, we propose FastDPM, a method that approximates the original DDPM model.
Revised: In this paper, we propose FastDPM, a method that approximates the original DDPM model.

Original: FastDPM constructs a bijective mapping between (continuous) diffusion steps and continuous noise levels.
Revised: FastDPM constructs a bijective mapping between (continuous) diffusion steps and continuous noise levels.

Original: This allows us to take advantage of the ﬂexibility of using these continuous noise levels.
Revised: This allows us to take advantage of the ﬂexibility of using these continuous noise levels.

Original: FastDPM generalizes Kong et al.
Revised: FastDPM generalizes Kong et al.

Original: (2020b)
Revised: (2020b)

Original: by using Gamma functions to compute noise levels, which naturally extends from discrete domain to continuous domain.
Revised: by using Gamma functions to compute noise levels, which naturally extends from discrete domain to continuous domain.

Original: FastDPM generalizes Song et al.
Revised: FastDPM generalizes Song et al.

Original: (2020a)
Revised: (2020a)

Original: by providing a special set of noise levels that exactly correspond to integer steps.
Revised: by providing a special set of noise levels that exactly correspond to integer steps.

Original: B. Derivations B.1. Derivation of q ( x t | x 0 )
Revised: B. Derivations B.1. Derivation of q ( x t | x 0 )

Original: According to the deﬁnition of diffusion process, we have
Revised: According to the deﬁnition of diffusion process, we have

Original: [Equation]
Revised: [Equation]

Original: (6) where each (cid:15) t is an i.i.d.
Revised: 6) where each (cid:15) t is an i.i.d.

Original: standard Gaussian.
Revised: standard Gaussian.

Original: Then, by recursion, we have
Revised: Then, by recursion, we have

Original: [Equation]
Revised: [Equation]

Original: As a result, q ( x t | x 0 ) is still Gaussian.
Revised: As a result, q ( x t | x 0 ) is still Gaussian.

Original: Its mean vector is √ ¯ α t x 0 , and its covariance matrix is ( α t α t − 1 · · · α 2 β 1 + · · · + α t β t − 1 + β t ) I =
Revised: Its mean vector is √ ¯ α t x 0 , and its covariance matrix is ( α t α t − 1 · · · α 2 β 1 + · · · + α t β t − 1 + β t ) I =

Original: (1 − ¯ α t ) I .
Revised: (1 − ¯ α t ) I .

Original: Formally, we have
Revised: Formally, we have

Revised: [Equation]

Revised: [Equation]

Revised: B.2. Derivation of ¯ α t

Original: [Equation]
Revised: [Equation]

Original: B.2. Derivation of DDIM ( κ = 1 )
Revised: B.3. Derivation of DDIM ( κ = 1 )

Original: When κ =
Revised: When κ =

Original: [Equation]
Revised: [Equation]

Original: the coefﬁcient of the (cid:15) θ term in the DDIM reverse process is
Revised: the coefﬁcient of the (cid:15) θ term in the DDIM reverse process is

Original: [Equation]
Revised: [Equation]

Original: C. Detailed Experimental Setup
Revised: C. Detailed Experimental Setup

Original: Image datasets.
Revised: Image datasets.

Original: We conduct unconditional image generation experiments on three datasets: CIFAR-10 (50k object images of resolution 32 × 32 (Krizhevsky et al., 2009)), CelebA ( ∼ 163k face images of resolution 64 × 64 (Liu et al., 2015)), and
Revised: We conduct unconditional image generation experiments on three datasets: CIFAR-10 (50k object images of resolution 32 × 32 (Krizhevsky et al., 2009)), CelebA ( ∼ 163k face images of resolution 64 × 64 (Liu et al., 2015)), and

Original: LSUN-bedroom ( ∼ 3M bedroom images of resolution 256 × 256 (Yu et al., 2015)).
Revised: LSUN-bedroom ( ∼ 3M bedroom images of resolution 256 × 256 (Yu et al., 2015)).

Original: Audio datasets.
Revised: Audio datasets.

Original: We conduct unconditional and class-conditional audio synthesis experiments on the Speech Commands 0-9 (SC09) dataset, the spoken digit subset of the full Speech Commands dataset (Warden, 2018).
Revised: We conduct unconditional and class-conditional audio synthesis experiments on the Speech Commands 0-9 (SC09) dataset, the spoken digit subset of the full Speech Commands dataset (Warden, 2018).

Original: SC09 contains ∼ 31k one-second long utterances of ten classes (0 through 9) with a sampling rate of 16kHz.
Revised: SC09 contains ∼ 31k one-second long utterances of ten classes (0 through 9) with a sampling rate of 16kHz.

Original: We conduct neural vocoding experiments (audio synthesis conditioned on mel spectrogram) on the LJSpeech dataset (Ito, 2017).
Revised: We conduct neural vocoding experiments (audio synthesis conditioned on mel spectrogram) on the LJSpeech dataset (Ito, 2017).

Original: It contains ∼ 24 hours of audio ( ∼ 13k utterances from a female speaker) recorded in home environment with a sampling rate of 22.05kHz.
Revised: It contains ∼ 24 hours of audio ( ∼ 13k utterances from a female speaker) recorded in home environment with a sampling rate of 22.05kHz.

Original: Models.
Revised: Models.

Original: In all experiments, we use pretrained checkpoints in prior works.
Revised: In all experiments, we use pretrained checkpoints in prior works.

Original: In detail, the pretrained models for CIFARand LSUN-bedroom are taken from DDPM (Ho et al., 2020; Esser, 2020), the pretrained model for CelebA is taken from
Revised: In detail, the pretrained models for CIFARand LSUN-bedroom are taken from DDPM (Ho et al., 2020; Esser, 2020), the pretrained model for CelebA is taken from

Original: DDIM (Song et al., 2020a).
Revised: DDIM (Song et al., 2020a).

Original: In these models, T is 1000 .
Revised: In these models, T is 1000 .

Original: The pretrained models for SC09 and LJSpeech are taken from
Revised: The pretrained models for SC09 and LJSpeech are taken from

Original: DiffWave (Kong et al., 2020b).
Revised: DiffWave (Kong et al., 2020b).

Original: In these models, T is 200 .
Revised: In these models, T is 200 .

Original: In all models, β 1 = 10 − 4 , β T = 2 × 10 − 2 , and all β t ’s are linearly interpolated between β 1 and β T .
Revised: In all models, β 1 = 10 − 4 , β T = 2 × 10 − 2 , and all β t ’s are linearly interpolated between β 1 and β T .

Original: Noise level schedules.
Revised: Noise level schedules.

Original: For each of the approximate diffusion process in Section 3.2, we examine two schedules: linear and quadratic.
Revised: For each of the approximate diffusion process in Section 3.2, we examine two schedules: linear and quadratic.

Original: For noise levels { η s } S s =1 from variances, the two schedules are:
Revised: For noise levels { η s } S s =1 from variances, the two schedules are:

Original: • Linear (VAR):
Revised: • Linear (VAR):

Original: η s = (1 + cs ) η 0 .
Revised: η s = (1 + cs ) η 0 .

Original: Quadratic (VAR):
Revised: Quadratic (VAR):

Original: η s = (1 + cs ) 2 η 0 .
Revised: η s = ( 1 + cs ) 2 η 0 .

Original: We let η 0 = β 0 and the constant c satisfy (cid:81) Ss =1 (1 − η s ) =
Revised: We let η 0 = β 0 and the constant c satisfy (cid:81) Ss =1 (1 − η s ) =

Original: ¯ α T .
Revised: ¯ α T .

Original: The noise level at step s is r s = √ ¯ γ s .
Revised: The noise level at step s is r s = √ ¯ γ s .

Original: For noise levels { η s } Ss =1 from steps, they are computed from selected steps { τ s } Ss =1 among { 1 , · · · , T } (Song et al., 2020a).
Revised: For noise levels { η s } Ss =1 from steps, they are computed from selected steps { τ s } Ss =1 among { 1 , · · · , T } (Song et al., 2020a).

Original: The two schedules are:
Revised: The two schedules are:

Original: • Linear (STEP): τ s = (cid:98) cs (cid:99) , where c = TS .
Revised: • Linear (STEP): τ s = (cid:98) cs (cid:99) , where c = TS .

Original: Quadratic (STEP): τ s = (cid:98) cs 2 (cid:99) , where c = 45 · TS 2 .
Revised: Quadratic (STEP): τ s = (cid:98) cs 2 (cid:99) , where c = 45 · TS 2 .

Original: Then, the noise level at step s is r s =
Revised: Then, the noise level at step s is r s =

Original: R ( τ s ) =
Revised: R ( τ s ) =

Original: √ ¯ α τ s .
Revised: √ ¯ α τ s .

Original: In image generation experiments, we follow the same noise level schedules as in Song et al.
Revised: In image generation experiments, we follow the same noise level schedules as in Song et al.

Original: (2020a): quadratic schedules for CIFAR-10 and linear schedules for CelebA and LSUN-bedroom.
Revised: (2020a): quadratic schedules for CIFAR-10 and linear schedules for CelebA and LSUN-bedroom.

Original: We use linear schedules in SC09 experiments and quadratic schedules in LJSpeech experiments; we ﬁnd these schedules have better quality.
Revised: We use linear schedules in SC09 experiments and quadratic schedules in LJSpeech experiments; we ﬁnd these schedules have better quality.

Original: Evaluations.
Revised: Evaluations.

Original: In all unconditional generation experiments, we use the Fréchet Inception Distance (FID) (Heusel et al., 2017;
Revised: In all unconditional generation experiments, we use the Fréchet Inception Distance (FID) (Heusel et al., 2017;

Original: Lang, 2020) to evaluate generated samples.
Revised: Lang, 2020) to evaluate generated samples.

Original: For the training set X t and the set of generated samples X g , the FID between these two sets is deﬁned as
Revised: For the training set X t and the set of generated samples X g , the FID between these two sets is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: 10) where µ t , µ g and Σ t , Σ g are the means and covariances of X t , X g after a feature transformation.
Revised: (11) where µ t , µ g and Σ t , Σ g are the means and covariances of X t , X g after a feature transformation.

Original: In each image generation experiment, X g is 50K generated images.
Revised: In each image generation experiment, X g is 50K generated images.

Original: The transformed feature is the 2048-dimensional vector output of the last layer of
Revised: The transformed feature is the 2048-dimensional vector output of the last layer of

Original: Inception-V3 (Szegedy et al., 2015).
Revised: Inception-V3 (Szegedy et al., 2015).

Original: In each audio synthesis experiment, X g is 5K generated utterances.
Revised: In each audio synthesis experiment, X g is 5K generated utterances.

Original: The transformed feature is the 1024-dimensional vector output of the last layer of a ResNeXT classiﬁer (Xu & Tuguldur, 2017), which achieves 99 .
Revised: The transformed feature is the 1024-dimensional vector output of the last layer of a ResNeXT classiﬁer (Xu & Tuguldur, 2017), which achieves 99 .

Original: 06% accuracy on the training set and 98 .
Revised: 06% accuracy on the training set and 98 .

Original: 76% accuracy on the test set.
Revised: 76% accuracy on the test set.

Original: The FID is the smaller the better.
Revised: The FID is the smaller the better.

Original: In the class-conditional generation experiment on SC09, we evaluate with accuracy and the Inception Score (IS).
Revised: In the class-conditional generation experiment on SC09, we evaluate with accuracy and the Inception Score (IS).

Original: The accuracy is computed by matching the predictions of the ResNeXT classiﬁer and the pre-speciﬁed labels in the dataset.
Revised: The accuracy is computed by matching the predictions of the ResNeXT classiﬁer and the pre-speciﬁed labels in the dataset.

Original: The
Revised: The

Original: IS of generated samples X g is deﬁned as
Revised: IS of generated samples X g is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: (11) where p ( x ) is the logit vector of the ResNeXT classiﬁer.
Revised: 12) where p ( x ) is the logit vector of the ResNeXT classiﬁer.

Original: The IS and accuracy are the larger the better.
Revised: The IS and accuracy are the larger the better.

Original: In the neural vocoding experiment on LJSpeech, we evaluate the speech quality with the crowdMOS tookit (Ribeiro et al., 2011), where the test utterances from all models were presented to Mechanical Turk workers.
Revised: In the neural vocoding experiment on LJSpeech, we evaluate the speech quality with the crowdMOS tookit (Ribeiro et al., 2011), where the test utterances from all models were presented to Mechanical Turk workers.

Original: We report the 5-scale Mean
Revised: We report the 5-scale Mean

Original: Opinion Scores (MOS), and it is the larger the better.
Revised: Opinion Scores (MOS), and it is the larger the better.

Original: D. Evaluation Results in Experiments
Revised: D. Evaluation Results in Experiments

Original: We report image generation results under different approximate diffusion processes, approximate reverse processes and S , the length of FastDPM.
Revised: We report image generation results under different approximate diffusion processes, approximate reverse processes and S , the length of FastDPM.

Original: Evaluation results on CIFAR-10, CelebA, and LSUN-bedroom measured in FID are shown in Table 1, Table 2, and Table 3, respectively.
Revised: Evaluation results on CIFAR-10, CelebA, and LSUN-bedroom measured in FID are shown in Table 1, Table 2, and Table 3, respectively.

Original: Approx.
Revised: [Table] [Table] [Table] We report audio synthesis results under different approximate diffusion processes, approximate reverse processes and S , the length of FastDPM.

Original: Approx.
Revised: Evaluation results of unconditional generation on SC09 measured in FID and IS are shown in Table 4. Evaluation results of class-conditional generation on SC09 measured in accuracy and IS are shown in Table 5. Evaluation results of neural vocoding on LJSpeech measured in MOS are shown in Table [Table]

Original: [Table]
Revised: [Table]

Revised: Approx.

Revised: Approx.

Revised: Accuracy ( ↑ )

Revised: IS ( ↑ )

Original: Diffusion
Revised: Diffusion

Original: Reverse
Revised: Reverse

Original: [Table]
Revised: S =

Original: STEP DDIM-rev ( κ =0 . 0 ) [Table] [Table]
Revised: S =

Original: [Table]
Revised: S =

Original: VAR DDIM-rev ( κ =[Table] [Table]
Revised: S =

Original: [Table] [Table]
Revised: S = S =

Original: STEP
Revised: STEP

Original: DDIM-rev ( κ = 0 . 2 ) 11.
Revised: DDIM-rev ( κ = 0 . 0 ) 66 .

Original: VAR
Revised: VAR

Original: DDIM-rev ( κ = 0 . 2 ) 10.
Revised: DDIM-rev ( κ = 0 . 0 ) 66 .

Original: STEP
Revised: STEP

Original: DDIM-rev ( κ = 0 .5 ) 13.
Revised: DDIM-rev ( κ = 0 .

Original: [Table]

Original: VAR
Revised: VAR

Original: DDIM-rev ( κ = 0 . 
Revised: DDIM-rev ( κ = 0 . 5 ) 86 . 0 % 88 .

Original: STEP [Table] 
Revised: [Table] STEP

Original: rev 36. [Table]
Revised: DDPM-rev 79 .

Original: VAR
Revised: VAR

Original: DDPM-rev 29.
Revised: DDPM-rev 81 . 0% 82 .

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: We report audio synthesis results under different approximate diffusion processes, approximate reverse processes and S , the length of FastDPM. Evaluation results of unconditional generation on SC09 measured in FID and IS are shown in Table 4. Evaluation results of class-conditional generation on SC09 measured in accuracy and IS are shown in Table 5. Evaluation results of neural vocoding on LJSpeech measured in MOS are shown in Table 6.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: DDIM-rev ( κ =
Revised: Approx. Diffusion Approx. Reverse

Original: [Table] [Table] [Table]
Revised: S

Original: [Table] [Table]
Revised: MOS ( ↑ ) STEP

Original: DDIM [Table] rev ( κ = 0 . 5 ) 
Revised: DDIM-rev ( κ = 0 . 0 ) 3 .

Original: [Table]
Revised: 72 ± 0 .

Original: DDIM
Revised: VAR

Original: [Table] rev ( κ = [Table]
Revised: DDIM-rev ( κ = 0 . 0 ) 3 .

Original: [Table] [Table] [Table] [Table]
Revised: 75 ± 0 .

Original: [Table]
Revised: STEP

Original: [Table] DDPM
Revised: DDPM-rev 4 .

Original: [Table]
Revised: 28 ± 0 .

Original: rev
Revised: VAR

Revised: DDPM-rev 4 . 31 ± 0 .

Original: [Table] [Table] [Table]
Revised: DiffWave ( T = 200 ) 4 .

Original: [Table]
Revised: 42 ± 0 .

Original: [Table] [Table] [Table]
Revised: Ground truth – 4 . 51 ± 0 .

Original: E. Generated Samples in Experiments
Revised: E. Generated Samples in Experiments

Original: In this section, we display generated samples of FastDPM, including image samples and mel-spectrogram of audio samples.
Revised: In this section, we display generated samples of FastDPM, including image samples and mel-spectrogram of audio samples.

Original: E.1. Unconditional Generation on CIFAR- 
Revised: E.1. Unconditional Generation on CIFAR-

Original: S = S = S = S =
Revised: [Equation]

Original: STEP VAR κ = 0 .κ = 0 .κ = 0 .
Revised: [Figure]

Original: DDPM-rev

Original: S = S =κ = 0 .κ = 0 .κ
Revised: [Equation]

Original: = 0 .

Original: DDPM-rev

Original: S =

Original: S =

Original: E.2. Unconditional Generation on CelebA
Revised: E.2. Unconditional Generation on CelebA

Original: S = S =
Revised: [Equation]

Original: S =
Revised: [Equation] [Equation] =

Original: S =
Revised: S =

Original: STEP
Revised: STEP

Original: VAR
Revised: VAR

Original: STEP
Revised: STEP

Original: VAR
Revised: VAR

Original: STEP
Revised: STEP

Original: VAR
Revised: VAR

Original: STEP
Revised: STEP

Original: VAR
Revised: VAR

Original: E.3. Unconditional Generation on LSUN-bedroom
Revised: E.3. Unconditional Generation on LSUN-bedroom

Original: STEP VAR Models STEP
Revised: [Figure]

Original: VAR STEP VAR STEP
Revised: [Figure]

Original: VAR

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E.4. Unconditional Generation on SC 
Revised: E.4. Unconditional Generation on SC

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: E.5. Conditional Generation on SC 
Revised: E.5. Conditional Generation on SC

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]