Original: L EARNING TO E FFICIENTLY S AMPLE FROM D IFFUSION P ROBABILISTIC M ODELS
Revised: L EARNING TO E FFICIENTLY S AMPLE FROM D IFFUSION P ROBABILISTIC M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Denoising Diffusion Probabilistic Models (DDPMs) have emerged as a powerful family of generative models that, yielding high-ﬁdelity samples and competitive log-likelihoods across a range of domains, including image and speech synthesis.
Revised: Denoising Diffusion Probabilistic Models (DDPMs) have emerged as a powerful family of generative models that, yielding high-ﬁdelity samples and competitive log-likelihoods across a range of domains, including image and speech synthesis.

Original: Key advantages of DDPMs include ease of training, in contrast to generative adversarial networks, and speed of generation, in contrast to autoregressive models.
Revised: Key advantages of DDPMs include ease of training, in contrast to generative adversarial networks, and speed of generation, in contrast to autoregressive models.

Original: However, DDPMs typically require hundreds-to-thousands of steps to generate a high ﬁdelity sample, making them prohibitively expensive for high dimensional problems.
Revised: However, DDPMs typically require hundreds-to-thousands of steps to generate a high ﬁdelity sample, making them prohibitively expensive for high dimensional problems.

Original: Fortunately, DDPMs allow trading generation speed for sample quality through adjusting the number of reﬁnement steps during inference.
Revised: Fortunately, DDPMs allow trading generation speed for sample quality through adjusting the number of reﬁnement steps during inference.

Original: Prior work has been successful in improving generation speed through handcrafting the time schedule through trial and error.
Revised: Prior work has been successful in improving generation speed through handcrafting the time schedule through trial and error.

Original: In our work, we view the selection of the inference time schedules as an optimization problem, and introduce an exact dynamic programming algorithm that ﬁnds the log-likelihood-optimal discrete time schedules for any pre-trained DDPM.
Revised: We instead view the selection of the inference time schedules as an optimization problem, and show that, with a simple dynamic programming algorithm, one can ﬁnd the log-likelihood-optimal discrete time schedules for any pre-trained DDPM.

Original: Our method exploits the fact that the evidence lower bound (ELBO) can be decomposed into separate KL divergence terms, and given any computation budget, we discover the time schedule that maximizes the training ELBO exactly.
Revised: Our method exploits the fact that the evidence lower bound (ELBO) can be decomposed into separate KL divergence terms, and given any computation budget, we discover the time schedule that maximizes the training ELBO exactly.

Original: Our method is efﬁcient, has no hyper-parameters of its own, and can be applied to any pre-trained DDPM with no retraining.
Revised: Our method is efﬁcient, has no hyper-parameters of its own, and can be applied to any pre-trained DDPM with no retraining.

Original: We discover inference time schedules requiring as few as 32 reﬁnement steps, while sacriﬁcing less than 0.1 bits per dimension compared to the default 4,000 steps used on an ImageNet 64x64 model.
Revised: We discover inference time schedules requiring as few as 32 reﬁnement steps, while sacriﬁcing less than 0.1 bits per dimension compared to the default 4,000 steps used on an ImageNet 64x64 model.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Denoising Diffusion Probabilistic Models (DDPMs) have emerged as a powerful class of generative models (Sohl-Dickstein et al., 2015; Ho et al., 2020).
Revised: Denoising Diffusion Probabilistic Models (DDPMs) have emerged as a powerful class of generative models (Sohl-Dickstein et al., 2015; Ho et al., 2020).

Original: DDPMs model the data distribution through an iterative denoising process, and have been applied successfully to a variety of applications, including unconditional image generation (Song & Ermon, 2019; Ho et al., 2020; Song et al., 2021;
Revised: DDPMs model the data distribution through an iterative denoising process, and have been applied successfully to a variety of applications, including unconditional image generation (Song & Ermon, 2019; Ho et al., 2020; Song et al., 2021;

Original: Nichol &
Revised: Nichol &

Original: Dhariwal, 2021), shape generation (Cai et al., 2020), text-to-speech (Chen et al., 2021; Kong et al., 2020) and single image super-resolution
Revised: Dhariwal, 2021), shape generation (Cai et al., 2020), text-to-speech (Chen et al., 2021; Kong et al., 2020) and single image super-resolution

Original: (Saharia et al., 2021; Li et al., 2021).
Revised: (Saharia et al., 2021; Li et al., 2021).

Original: DDPMs are easy to train, featuring a simple denoising objective (Ho et al., 2020) with noise schedules that successfully transfer across different models and datasets.
Revised: DDPMs are easy to train, featuring a simple denoising objective (Ho et al., 2020) with noise schedules that successfully transfer across different models and datasets.

Original: This contrasts to Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), which require an inner-outer loop optimization procedure that often entails instability and requires careful hyperparameter tuning.
Revised: This contrasts to Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), which require an inner-outer loop optimization procedure that often entails instability and requires careful hyperparameter tuning.

Original: DDPMs also admit a simple non-autoregressive inference process; this contrasts to autoregressive models with often prohibitive computational costs on high dimensional data.
Revised: DDPMs also admit a simple non-autoregressive inference process; this contrasts to autoregressive models with often prohibitive computational costs on high dimensional data.

Original: The DDPM inference process starts with samples from the corresponding prior noise distribution (e.g., standard Gaussian), and iteratively denoises the samples under the ﬁxed noise schedule.
Revised: The DDPM inference process starts with samples from the corresponding prior noise distribution (e.g., standard Gaussian), and iteratively denoises the samples under the ﬁxed noise schedule.

Original: However, DDPMs often need hundreds-tothousands of denoising steps (each involving a feedforward pass of a large neural network) to achieve strong results.
Revised: However, DDPMs often need hundreds-tothousands of denoising steps (each involving a feedforward pass of a large neural network) to achieve strong results.

Original: While this process is still much faster than autoregressive models, this is still often computationally prohibitive, especially when modeling high dimensional data.
Revised: While this process is still much faster than autoregressive models, this is still often computationally prohibitive, especially when modeling high dimensional data.

Original: There has been much recent work focused on improving the sampling speed of DDPMs.
Revised: There has been much recent work focused on improving the sampling speed of DDPMs.

Original: WaveGrad (Chen et al., 2021) introduced a manually crafted schedule requiring only 6 reﬁnement steps; however, this schedule seems to be only applicable to the vocoding task where there is a very strong conditioning signal.
Revised: WaveGrad (Chen et al., 2021) introduced a manually crafted schedule requiring only 6 reﬁnement steps; however, this schedule seems to be only applicable to the vocoding task where there is a very strong conditioning signal.

Original: Denoising Diffusion
Revised: Denoising Diffusion

Original: Implicit Models (DDIMs) (Song et al., 2020) accelerate sampling from pre-trained DDPMs by relying on a family of non-Markovian processes.
Revised: Implicit Models (DDIMs) (Song et al., 2020) accelerate sampling from pre-trained DDPMs by relying on a family of non-Markovian processes.

Original: They accelerate the generative process through taking multiple steps in the diffusion process.
Revised: They accelerate the generative process through taking multiple steps in the diffusion process.

Original: However, DDIMs sacriﬁce the ability to compute log-likelihoods.
Revised: However, DDIMs sacriﬁce the ability to compute log-likelihoods.

Original: Nichol & Dhariwal (2021) also explored the use of ancestral sampling with a subsequence of the original denoising steps, trying both a uniform stride and other hand-crafted strides.
Revised: Nichol & Dhariwal (2021) also explored the use of ancestral sampling with a subsequence of the original denoising steps, trying both a uniform stride and other hand-crafted strides.

Original: San-Roman et al.
Revised: San-Roman et al.

Original: (2021) improve few-step sampling further by training a separate model after training a DDPM to estimate the level of noise, and modifying inference to dynamically adjust the noise schedule at every step to match the predicted noise level.
Revised: (2021) improve few-step sampling further by training a separate model after training a DDPM to estimate the level of noise, and modifying inference to dynamically adjust the noise schedule at every step to match the predicted noise level.

Original: All these fast-sampling techniques rely on a key property of DDPMs – there is a decoupling between the training and inference schedule.
Revised: All these fast-sampling techniques rely on a key property of DDPMs – there is a decoupling between the training and inference schedule.

Original: The training schedule need not be the same as the inference schedule, e.g., a diffusion model trained to use 1000 steps may actually use only 10 steps during inference.
Revised: The training schedule need not be the same as the inference schedule, e.g., a diffusion model trained to use 1000 steps may actually use only 10 steps during inference.

Original: This decoupling characteristic is typically not found in other generative models.
Revised: This decoupling characteristic is typically not found in other generative models.

Original: In past work, the choice of inference schedule was often considered a hyperpameter selection problem, and often selected via intuition or extensive hyperparmeter exploration (Chen et al., 2021).
Revised: In past work, the choice of inference schedule was often considered a hyperpameter selection problem, and often selected via intuition or extensive hyperparmeter exploration (Chen et al., 2021).

Original: In this work, we view the choice of  inference schedule  path  as an independent optimization problem, wherein we attempt to learn the best schedule.
Revised: In this work, we view the choice of the timesteps of the inference schedule (which we just call an inference path ) as an independent optimization problem, wherein we attempt to learn the best schedule.

Original: Our approach relies on a dynamic programming algorithm, where given a ﬁxed budget of K reﬁnement steps and a pre-trained DDPM, we ﬁnd the set of timesteps that maximizes the corresponding evidence lower bound (ELBO).
Revised: Our approach relies on the observation that we can solve this optimization problem with dynamic programming. Given a ﬁxed budget of K reﬁnement steps and a pre-trained DDPM, we ﬁnd the set of timesteps that maximizes the corresponding evidence lower bound (ELBO).

Original: As an optimization objective, the
Revised: As an optimization objective, the

Original: ELBO has a key decomposability property: the total ELBO is the sum of individual KL terms, and for any two inference paths, if the timesteps ( s, t ) contiguously occur in both, they share a common
Revised: ELBO has a key decomposability property: the total ELBO is the sum of individual KL terms, and for any two inference paths, if the timesteps ( s, t ) contiguously occur in both, they share a common

Original: KL term, therefore admitting memoization.
Revised: KL term, therefore admitting memoization (see Section 4.1 for a precise deﬁnition).

Original: Our main contributions are the following:
Revised: Our main contributions are the following:

Original: • We introduce a dynamic programming algorithm that ﬁnds the likelihood-optimal inference paths based on the ELBO for all possible computation budgets of K reﬁnement steps.
Revised: • We introduce a method that that ﬁnds the likelihood-optimal inference paths with a simple dynamic programming algorithm for all possible computation budgets of K reﬁnement steps.

Original: The algorithm searches over T > K timesteps, only requiring O ( T ) neural network forward passes.
Revised: The algorithm searches over T > K timesteps, only requiring O ( T ) neural network forward passes.

Original: It only needs to be applied once to a pre-trained DDPM, does not require training or retraining a DDPM, and is applicable to both time-discrete and time-continuous DDPMs.
Revised: It only needs to be applied once to a pre-trained DDPM, does not require training or retraining a DDPM, and is applicable to both time-discrete and time-continuous DDPMs.

Original: • We experiment with DDPM models from prior work.
Revised: • We experiment with DDPM models from prior work.

Original: On both L simple CIFAR10 and L hybrid ImageNet 64x64, we discover schedules which require only 32 reﬁnement steps, yet sacriﬁce only 0.1 bits per dimension compared to their original counterparts with 1,000 and 4,000 steps, respectively.
Revised: On both L simple CIFAR10 and L hybrid ImageNet 64x64, we discover schedules which require only 32 reﬁnement steps, yet sacriﬁce only 0.1 bits per dimension compared to their original counterparts with 1,000 and 4,000 steps, respectively.

Original: • We show that our algorithm can be applied to any decomposable objective.
Revised: • We show that our method can be applied to any decomposable set of objectives.

Original: In particular, optimizing a reweighted ELBO can favourably bias our algorithm towards solutions with better FID scores, as we ﬁnd that optimizing the exact variational lower bound may lead to worse FID scores, which is consistent with prior work on unconditional image generation.
Revised: In particular, optimizing a reweighted ELBO can favourably bias our algorithm towards solutions with better FID scores, as we ﬁnd that optimizing the exact variational lower bound may lead to worse FID scores, which is consistent with prior work on unconditional image generation.

Original:  B ACKGROUND ON D ENOISING D IFFUSION P ROBABILISTIC M ODELS
Revised:  B ACKGROUND ON D ENOISING D IFFUSION P ROBABILISTIC M ODELS

Original: Denoising Diffusion Probabilistic Models (DDPMs) (Ho et al., 2020; Sohl-Dickstein et al., 2015) are deﬁned in terms of a forward Markovian diffusion process q and a learned reverse process p θ .
Revised: Denoising Diffusion Probabilistic Models (DDPMs) (Ho et al., 2020; Sohl-Dickstein et al., 2015) are deﬁned in terms of a forward Markovian diffusion process q and a learned reverse process p θ .

Original: The forward diffusion process gradually adds Gaussian noise to a data point x 0 through T iterations,
Revised: The forward diffusion process gradually adds Gaussian noise to a data point x 0 through T iterations,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  where the scalar parameters α 1: T determine the variance of the noise added at each diffusion step, subject to 0 < α t < 1 .
Revised: (2) where the scalar parameters α 1: T determine the variance of the noise added at each diffusion step, subject to 0 < α t < 1 .

Original: The learned reverse process aims to model q ( x 0 ) by inverting the forward process, gradually removing noise from signal starting from pure Gaussian noise x T ,
Revised: The learned reverse process aims to model q ( x 0 ) by inverting the forward process, gradually removing noise from signal starting from pure Gaussian noise x T ,

Original: [Equation]
Revised: [Equation]

Original: The parameters of the reverse process can be optimized by maximizing the following variational lower bound on the training set,
Revised: The parameters of the reverse process can be optimized by maximizing the following variational lower bound on the training set:

Original: [Equation]
Revised: [Equation]

Original: where L T ( x 0 )
Revised: where L T ( x 0 )

Original: =
Revised: =

Original: D KL  q ( x T | x 0 )  p ( x T )  .
Revised: D KL (cid:0) q ( x T | x 0 ) (cid:107) p ( x T ) (cid:1) .

Original: Nichol & Dhariwal (2021) have demonstrated that training DDPMs by maximizing the ELBO yields competitive log-likelihood scores on both CIFAR-10 and ImageNet 64 × 64 achieving 2 .
Revised: Nichol & Dhariwal (2021) have demonstrated that training DDPMs by maximizing the ELBO yields competitive log-likelihood scores on both CIFAR10 and ImageNet 64 × 64 achieving 2 .

Original: 94 and 3 .
Revised: 94 and 3 .

Original: 53 bits per dimension respectively.
Revised: 53 bits per dimension respectively.

Original: Two notable properties of Gaussian diffusion process that help formulate DDPMs tractably and efﬁciently include:
Revised: Two notable properties of Gaussian diffusion process that help formulate DDPMs tractably and efﬁciently include:

Original: [Equation]
Revised: [Equation]

Original: Given the marginal distribution of x t given x 0 in (7), one can sample from the q ( x t | x 0 ) independently for different t and perform SGD on a randomly chosen KL term in (6).
Revised: Given the marginal distribution of x t given x 0 in (7), one can sample from the q ( x t | x 0 ) independently for different t and perform SGD on a randomly chosen KL term in (6).

Original: Furthermore, given that the posterior distribution of x t − 1 given x t and x 0 is Gaussian, one can compute each KL term in (6) between two Gaussians in closed form and avoid high variance Monte Carlo estimation.
Revised: Furthermore, given that the posterior distribution of x t − 1 given x t and x 0 is Gaussian, one can compute each KL term in (6) between two Gaussians in closed form and avoid high variance Monte Carlo estimation.

Original:  L INKING DDPM S TO C ONTINUOUS T IME A FFINE D IFFUSION P ROCESSES
Revised:  L INKING DDPM S TO C ONTINUOUS T IME A FFINE D IFFUSION P ROCESSES

Original: Before describing our approach to efﬁciently sampling from DDPMs, it is helpful to link DDPMs to continuous time afﬁne diffusion processes, as it shows the compatibility of our approach to both time-discrete and time-continuous DDPMs (Song et al., 2021; Kingma et al., 2021).
Revised: Before describing our approach to efﬁciently sampling from DDPMs, it is helpful to link DDPMs to continuous time afﬁne diffusion processes, as it shows the compatibility of our approach to both time-discrete and time-continuous DDPMs (Song et al., 2021; Kingma et al., 2021).

Original: Let x 0 ∼ q ( x 0 ) denote a data point drawn from the empirical distribution of interest and let q ( x t | x 0 ) denote a stochastic process for t ∈ [0 , 1] deﬁned through an afﬁne diffusion process through the following stochastic differential equation (SDE):
Revised: Let x 0 ∼ q ( x 0 ) denote a data point drawn from the empirical distribution of interest and let q ( x t | x 0 ) denote a stochastic process for t ∈ [0 , 1] deﬁned through an afﬁne diffusion process through the following stochastic differential equation (SDE):

Original: [Equation]
Revised: [Equation]

Original: where f sde , g sde : [0 , 1] → [0 , 1] are integrable functions satisfying f sde (0)
Revised: where f sde , g sde : [0 , 1] → [0 , 1] are integrable functions satisfying f sde (0)

Original: = 1 and g sde (0)
Revised: = 1 and g sde (0)

Original: = 0 .
Revised: = 0 .

Original: Following Särkkä & Solin (2019) (section 6.1), we can compute the exact marginals q ( x t | x s ) for any 0 ≤ s < t ≤ 1 .
Revised: Following Särkkä & Solin (2019) (section 6.1), we can compute the exact marginals q ( x t | x s ) for any 0 ≤ s < t ≤ 1 .

Original: We get:
Revised: We get:

Original: [Equation]
Revised: [Equation]

Original: where ψ ( t, s ) =
Revised: where ψ ( t, s ) =

Original: exp (cid:82) t s f ( u ) du .
Revised: exp (cid:82) t s f ( u ) du .

Original: Since these integrals are difﬁcult to work with, we instead propose (in parallel to Kingma et al.
Revised: Since these integrals are difﬁcult to work with, we instead propose (in parallel to Kingma et al.

Original: to deﬁne the marginals directly:
Revised: to deﬁne the marginals directly:

Original: [Equation]
Revised: [Equation]

Original: where f, g : [0 , 1] → [0 , 1] are differentiable, monotonic functions satisfying f (0)
Revised: where f, g : [0 , 1] → [0 , 1] are differentiable, monotonic functions satisfying f (0)

Original: = 1 , f (1) = 0 , g (0) = 0 , g (1) = 1 .
Revised: = 1 , f (1) = 0 , g (0) = 0 , g (1) = 1 .

Original: Then, by implicit differentiation it follows that the corresponding diffusion is
Revised: Then, by implicit differentiation it follows that the corresponding diffusion is

Original: [Equation]
Revised: [Equation]

Original: We provide a proof for Equation 12 in the appendix (A.1).
Revised: We provide a proof for Equation 12 in the appendix (A.1).

Original: To complete our formulation, let f ts = f
Revised: To complete our formulation, let f ts = f

Original: ( t ) f
Revised: ( t ) f

Original: ( s ) and g ts = (cid:112) g
Revised: ( s ) and g ts = (cid:112) g

Original: ( t ) 2 − f 

Revised: ( t ) 2 − f

Original: ts g ( s ) 2 .
Revised: ts g ( s ) 2 .

Original: Then, it follows that for any 0 < s < t ≤ 1 we have that
Revised: Then, it follows that for any 0 < s < t ≤ 1 we have that

Original: [Equation]
Revised: [Equation]

Original: We include proofs for (13) and (14) in the appendix (A.2).
Revised: We include proofs for (13) and (14) in the appendix (A.2).

Original: These equations show that we can perform inference with any ancestral sampling path (i.e., the timesteps can attain continuous values) by formulating the reverse process in terms of the posterior distribution as
Revised: These equations show that we can perform inference with any ancestral sampling path (i.e., the timesteps can attain continuous values) by formulating the reverse process in terms of the posterior distribution as

Original: [Equation]
Revised: [Equation]

Original: justifying the compatibility of our main approach with time-continuous DDPMs.
Revised: justifying the compatibility of our main approach with time-continuous DDPMs.

Original: We note that this reverse process is also mathematically equivalent to a reverse process based on a time-discrete
Revised: We note that this reverse process is also mathematically equivalent to a reverse process based on a time-discrete

Original: DDPM derived from a subsequence of the original timesteps as done by Song et al.
Revised: DDPM derived from a subsequence of the original timesteps as done by Song et al.

Original: ; Nichol &
Revised: ; Nichol &

Original: Dhariwal (2021).
Revised: Dhariwal (2021).

Original: For the case of s = 0 in the reverse process, we follow the parametrization of Ho et al.
Revised: For the case of s = 0 in the reverse process, we follow the parametrization of Ho et al.

Original: (2020) to obtain discretized log likelihoods and compare our log likelihoods fairly with prior work.
Revised: (2020) to obtain discretized log likelihoods and compare our log likelihoods fairly with prior work.

Revised: Algorithm 1:

Revised: Given a matrix L ∼ ( T +1) × ( T +1) of precomputed L ( · , · ) terms, ﬁnd the likelihoodoptimal schedules for all step budgets.

Revised: [Equation]

Revised: k in

Original: L EARNING TO E FFICIENTLY S AMPLE FROM DDPM S
Revised: [Equation] : bpds [Equation] D

Revised: [Equation]

Revised: (bpds

Revised: [Equation]

Revised: return D

Revised: Algorithm 2: Fetch the shortest path of K steps from the dynamic programming results implicitly returned by Algorithm 1.

Revised: def

Revised: [Equation]

Original: We now introduce our dynamic programming (DP) approach.
Revised: _shortest_ [Equation] [Equation] [Equation]

Original: In general, after training a DDPM, there is a decoupling between training and inference schedules.
Revised: = [Equation] [Equation]

Original: One can use a different inference schedule compared to training.
Revised: = K for k in reversed(range(K)): optpath.append(t)t=D[k,t] return optpath  L EARNING TO E FFICIENTLY S AMPLE FROM DDPM S We now introduce our dynamic programming (DP) approach. In general, after training a DDPM, one can use a different inference path than the one used during training.

Original: Additionally, we can optimize a loss or reward function with respect to the timesteps themselves (after the DDPM is trained).
Revised: Additionally, one can optimize a loss or reward function with respect to the timesteps themselves after the DDPM is trained.

Original: In this paper, we use the ELBO as our objective, however we note that it is possible to directly optimize the timesteps with other metrics.
Revised: In this paper, we use the ELBO as our loss function, however we note that it is possible to directly optimize the timesteps with other objectives.

Original: 4. O PTIMIZING THE ELBO
Revised: 4. O PTIMIZING THE ELBO

Original: In our work, we choose to optimize ELBO as our objective.
Revised: In our work, we choose to optimize ELBO as our objective.

Original: We rely on one key property of ELBO, its decomposability .
Revised: We rely on one key property of ELBOs, their decomposability .

Original: We ﬁrst make a few observations. The DDPM models the transition probability p
Revised: Before deﬁning decomposability, we formally deﬁne a K - step inference path as a ﬁnite, monotonically increasing sequence of timesteps 0 = t (cid:48) 0 < t (cid:48) 1 < ... < t (cid:48) K − 1 < t (cid:48)

Original: θ
Revised: K

Original: ( x s | x t ) , or the cost to move from x t → x s . Given a pretrained DDPM, one can construct any valid ELBO path through it as long as two properties hold: The path starts at t = 0 and ends at t = 1 .2. The path is contiguously connected without breaks.
Revised: = 1 .

Original: We can construct an ELBO path that entails K ∈ N reﬁnement steps. I.e., for any K , and any given path of inference timesteps 0 = t (cid:48) < t (cid:48) 1 < ... <
Revised: Now, given a set S ⊆ [0 , 1] , we deﬁne a family of lower bounds L of an “ideal” objective L ideal to be

Original: t (cid:48)
Revised: S - decomposable if:

Original: K − 1 < t (cid:48) K = 1  , one can derive a corresponding
Revised: There is a bijection from L to the set of all inference paths t with all timesteps in S , and 2. There exists a function L : S × S → [0 , ∞ ) such that, for all inference paths t with all timesteps in S , L ideal ≥ (cid:80) | t |− 1 i =1 L ( t i , t i − 1 ) + C ( C a constant).

Original:  ELBO
Revised: We now show that DDPM ELBOs are decomposable. As shown by Song et al. ; Nichol & Dhariwal (2021) and the equations in Section 3, for any K and any K -step inference path t , there is a corresponding ELBO

Original: [Equation]
Revised: [Equation]

Original: 16) where 
Revised: where L (

Original: [Equation]
Revised: [Equation]

Revised: Since all of these are lower bounds of E q log p ( x 0 ) , we conclude the family of DDPM evidence lower bounds is decomposable.

Original: In other words, the ELBO is a sum of individual ELBO terms that are functions ofcontiguous timesteps ( t (cid:48) i , t (cid:48)
Revised: Speciﬁcally, a DDPM trained on a set of timesteps S admits Sdecomposable ELBOs. For DDPMs trained with continuous timesteps, S =

Original: i − 1 ) .
Revised: For DDPMs trained on discrete timesteps, S is the set of those timesteps, as there is no guarantee that the behavior of the model won’t be pathological when give timesteps it has never seen during training.

Original: Now the question remains, given a ﬁxed budget K steps, what is the optimal ELBO path?
Revised: Now the question remains, given a ﬁxed budget K steps, what is the optimal inference path?

Original: First, we observe that any two paths that share a ( t, s ) transition will share a common L ( t, s ) term.
Revised: First, we observe that any two paths that share a ( t, s ) transition will share a common L ( t, s ) term.

Original: We exploit this property in our dynamic programming algorithm.
Revised: We exploit this property in our dynamic programming algorithm.

Original: When given a grid  of timesteps 0 = t 0 < t 1 < ... <
Revised: When given a grid S of plausible inference paths 0 = t 0 < t 1 < ... <

Original: t T − 1 < t T = 1 with T ≥ K , it is possible to efﬁciently ﬁnd theexact optimum (i.e., ﬁnding { t (cid:48) 1 , ..., t (cid:48)
Revised: t T − 1 < t T = 1 with T ≥ K , it is possible to efﬁciently ﬁnd the

Original:  K − 1 } ⊂ { t 1 , ..., t T − 1 } with the best ELBO) by memoizing all the individual L ( t, s ) ELBO terms for s, t ∈ { t 0 , ..., t T } with s < t .
Revised: ELBO-optimal K -step inference path contained in S by memoizing all the individual L ( t, s ) ELBO terms for s, t ∈ { t 0 , ..., t T } with s < t .

Original: We can then solve the canonical least-cost-path problem on a directed graph where s → t are nodes and the edge connecting them has cost L ( t, s ) .
Revised: We can then solve the canonical least-cost-path problem on a directed graph where s → t are nodes and the edge connecting them has cost L ( t, s ) .

Original: For time-continuous DDPMs, the choice of grid (i.e., the t 1 , ..., t T − 1 ) can be arbitrary.

Original: For models trained with discrete timesteps, the grid must be a subset of (or the full) original steps used during training, unless the model was regularized during training with methods such as the sampling procedure proposed by Chen et al.

Original: 4. D YNAMIC P ROGRAMMING A LGORITHM
Revised: 4. D YNAMIC P ROGRAMMING A LGORITHM

Original: We now outline our methodology to solve the least-cost-path problem.
Revised: We now outline our methodology to solve the least-cost-path problem.

Original: Our solution is similar to
Revised: Our solution is similar to

Original: Dijkstra’s algorithm, but it differs to the classical least-cost-path problem where the latter is typically used, as our problem has additional constraints: we restrict our search to paths of exactly K +nodes, and the start and end nodes are ﬁxed.
Revised: Dijkstra’s algorithm, but it differs to the classical least-cost-path problem where the latter is typically used, as our problem has additional constraints: we restrict our search to paths of exactly K +nodes, and the start and end nodes are ﬁxed.

Original: Let C and D be ( K + 1) ×
Revised: Let C and D be ( K + 1) ×

Original: ( T + 1) matrices.
Revised: ( T + 1) matrices.

Original: C [ k, t ] will be the total cost of the least-cost-path of length k from t to 0.
Revised: C [ k, t ] will be the total cost of the least-cost-path of length k from t to 0.

Original: D will be ﬁlled with the timesteps corresponding to such paths; i.e., D [ k, t ] will be the timestep s immediately previous to t for the optimal k -step path (assuming t is also part of such path).
Revised: D will be ﬁlled with the timesteps corresponding to such paths; i.e., D [ k, t ] will be the timestep s immediately previous to t for the optimal k -step path (assuming t is also part of such path).

Original: We initialize C [0 , 0]
Revised: We initialize C [0 , 0]

Original: = 0 and all the other C [0 , · ] to ∞ (the D [0 , · ] are irrelevant, but for ease of index notation we keep them in this section).
Revised: = 0 and all the other C [0 , · ] to ∞ (the D [0 , · ] are irrelevant, but for ease of index notation we keep them in this section).

Original: Then, for each k from 1 to K , we iteratively set, for each t ,
Revised: Then, for each k from 1 to K , we iteratively set, for each t ,

Original: [Equation]
Revised: [Equation]

Original: input : L, T # L = KL cost table (Equation 17) D = np . full(( T + 1 , T + 1) , − 1) C = np .
Revised: where L ( t, s ) is the cost to transition from t to s (see Equation 17).

Original: full(( T + 1 , T + 1) , np . inf)

Original: C [0 , 0]

Original: =for k in range(1 , T + 1) do bpds =

Original: C [ k − 1 , None ] + L [: , :]

Original: C [ k ] = np .

Original: amin( bpds, axis = − 1 )

Original: D [ k ] = np .

Original: argmin( bpds, axis = − 1 ) endreturn D input : D, K optpath

Original: =

Original: [ ] t = K for k in reversed(range(( K )) do optpath.

Original: append( t ) t =

Original: D [ k, t ] endreturn optpath where L ( t, s ) is the cost to transition from t to s (see Equation 17).

Original: For all s ≥ t , we set L ( t, s ) =
Revised: For all s ≥ t , we set L ( t, s ) =

Original: ∞ (e.g., we only move backwards in the diffusion process).
Revised: ∞ (e.g., we only move backwards in the diffusion process).

Original: This procedure captures the shortest path cost in C and the shortest path itself in D .
Revised: This procedure captures the shortest path cost in C and the shortest path itself in D .

Original: We further observe that running the DP algorithm for each k from 1 to T (instead of K ), we can extract the optimal paths for all possible budgets K .
Revised: We further observe that running the DP algorithm for each k from 1 to T (instead of K ), we can extract the optimal paths for all possible budgets K .

Original: Algorithm 1 illustrates a vectorized version of the procedure we have outlined in this section, while Algorithmshows how to explicitly extract the optimal paths from D .
Revised: Algorithm 1 illustrates a vectorized version of the procedure we have outlined in this section, while Algorithmshows how to explicitly extract the optimal paths from D .

Original: 4. E FFICIENT M EMOIZATION
Revised: 4. E FFICIENT M EMOIZATION

Original: A priori, our dynamic programming approach appears to be inefﬁcient because it requires computing
Revised: A priori, our dynamic programming approach appears to be inefﬁcient because it requires computing

Original: O ( T 2 ) terms (recall, as we rely on all the L ( t, s ) terms which depend on a neural network forward pass).
Revised: O ( T 2 ) terms (recall, as we rely on all the L ( t, s ) terms which depend on a neural network forward pass).

Original: We however observe that a single forward pass of the DDPM can be used to compute all the
Revised: We however observe that a single forward pass of the DDPM can be used to compute all the

Original: L ( t, · ) terms.
Revised: L ( t, · ) terms.

Original: This holds true even in the case where the pre-trained DDPM learns the variances.
Revised: This holds true even in the case where the pre-trained DDPM learns the variances.

Original: For example, in Nichol & Dhariwal (2021) instead of ﬁxing them to ˜ g ts = g
Revised: For example, in Nichol & Dhariwal (2021) instead of ﬁxing them to ˜ g ts = g

Original: ts g s 0 g t 0 as we outlined in the previous section, the forward pass itself still only depends on t and not s , and the variance of p θ ( x s | x t ) is obtained by interpolating the forward pass’s output logits v with exp( v log g 2 ts
Revised: ts g s 0 g t 0 as we outlined in the previous section, the forward pass itself still only depends on t and not s , and the variance of p θ ( x s | x t ) is obtained by interpolating the forward pass’s output logits v with exp( v log g 2 ts

Original: + (1 − v ) log ˜ g 2 ts ) .
Revised: + (1 − v ) log ˜ g 2 ts ) .

Original: Thus, computing the table of all the L ( t, s ) ELBO terms only requires O ( T ) forward passes.
Revised: Thus, computing the table of all the L ( t, s ) ELBO terms only requires O ( T ) forward passes.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We apply our method on a wide variety of pre-trained DDPMs from prior work.
Revised: We apply our method on a wide variety of pre-trained DDPMs from prior work.

Original: This emphasizes the fact that our method is applicable to any pre-trained DDPM model.
Revised: This emphasizes the fact that our method is applicable to any pre-trained DDPM model.

Original: In particular, we rely the CIFAR model checkpoints released by Nichol & Dhariwal (2021) on both their L hybrid and L vlb objectives.
Revised: In particular, we rely the CIFAR model checkpoints released by Nichol & Dhariwal (2021) on both their L hybrid and L vlb objectives.

Original: We also showcase results on CIFAR10 (Krizhevsky et al., 2009) with the exact conﬁguration used by Ho et al.
Revised: We also showcase results on CIFAR10 (Krizhevsky et al., 2009) with the exact conﬁguration used by Ho et al.

Original: (2020), which we denote as L simple , as well as L hybrid on ImageNet 64x64 (Deng et al.,2009) following Nichol & Dhariwal (2021), training these last two models ourselves for 800K and 3M steps, respectively, but otherwise using the exact same conﬁgurations as the authors.
Revised: (2020), which we denote as L simple , as well as L hybrid on ImageNet 64x64 (Deng et al.,

Original: [Table]
Revised: [Table]

Original: dataset. We underline that, with just 32 steps, our DP stride achieves a score of ≤ 0 .
Revised: that, with just 32 steps, our DP stride achieves a score of ≤ 0 .

Original: 1 bits/dim higher than the same model with the original 4,000 step budget ( ∗ the authors report 3.57 bits/dim, but we trained the model for 3M rather than 1.5M steps).
Revised: 1 bits/dim higher than the same model with the original 4,000 step budget ( ∗ the authors report 3.57 bits/dim, but we trained the model for 3M rather than 1.5M steps).

Original: [Table]
Revised: [Table]

Revised: 2009) following Nichol & Dhariwal (2021), training these last two models from scratch for 800K and 3M steps, respectively, but otherwise using the exact same conﬁgurations as the authors.

Original: In our experiments, we always search over a grid that includes all the timesteps used to train the model, i.e., { t/T : t ∈ { 1 , ..., T − 1 }} .
Revised: In our experiments, we always search over a grid that includes all the timesteps used to train the model, i.e., { t/T : t ∈ { 1 , ..., T − 1 }} .

Original: For our CIFAR10 results, we computed the memoization tables with Monte Carlo estimates over the full training dataset, while on ImageNet 64x64 we limited the number of datapoints in the Monte Carlo estimates to 16,384 images on the training dataset.
Revised: For our CIFAR10 results, we computed the memoization tables with Monte Carlo estimates over the full training dataset, while on ImageNet 64x64 we limited the number of datapoints in the Monte Carlo estimates to 16,384 images on the training dataset.

Original: For each pre-trained model, we compare the negative log likelihoods (estimated using the full heldout dataset) of the strides discovered by our dynamic programming algorithm against even and quadratic strides, following Song et al.
Revised: For each pre-trained model, we compare the negative log likelihoods (estimated using the full heldout dataset) of the strides discovered by our dynamic programming algorithm against even and quadratic strides, following Song et al.

Original: We ﬁnd that our dynamic programming algorithm discovers strides resulting in much better log likelihoods than the hand-crafted strides used in prior work, particularly in the few-step regime.
Revised: We ﬁnd that our dynamic programming algorithm discovers strides resulting in much better log likelihoods than the hand-crafted strides used in prior work, particularly in the few-step regime.

Original: We provide a visualization of the log likelihood curves as a function of computation budget in Figure 1 for L simple CIFAR10 and L hybrid ImageNet 64x64 (Deng et al., 2009), a full list of the scores in the few-step regime in Table 1, and a visualization of the discovered steps themselves in Figure 2.
Revised: We provide a visualization of the log likelihood curves as a function of computation budget in Figure 1 for L simple CIFAR10 and L hybrid ImageNet 64x64 (Deng et al., 2009), a full list of the scores in the few-step regime in Table 1, and a visualization of the discovered steps themselves in Figure 2.

Original: 5. C OMPARISON WITH FID
Revised: 5. C OMPARISON WITH FID

Original: We further evaluate our discovered strides by reporting FID scores (Heusel et al., 2017) on 50, model samples against the same number of samples from the training dataset, as is standard in the literature.
Revised: We further evaluate our discovered strides by reporting FID scores (Heusel et al., 2017) on 50, model samples against the same number of samples from the training dataset, as is standard in the literature.

Original: We ﬁnd that, although our strides are yield much better log likelihoods, such optimization does not necessarily translate to also improving the FID scores.
Revised: We ﬁnd that, although our strides are yield much better log likelihoods, such optimization does not necessarily translate to also improving the FID scores.

Original: Results are included in Figure 3.
Revised: Results are included in Figure 3.

Original: This weakened correlation between log-likehoods and FID is consistent with observations in prior work (Ho et al., 2020; Nichol & Dhariwal, 2021).
Revised: This weakened correlation between log-likehoods and FID is consistent with observations in prior work (Ho et al., 2020; Nichol & Dhariwal, 2021).

Original: To remedy this issue, we show that despite our focus in this work being likelihood, we can signiﬁcantly improve the FID scores discovered by our method simply by optimizing a reweighted ELBO.
Revised: To remedy this issue, we show that despite our focus in this work being likelihood, we can signiﬁcantly improve the FID scores discovered by our method simply by optimizing a reweighted ELBO.

Original: Recall that, as discussed in Section 4, our proposed method is compatible with any decomposable objective.
Revised: Recall that, as discussed in Section 4, our proposed method is compatible with any decomposable objective.

Original: Moreover, prior work has shown that the choice of ELBO weights has a signiﬁcant effect on sample quality (Ho et al., 2020; Durkan & Song, 2021), and that choosing weights corresponds to choosing an equally valid variational lower bound of the data for a DDIM (Song et al., 2020).
Revised: Moreover, prior work has shown that the choice of ELBO weights has a signiﬁcant effect on sample quality (Ho et al., 2020; Durkan & Song, 2021), and that choosing weights corresponds to choosing an equally valid variational lower bound of the data for a DDIM (Song et al., 2020).

Original: Similarly to prior work in the VAE literature, we thus stumble upon an open problem where different variational lower bounds compatible with the model (even with the same BPD) can lead to samples with different qualitative charachteristics (Alemi et al., 2018).
Revised: Similarly to prior work in the VAE literature, we thus stumble upon an open problem where different variational lower bounds compatible with the model (even with the same bits/dim) can lead to samples with different qualitative charachteristics (Alemi et al., 2018).

Original: As our focus is likelihood, we leave this research question of ﬁnding the weights / ELBO that lead to most agreement with FID for future work, but nevertheless identify one such choice that favourably biases our algorithm toward this front.
Revised: As our focus is likelihood, we leave this research question of ﬁnding the weights / ELBO that lead to most agreement with FID for future work, but nevertheless identify one such choice that favourably biases our algorithm toward this front.

Original: Details about our construction of weights and results are included in the appendix (A.3).
Revised: Details about our construction of weights and results are included in the appendix (A.3).

Original: 5. M ONTE C ARLO A BLATION
Revised: 5. M ONTE C ARLO A BLATION

Original: To investigate the feasibility of our approach using minimal computation, we experimented with setting the number of Monte Carlo datapoints used to compute the dynamic programming table of negative log likelihood terms to 128 samples (i.e., easily ﬁt into a single batch of GPU memory).
Revised: To investigate the feasibility of our approach using minimal computation, we experimented with setting the number of Monte Carlo datapoints used to compute the dynamic programming table of negative log likelihood terms to 128 samples (i.e., easily ﬁt into a single batch of GPU memory).

Original: We ﬁnd that, for CIFAR10, the difference in log likelihoods is negligible, while on ImageNet 64x there is a visible yet slight improvement in negative log likelihood when ﬁlling the table with more samples.
Revised: We ﬁnd that, for CIFAR10, the difference in log likelihoods is negligible, while on ImageNet 64x there is a visible yet slight improvement in negative log likelihood when ﬁlling the table with more samples.

Original: We hypothesize that this is due to the higher diversity of ImageNet.
Revised: We hypothesize that this is due to the higher diversity of ImageNet.

Original: Nevertheless, we highlight that our procedure can be applied very quickly (i.e., with just T forward passes of a neural network when using a single batch, as opposed to a running average over batches), even for large models, to signiﬁcantly improve log their likelihoods in the few-step regime.
Revised: Nevertheless, we highlight that our procedure can be applied very quickly (i.e., with just T forward passes of a neural network when using a single batch, as opposed to a running average over batches), even for large models, to signiﬁcantly improve log their likelihoods in the few-step regime.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: DDPMs (Ho et al., 2020) have recently shown results that are competitive with GANs (Goodfellow et al., 2014), and they can be traced back to the work of Sohl-Dickstein et al.
Revised: DDPMs (Ho et al., 2020) have recently shown results that are competitive with GANs (Goodfellow et al., 2014), and they can be traced back to the work of Sohl-Dickstein et al.

Original: (2015) as a restricted family of deep latent variable models.
Revised: (2015) as a restricted family of deep latent variable models.

Original: Dhariwal & Nichol (2021) have more recently shown that
Revised: Dhariwal & Nichol (2021) have more recently shown that

Original: DDPMs can outperform GANs in FID scores (Heusel et al., 2017).
Revised: DDPMs can outperform GANs in FID scores (Heusel et al., 2017).

Original: Song & Ermon (2019) have also linked DDPMs to denoising score matching (Vincent et al., 2008; 2010), which is crucial to the continuous-time formulation (Song et al., 2021; Kingma et al., 2021).
Revised: Song & Ermon (2019) have also linked DDPMs to denoising score matching (Vincent et al., 2008; 2010), which is crucial to the continuous-time formulation (Song et al., 2021; Kingma et al., 2021).

Original: More recent work on the few-step regime of DDPMs (Song et al., 2020;
Revised: More recent work on the few-step regime of DDPMs (Song et al., 2020;

Original: Chen et al., 2021; Nichol & Dhariwal, 2021;
Revised: Chen et al., 2021; Nichol & Dhariwal, 2021;

Original: San-Roman et al., 2021; Kong & Ping, 2021; Jolicoeur-Martineau et al., 2021) has also guided our research efforts.
Revised: San-Roman et al., 2021; Kong & Ping, 2021; Jolicoeur-Martineau et al., 2021) has also guided our research efforts.

Original: DDPMs are also very closely related to variational autoencoders (Kingma & Welling,
Revised: DDPMs are also very closely related to variational autoencoders (Kingma & Welling,

Original: 2013), where more recent work has shown that, with many stochastic layers, they can also attain competitive negative log likelihoods in unconditional image generation (Child, 2020).
Revised: 2013), where more recent work has shown that, with many stochastic layers, they can also attain competitive negative log likelihoods in unconditional image generation (Child, 2020).

Original: Also very closely related to DDPMs, there has also been work on non-autoregressive modeling of text sequences that can be regarded as discrete-space DDPMs with a forward process that masks or remove tokens (Lee et al., 2018;
Revised: Also very closely related to DDPMs, there has also been work on non-autoregressive modeling of text sequences that can be regarded as discrete-space DDPMs with a forward process that masks or remove tokens (Lee et al., 2018;

Original: Gu et al., 2019; Stern et al., 2019; Chan et al., 2020; Saharia et al., 2020).
Revised: Gu et al., 2019; Stern et al., 2019; Chan et al., 2020; Saharia et al., 2020).

Original: The UNet architecture (Ronneberger et al., 2015) has been key to the recent success of DDPMs, and as shown by Ho et al.
Revised: The UNet architecture (Ronneberger et al., 2015) has been key to the recent success of DDPMs, and as shown by Ho et al.

Original: (2020); Nichol & Dhariwal (2021), augmenting UNet with self-attention (Shaw et al., 2018) in scales where attention is computationally feasible has helped bring DDPMs closer to the current state-of-the-art autoregressive generative models (Child et al., 2019;
Revised: (2020); Nichol & Dhariwal (2021), augmenting UNet with self-attention (Shaw et al., 2018) in scales where attention is computationally feasible has helped bring DDPMs closer to the current state-of-the-art autoregressive generative models (Child et al., 2019;

Original: Jun et al., 2020; Roy et al., 2021).
Revised: Jun et al., 2020; Roy et al., 2021).

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: By regarding the selection of the inference schedule as an optimization problem, we present a novel and efﬁcient dynamic programming algorithm to discover a likelihood-optimal inference schedule for a pre-trained DDPM.
Revised: By regarding the selection of the inference schedule as an optimization problem, we present a novel and efﬁcient approach to discover a likelihood-optimal inference schedule for a pre-trained DDPM with a simple dynamic programming algorithm.

Original: Our method need only be applied once to discover the schedule, and does not require re-training the DPPM.
Revised: Our method need only be applied once to discover the schedule, and does not require re-training the DPPM.

Original: In the few-step regime, we discover schedules on L simple CIFAR10 and L hybrid ImageNet 64x64 that require only 32 steps, yet sacriﬁce ≤ 0 .
Revised: In the few-step regime, we discover schedules on L simple CIFAR10 and L hybrid ImageNet 64x64 that require only 32 steps, yet sacriﬁce ≤ 0 .

Original: 1 bits per dimension compared to state-of-the-art DDPMs using hundreds-to-thousands of reﬁnement steps.
Revised: 1 bits per dimension compared to state-of-the-art DDPMs using hundreds-to-thousands of reﬁnement steps.

Original: Our approach only needs forward passes of the DDPM neural network to ﬁll the dynamic programming table of L ( t, s ) terms, and we show that we can ﬁll the dynamic programming table with just O ( T ) forward passes.
Revised: Our approach only needs forward passes of the DDPM neural network to ﬁll the dynamic programming table of L ( t, s ) terms, and we show that we can ﬁll the dynamic programming table with just O ( T ) forward passes.

Original: Moreover, we can estimate the table using only 128 Monte Carlo samples, ﬁnding this to be sufﬁcient even for datasets such as ImageNet with high diversity.
Revised: Moreover, we can estimate the table using only 128 Monte Carlo samples, ﬁnding this to be sufﬁcient even for datasets such as ImageNet with high diversity.

Original: Our method achieves strong likelihoods with very few reﬁnement steps, outperforming prior work utilizing hand-crafted strides (Ho et al., 2020; Nichol & Dhariwal, 2021).
Revised: Our method achieves strong likelihoods with very few reﬁnement steps, outperforming prior work utilizing hand-crafted strides (Ho et al., 2020; Nichol & Dhariwal, 2021).

Original: Despite very strong log-likelihood results, we observe that maximizing the unweighted ELBO can actually lead to higher (worse) FID scores, and on ImageNet 64x64, a decrease in sample quality for the smallest budgets K ∈ { 32 , 64 } ; this is consistent with ﬁndings in prior work (Ho et al., 2020;
Revised: Despite very strong log-likelihood results, we observe that maximizing the unweighted ELBO can actually lead to higher (worse) FID scores, and on ImageNet 64x64, a decrease in sample quality for the smallest budgets K ∈ { 32 , 64 } ; this is consistent with ﬁndings in prior work (Ho et al., 2020;

Original: Nichol & Dhariwal, 2021).
Revised: Nichol & Dhariwal, 2021).

Original: Nevertheless, our method is compatible with any decomposable objective such as reweighted variational lower bounds, and we show that a simple choice of reweighted ELBO (or equivalently a choice of DDIM) can remedy this issue.
Revised: Nevertheless, our method is compatible with any decomposable objective such as reweighted variational lower bounds, and we show that a simple choice of reweighted ELBO (or equivalently a choice of DDIM) can remedy this issue.

Original: Developing principled methods to choose variational lower bounds or other decomposable metrics that correlate best with image quality is thus an important direction for future research.
Revised: Developing principled methods to choose variational lower bounds or other decomposable metrics that correlate best with image quality is thus an important direction for future research.

Original: Finally, we remark that likelihood optimization itself is useful for speciﬁc applications: as well as better compression, in domains such as non-autoregressive text generation where likelihood correlates much better with sample quality and where diffusion models are starting to make progress (Austin et al., 2021), our method has the potential to improve sampling speed with far less cost in generation ﬁdelity and without such adaptations.
Revised: Finally, we remark that likelihood optimization itself is useful for speciﬁc applications: as well as better compression, in domains such as non-autoregressive text generation where likelihood correlates much better with sample quality and where diffusion models are starting to make progress (Austin et al., 2021), our method has the potential to improve sampling speed with far less cost in generation ﬁdelity and without such adaptations.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We will fully open source our work and provide code pointers in the paper in the camera-ready version.
Revised: We will fully open source our work and provide code pointers in the paper in the camera-ready version.

Original: Nevertheless, we provide pseudocode with a complete implementation of our proposed algorithm to maximize ease of reproducibility while we work on open-sourcing our work (see Algorithmsand 2).
Revised: Nevertheless, we provide pseudocode with a complete implementation of our proposed algorithm to maximize ease of reproducibility while we work on open-sourcing our work (see Algorithms ?? and ?? )

Original: Since we experiment with open-sourced datasets and pre-trained models that already have publicly available checkpoints, our work is fully reproducible.
Revised: Since we experiment with open-sourced datasets and pre-trained models that already have publicly available checkpoints, our work is fully reproducible.

Original: We additionally emphasize that our method has no hyperparameters of its own.
Revised: We additionally emphasize that our method has no hyperparameters of its own.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Innovations in generative models have the potential to enable harmful and unethical applications.
Revised: Innovations in generative models have the potential to enable harmful and unethical applications.

Original: In applications where no harm is intended, bias and other failure modes of generative models and datasets used to train them can also lead to issues with fairness, discrimination, and other forms of harm.
Revised: In applications where no harm is intended, bias and other failure modes of generative models and datasets used to train them can also lead to issues with fairness, discrimination, and other forms of harm.

Original: While our work is focused on making diffusion models more efﬁcient, we believe its public release will not cause any form of immediate harm, as much more efﬁcient generative models for images like GANs can still achieve high sample quality at a fraction of the speed of diffusion models
Revised: While our work is focused on making diffusion models more efﬁcient, we believe its public release will not cause any form of immediate harm, as much more efﬁcient generative models for images like GANs can still achieve high sample quality at a fraction of the speed of diffusion models

Original: A A PPENDIX A. P ROOF FOR E QUATION
Revised: A A PPENDIX A. P ROOF FOR E QUATION

Original: From Equation 10, we get by implicit differentiation that
Revised: From Equation 10, we get by implicit differentiation that

Original: [Equation]
Revised: [Equation]

Original: Similarly as above and also using the fact that ψ ( t, s ) =
Revised: Similarly as above and also using the fact that ψ ( t, s ) =

Original: ψ ( t,
Revised: ψ ( t,

Original: [Equation]
Revised: [Equation]

Original: ψ ( s
Revised: ψ ( s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ⇒ 2 g ( t ) g
Revised: ⇒ 2 g ( t ) g

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: ) f
Revised: ) f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ⇒ g sde ( t ) =
Revised: ⇒ g sde ( t ) =

Original: (cid:112)
Revised: (cid:112)

Original: [Equation]
Revised: [Equation]

Original: g (
Revised: g (

Original: [Equation]
Revised: [Equation]

Original: g (cid:48) ( t ) − f sde ( t ) g ( t )
Revised: g (cid:48) ( t ) − f sde ( t ) g ( t )

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF FOR E QUATIONS 13 AND
Revised: A. P ROOF FOR E QUATIONS 13 AND

Original: From Equation 10 and ψ ( t, s ) =
Revised: From Equation 10 and ψ ( t, s ) =

Original: ψ ( t, 0) ψ ( s, 0) it is immediate that f ts is the mean of q ( x t | x s ) .
Revised: ψ ( t, 0) ψ ( s, 0) it is immediate that f ts is the mean of q ( x t | x s ) .

Original: To show that g 2 ts is the variance of q ( x t | x s ) , Equation 10 implies that
Revised: To show that g 2 ts is the variance of q ( x t | x s ) , Equation 10 implies that

Original: Var
Revised: Var

Original: [Equation]
Revised: [Equation]

Original: The mean of q ( x s | x t , x 0 ) is given by the Gaussian conjugate prior formula (where all the distributions are conditioned on x 0 ).
Revised: The mean of q ( x s | x t , x 0 ) is given by the Gaussian conjugate prior formula (where all the distributions are conditioned on x 0 ).

Original: Let µ = f ts x s , so we have a prior over µ given by x
Revised: Let µ = f ts x s , so we have a prior over µ given by x

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: ∼ N
Revised: ∼ N

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: g
Revised: g

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ⇒
Revised: ⇒

Original: [Equation]
Revised: [Equation]

Original: f 2 ts g 2 s 0 I d ) , and a likelihood with mean µ
Revised: f 2 ts g 2 s 0 I d ) , and a likelihood with mean µ

Original: [Equation]
Revised: [Equation]

Original: Then it follows by the formula that µ | x t , x 0 has variance
Revised: Then it follows by the formula that µ | x t , x 0 has variance

Original: Var[ µ | x
Revised: Var[ µ | x

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: and mean
Revised: and mean

Original: [Equation]
Revised: [Equation]

Original: A. R EWEIGHTED ELBO RESULTS
Revised: A. R EWEIGHTED ELBO RESULTS

Original: While the focus of our work is likelihood, we report FID scores for the sake of completeness, as well as to show the adaptability of our method via reweighting to focus on sample quality, as mentioned in
Revised: While the focus of our work is likelihood, we report FID scores for the sake of completeness, as well as to show the adaptability of our method via reweighting to focus on sample quality, as mentioned in

Original: Section 5.1.
Revised: Section 5.1.

Original: To choose a reweighting scheme for each L ( t, s ) term that takes into account both t and s , Kingma et al.
Revised: To choose a reweighting scheme for each L ( t, s ) term that takes into account both t and s , Kingma et al.

Original: (2021) show that choosing discretized terms
Revised: (2021) show that choosing discretized terms

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 18) ensures that at the limit of inﬁnite timesteps the continuous
Revised: ensures that at the limit of inﬁnite timesteps the continuous

Original: ELBO becomes−
Revised: ELBO becomes

Original: [Equation]
Revised: [Equation]

Original: (where SNR( t ) = f 2 t
Revised: (where SNR( t ) = f 2 t

Original: [Equation]
Revised: [Equation]

Original: g 2 t 0 , and a constant w ( t ) =yields the unweighted ELBO).
Revised: g 2 t 0 , and a constant w ( t ) =yields the unweighted ELBO).

Original: While choosing w ( t ) = − SNR( t ) SNR (cid:48) ( t ) (which is L simple at the limit) did not work, we ﬁnd that choosing w ( t ) =
Revised: While choosing w ( t ) = − SNR( t ) SNR (cid:48) ( t ) (which is L simple at the limit) did not work, we ﬁnd that choosing w ( t ) =

Original: − 1 SNR (cid:48) ( t ) (which leads to a continuous objective of unweighted mean square errors and
Revised: − 1 SNR (cid:48) ( t ) (which leads to a continuous objective of unweighted mean square errors and

Original: L w ( t, s ) =
Revised: L w ( t, s ) =

Original: ( t − s ) (cid:107) x 0 − ˆ x 0 ( x t , t ) (cid:107) 2 ) allows our algorithm to outperform DDPM FID scores and achieve similar scores to DDIM (Song et al., 2020).
Revised: ( t − s ) (cid:107) x 0 − ˆ x 0 ( x t , t ) (cid:107) 2 ) allows our algorithm to outperform DDPM FID scores and achieve similar scores to DDIM (Song et al., 2020).

Original: We call this “MSE reweighting” and include
Revised: We call this “MSE reweighting” and include

Original: FID scores in Table 3, comparing to DDPM but also DDIM( η = 0 ) which does not admit likelihood computation but has been shown to be one of the strongest FID baselines in the few-step regime.
Revised: FID scores in Table 5, comparing to DDPM but also DDIM( η = 0 ) which does not admit likelihood computation but has been shown to be one of the strongest FID baselines in the few-step regime.

Original: The FID scores were estimated with 50,000 model and training data samples, as is standard in the literature.
Revised: The FID scores were estimated with 50,000 model and training data samples, as is standard in the literature.