Original: S OFT D IFFUSION
Revised: S OFT D IFFUSION

Original: S CORE M ATCHING F OR G ENERAL C ORRUPTIONS
Revised: S CORE M ATCHING F OR G ENERAL C ORRUPTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We deﬁne a broader family of corruption processes that generalizes previously known diffusion models.
Revised: We deﬁne a broader family of corruption processes that generalizes previously known diffusion models.

Original: To reverse these general diffusions, we propose a new objective called Soft Score Matching that provably learns the score function for any linear corruption process and yields state of the art results for CelebA. Soft
Revised: To reverse these general diffusions, we propose a new objective called Soft Score Matching that provably learns the score function for any linear corruption process and yields state of the art results for CelebA. Soft

Original: Score Matching incorporates the degradation process in the network.
Revised: Score Matching incorporates the degradation process in the network.

Original: Our new loss trains the model to predict a clean image, that after corruption , matches the diffused observation.
Revised: Our new loss trains the model to predict a clean image, that after corruption , matches the diffused observation.

Original: We show that our objective learns the gradient of the likelihood under suitable regularity conditions for a family of corruption processes.
Revised: We show that our objective learns the gradient of the likelihood under suitable regularity conditions for a family of corruption processes.

Original: We further develop a principled way to select the corruption levels for general diffusion processes and a novel sampling method that we call Momentum Sampler.
Revised: We further develop a principled way to select the corruption levels for general diffusion processes and a novel sampling method that we call Momentum Sampler.

Original: We show experimentally that our framework works for general linear corruption processes, such as Gaussian blur and masking.
Revised: We show experimentally that our framework works for general linear corruption processes, such as Gaussian blur and masking.

Original: We achieve state-of-the-art FID score 1 . 85 on
Revised: We achieve state-of-the-art FID score 1 . 85 on

Original: CelebA-64, outperforming all previous linear diffusion models.
Revised: CelebA-64, outperforming all previous linear diffusion models.

Original: We also show signiﬁcant computational beneﬁts compared to vanilla denoising diffusion.
Revised: We also show signiﬁcant computational beneﬁts compared to vanilla denoising diffusion.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Score-based models (Song & Ermon, 2019; 2020; Song et al., 2021b) and Denoising Diffusion
Revised: Score-based models (Song & Ermon, 2019; 2020; Song et al., 2021b) and Denoising Diffusion

Original: Probabilistic Models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021a) are two powerful classes of generative models that produce samples by inverting a diffusion process.
Revised: Probabilistic Models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021a) are two powerful classes of generative models that produce samples by inverting a diffusion process.

Original: These two classes have been uniﬁed under a single framework (Song et al., 2021b) and are widely known as diffusion models.
Revised: These two classes have been uniﬁed under a single framework (Song et al., 2021b) and are widely known as diffusion models.

Original: Diffusion modeling has found great success in a wide range of applications (Croitoru et al., 2022;
Revised: Diffusion modeling has found great success in a wide range of applications (Croitoru et al., 2022;

Original: Yang et al., 2022), including image (Saharia et al., 2022a; Ramesh et al., 2022; Rombach et al., 2022; Dhariwal & Nichol, 2021), audio (Kong et al., 2021; Richter et al., 2022; Serr ` a et al., 2022), video generation (Ho et al., 2022b), as well as solving inverse problems (Daras et al., 2022; Kadkhodaie & Simoncelli, 2021; Kawar et al., 2022; 2021; Jalal et al., 2021; Saharia et al., 2022b; Laumont et al., 2022; Whang et al., 2022; Chung et al., 2022).
Revised: Yang et al., 2022), including image (Saharia et al., 2022a; Ramesh et al., 2022; Rombach et al., 2022; Dhariwal & Nichol, 2021), audio (Kong et al., 2021; Richter et al., 2022; Serr ` a et al., 2022), video generation (Ho et al., 2022b), as well as solving inverse problems (Daras et al., 2022; Kadkhodaie & Simoncelli, 2021; Kawar et al., 2022; 2021; Jalal et al., 2021; Saharia et al., 2022b; Laumont et al., 2022; Whang et al., 2022; Chung et al., 2022).

Original: Karras et al.
Revised: Karras et al.

Original: (2022) analyze the design space of diffusion models.
Revised: (2022) analyze the design space of diffusion models.

Original: The authors identify three stages: i) the noise scheduling, ii) the network parametrization (each one leads to a different loss function), iii) the sampling algorithm.
Revised: The authors identify three stages: i) the noise scheduling, ii) the network parametrization (each one leads to a different loss function), iii) the sampling algorithm.

Original: We argue that there is one more important step: choosing how to corrupt .
Revised: We argue that there is one more important step: choosing how to corrupt .

Original: Typically, the diffusion is additive noise of different magnitudes (and sometimes input rescalings).
Revised: Typically, the diffusion is additive noise of different magnitudes (and sometimes input rescalings).

Original: There have been a few recent attempts to use different corruptions (Deasy et al., 2021;
Revised: There have been a few recent attempts to use different corruptions (Deasy et al., 2021;

Original: Hoogeboom et al., 2022a;b; Avrahami et al., 2022; Nachmani et al., 2021; Johnson et al., 2021;
Revised: Hoogeboom et al., 2022a;b; Avrahami et al., 2022; Nachmani et al., 2021; Johnson et al., 2021;

Original: Lee et al., 2022;
Revised: Lee et al., 2022;

Original: Ye et al., 2022), but the results are usually inferior to diffusion with additive noise.
Revised: Ye et al., 2022), but the results are usually inferior to diffusion with additive noise.

Original: Also, a common framework on how to properly design general corruption processes is missing.
Revised: Also, a common framework on how to properly design general corruption processes is missing.

Original: We present such a principled framework for learning to invert a general class of corruption processes.
Revised: We present such a principled framework for learning to invert a general class of corruption processes.

Original: We propose a new objective called Soft Score Matching that provably learns the score for any regular linear corruption process.
Revised: We propose a new objective called Soft Score Matching that provably learns the score for any regular linear corruption process.

Original: Soft Score Matching incorporates the ﬁltering process in the network and trains the model to predict a clean image that after corruption matches the diffused observation .
Revised: Soft Score Matching incorporates the ﬁltering process in the network and trains the model to predict a clean image that after corruption matches the diffused observation .

Original: Our theoretical results show that Soft Score Matching learns the score (i.e. likelihood gradients) for corruption processes that satisfy a regularity condition that we identify: the diffusion must transform any image into any other image with nonzero likelihood.
Revised: Our theoretical results show that Soft Score Matching learns the score (i.e. likelihood gradients) for corruption processes that satisfy a regularity condition that we identify: the diffusion must transform any image into any other image with nonzero likelihood.

Original: Using our method and Gaussian Blur paired with little noise as the diffusion mechanism, we achieve state-of-the-art FID on CelebA (FID 1.85) for linear diffusion models.
Revised: Using our method and Gaussian Blur paired with little noise as the diffusion mechanism, we achieve state-of-the-art FID on CelebA (FID 1.85) for linear diffusion models.

Original: We also show that our corruption process leads to generative models that are faster compared to vanilla Gaussian denoising diffusion.
Revised: We also show that our corruption process leads to generative models that are faster compared to vanilla Gaussian denoising diffusion.

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t= t
Revised: t= t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Our contributions: a) We propose a learning objective that: i) provably learns the score for a wide family of regular diffusion processes and ii) enables learning under limited randomness in the diffusion.
Revised: Our contributions: a) We propose a learning objective that: i) provably learns the score for a wide family of regular diffusion processes and ii) enables learning under limited randomness in the diffusion.

Original: b) We present a principled way to select the intermediate distributions.
Revised: b) We present a principled way to select the intermediate distributions.

Original: Our method minimizes the Wasserstein distance along the path from the initial to the ﬁnal distribution.
Revised: Our method minimizes the Wasserstein distance along the path from the initial to the ﬁnal distribution.

Original: c)
Revised: c)

Original: We propose a novel sampling method that we call Momentum Sampler: our sampler uses a convex combination of corruptions at different diffusion levels and is inspired by momentum methods in optimization.
Revised: We propose a novel sampling method that we call Momentum Sampler: our sampler uses a convex combination of corruptions at different diffusion levels and is inspired by momentum methods in optimization.

Original: d) We train models on CelebA and CIFAR-10.
Revised: d) We train models on CelebA and CIFAR-10.

Original: Our trained models on CelebA achieve a new state-of-the-art FID score of 1 . 85 for linear diffusion models while being signiﬁcantly faster compared to models trained with vanilla Gaussian denoising diffusion.
Revised: Our trained models on CelebA achieve a new state-of-the-art FID score of 1 . 85 for linear diffusion models while being signiﬁcantly faster compared to models trained with vanilla Gaussian denoising diffusion.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Diffusion models are generative models that produce samples by inverting a corruption process.
Revised: Diffusion models are generative models that produce samples by inverting a corruption process.

Original: The corruption level is typically indexed by a time t , with t = 0 corresponding to clean and t = 1 to fully corrupted images.
Revised: The corruption level is typically indexed by a time t , with t = 0 corresponding to clean and t = 1 to fully corrupted images.

Original: The diffusion process can be discrete or continuous.
Revised: The diffusion process can be discrete or continuous.

Original: The two general classes of diffusion models are Score-Based Models (Song & Ermon, 2019; 2020; Song et al., 2021b) and
Revised: The two general classes of diffusion models are Score-Based Models (Song & Ermon, 2019; 2020; Song et al., 2021b) and

Original: Denoising Diffusion Probabilistic Models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020).
Revised: Denoising Diffusion Probabilistic Models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020).

Original: The typical diffusion in score-based modeling is additive noise of increasing magnitude.
Revised: The typical diffusion in score-based modeling is additive noise of increasing magnitude.

Original: The perturbation kernel at time t is: q t ( x t | x 0 ) =
Revised: The perturbation kernel at time t is: q t ( x t | x 0 ) =

Original: N ( x t | µ = x 0 , Σ
Revised: N ( x t | µ = x 0 , Σ

Original: = σ 2 t I ) , where x 0 ∼ q 0 is a clean image.
Revised: = σ 2 t I ) , where x 0 ∼ q 0 is a clean image.

Original: Score models are trained with the Denoising Score Matching (DSM) objective:
Revised: Score models are trained with the Denoising Score Matching (DSM) objective:

Original: [Equation]
Revised: [Equation]

Original: where w t scales differently the weights of the inner objectives.
Revised: where w t scales differently the weights of the inner objectives.

Original: If we train for each noise level t independently, given enough data and model capacity, the network is guaranteed to recover the gradient of the log likelihood (Vincent, 2011), known as the score function.
Revised: If we train for each noise level t independently, given enough data and model capacity, the network is guaranteed to recover the gradient of the log likelihood (Vincent, 2011), known as the score function.

Original: In other words, the model s θ ( x t | t ) is trained such that: s θ ( x t | t )
Revised: In other words, the model s θ ( x t | t ) is trained such that: s θ ( x t | t )

Original: ≈ ∇ x t log q t ( x t ) .
Revised: ≈ ∇ x t log q t ( x t ) .

Original: In practice, we use parameter sharing and conditioning on time t to learn all the scores.
Revised: In practice, we use parameter sharing and conditioning on time t to learn all the scores.

Original: Once the model is trained, we start from a sample of the ﬁnal distribution, q 1 , and then use the learned score to gradually denoise it (Song &
Revised: Once the model is trained, we start from a sample of the ﬁnal distribution, q 1 , and then use the learned score to gradually denoise it (Song &

Original: Ermon, 2019; 2020).
Revised: Ermon, 2019; 2020).

Original: The ﬁnal variance σ 21 is selected to be very large such that the distribution qis approximately Gaussian, i.e. the signal to noise ratio tends to 0 .
Revised: The ﬁnal variance σ 21 is selected to be very large such that the distribution qis approximately Gaussian, i.e. the signal to noise ratio tends to 0 .

Original: DDPMs corrupt by rescaling the input images and by adding noise.
Revised: DDPMs corrupt by rescaling the input images and by adding noise.

Original: The corruption can be modelled with a Markov chain with perturbation kernel q t ( x t | x t − ∆ t ) =
Revised: The corruption can be modelled with a Markov chain with perturbation kernel q t ( x t | x t − ∆ t ) =

Original: N ( x t | µ = √ 1 − β t x t − ∆ t , Σ = β t I ) .
Revised: N ( x t | µ = √ 1 − β t x t − ∆ t , Σ = β t I ) .

Original: Typically, β 1 = 1 and hence q 1 = N (0 , I ) .
Revised: Typically, β 1 = 1 and hence q 1 = N (0 , I ) .

Original: DDPMs are also trained with the DSM objective which is derived by minimizing an evidence lower bound (ELBO)
Revised: DDPMs are also trained with the DSM objective which is derived by minimizing an evidence lower bound (ELBO)

Original: (Ho et al., 2020).
Revised: (Ho et al., 2020).

Original: In their seminal work, Song et al.
Revised: In their seminal work, Song et al.

Original: (2021b) observe that the diffusions of both Score-Based models and DDPMs can be expressed as solutions of Stochastic Differential Equations (SDEs) of the form:
Revised: (2021b) observe that the diffusions of both Score-Based models and DDPMs can be expressed as solutions of Stochastic Differential Equations (SDEs) of the form:

Original: [Equation]
Revised: [Equation]

Original: where w is the standard Wiener process.
Revised: where w is the standard Wiener process.

Original: Particularly, Score-Based models use: f ( x , t ) = 0 , g ( t ) =
Revised: Particularly, Score-Based models use: f ( x , t ) = 0 , g ( t ) =

Original: (cid:113)
Revised: (cid:113)

Original: d σ 2 t d t and DDPMs use: f ( x , t ) =
Revised: d σ 2 t d t and DDPMs use: f ( x , t ) =

Original: − 12 β t x , g ( t ) = √ β t .
Revised: − 12 β t x , g ( t ) = √ β t .

Original: As explained earlier, for Score-Based models we need large noise at the end for the ﬁnal distribution to approximate a normal distribution.
Revised: As explained earlier, for Score-Based models we need large noise at the end for the ﬁnal distribution to approximate a normal distribution.

Original: Hence, the corresponding SDE is named V ariance E xploding (VE) SDE (Song et al., 2021b).
Revised: Hence, the corresponding SDE is named V ariance E xploding (VE) SDE (Song et al., 2021b).

Original: DDPMs usually have a ﬁnal distribution of unit variance and hence their SDE is known as the
Revised: DDPMs usually have a ﬁnal distribution of unit variance and hence their SDE is known as the

Original: V ariance P reserving (VP) SDE (Song et al., 2021b).
Revised: V ariance P reserving (VP) SDE (Song et al., 2021b).

Original: Song et al.
Revised: Song et al.

Original: (2021b) also propose another SDE with bounded variance, the subVP-SDE, that experimentally yields better likelihoods.
Revised: (2021b) also propose another SDE with bounded variance, the subVP-SDE, that experimentally yields better likelihoods.

Original: For both Score-Based models and DDPMs, Eq.
Revised: For both Score-Based models and DDPMs, Eq.

Original: (2) is known as the Forward SDE.
Revised: (2) is known as the Forward SDE.

Original: This SDE is reversible Anderson (1982) and the Reverse SDE is given below:
Revised: This SDE is reversible Anderson (1982) and the Reverse SDE is given below:

Original: [Equation]
Revised: [Equation]

Original: where ¯ w is the reverse time standard Wiener process.
Revised: where ¯ w is the reverse time standard Wiener process.

Original: Typically, ∇ x log q t ( x ) is approximated by s θ ( x t | t ) and samples are generated by solving the Reverse SDE (Song et al., 2021b).
Revised: Typically, ∇ x log q t ( x ) is approximated by s θ ( x t | t ) and samples are generated by solving the Reverse SDE (Song et al., 2021b).

Original:  M ETHOD
Revised:  M ETHOD

Original: Our framework for training diffusion models with more general corruptions includes three components: i) the training objective, ii) the sampling, iii) the scheduling of the corruption mechanism.
Revised: Our framework for training diffusion models with more general corruptions includes three components: i) the training objective, ii) the sampling, iii) the scheduling of the corruption mechanism.

Original: 3. T RAINING O BJECTIVE
Revised: 3. T RAINING O BJECTIVE

Original: We study corruption processes of the form:
Revised: We study corruption processes of the form:

Original: [Equation]
Revised: [Equation]

Original: where C t : R n → R n is a deterministic linear operator, η t is a Wiener process, and s t is a nonnegative scalar controlling the noise level at time t , and x 0 ∼ q 0 ( x ) .
Revised: where C t : R n → R n is a deterministic linear operator, η t is a Wiener process, and s t is a nonnegative scalar controlling the noise level at time t , and x 0 ∼ q 0 ( x ) .

Original: We further denote with σ 2 t the variance of the noise at level t and we assume that it is a non-decreasing function of t .
Revised: We further denote with σ 2 t the variance of the noise at level t and we assume that it is a non-decreasing function of t .

Original: Unless stated otherwise, we assume that time is continuous and runs from t = 0 to t = 1 .
Revised: Unless stated otherwise, we assume that time is continuous and runs from t = 0 to t = 1 .

Original: Additionally, we assume that at t = 0 , we have C 

Revised: Additionally, we assume that at t = 0 , we have C

Original: = I n × n and σ 0 = 0 , i.e. t = 0 corresponds to natural images.
Revised: = I n × n and σ 0 = 0 , i.e. t = 0 corresponds to natural images.

Original: We also assume that recovering x 0 from x t is harder as t gets larger (i.e., entropy of q t ( x 0 | x t ) increases with t ).
Revised: We also assume that recovering x 0 from x t is harder as t gets larger (i.e., entropy of q t ( x 0 | x t ) increases with t ).

Original: Eq.
Revised: Eq.

Original: (4) deﬁnes a general class of diffusion processes, that includes (as special cases) the VE,
Revised: (4) deﬁnes a general class of diffusion processes, that includes (as special cases) the VE,

Original: VP and subVP SDEs used in Song et al.
Revised: VP and subVP SDEs used in Song et al.

Original: (2021b).
Revised: (2021b).

Original: Our diffusion is the sum of a deterministic linear corruption of x 0 and a stochastic part that progressively adds noise.
Revised: Our diffusion is the sum of a deterministic linear corruption of x 0 and a stochastic part that progressively adds noise.

Original: For any corruption process of this family, we are interested in learning the scores, i.e. ∇ x t log q t ( x t ) for all t .
Revised: For any corruption process of this family, we are interested in learning the scores, i.e. ∇ x t log q t ( x t ) for all t .

Original: For the vanilla Gaussian denoising diffusion, the celebrated result of Vincent (2011) shows that we only need access to the gradient of the conditional log-likelihood, ∇ x t log q t ( x t | x 0 ) , in order to learn the score, ∇ x t log q t ( x t ) .
Revised: For the vanilla Gaussian denoising diffusion, the celebrated result of Vincent (2011) shows that we only need access to the gradient of the conditional log-likelihood, ∇ x t log q t ( x t | x 0 ) , in order to learn the score, ∇ x t log q t ( x t ) .

Original: By revisiting the proof of Vincent (2011), we ﬁnd that this is actually true for a wide set of corruption processes, as long as some mild technical conditions are satisﬁed.
Revised: By revisiting the proof of Vincent (2011), we ﬁnd that this is actually true for a wide set of corruption processes, as long as some mild technical conditions are satisﬁed.

Original: In fact, the following general Theorem holds:
Revised: In fact, the following general Theorem holds:

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Let q 0 , q t be two distributions in R n .
Revised: Let q 0 , q t be two distributions in R n .

Original: Assume that all conditional distributions, q t ( x t | x 0 ) , are fully supported and differentiable in R n .
Revised: Assume that all conditional distributions, q t ( x t | x 0 ) , are fully supported and differentiable in R n .

Original: Let:
Revised: Let:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then, there is a universal constant C (that does not depend on θ ) such that: J 1 ( θ ) =
Revised: Then, there is a universal constant C (that does not depend on θ ) such that: J 1 ( θ ) =

Original: J 2 ( θ ) + C .
Revised: J 2 ( θ ) + C .

Original: Theorem 3.1 implies that minimizing the second function is equivalent to minimizing the ﬁrst one.
Revised: Theorem 3.1 implies that minimizing the second function is equivalent to minimizing the ﬁrst one.

Original: The second function is nothing else than the DSM objective.
Revised: The second function is nothing else than the DSM objective.

Original: Our main observation is that noise is not always necessary for learning the score using the DSM objective.
Revised: Our main observation is that noise is not always necessary for learning the score using the DSM objective.

Original: A necessary condition is that the corruption process gives non-zero probability to all x t for any image x 0 .
Revised: A necessary condition is that the corruption process gives non-zero probability to all x t for any image x 0 .

Original: This is easily achieved by adding Gaussian noise, but this is not the only option.
Revised: This is easily achieved by adding Gaussian noise, but this is not the only option.

Original: The proof of this Theorem is deferred in the Appendix and it is following the calculations of Vincent (2011).
Revised: The proof of this Theorem is deferred in the Appendix and it is following the calculations of Vincent (2011).

Original: Network parametrization.
Revised: Network parametrization.

Original: For the class of diffusion processes given by Eq.
Revised: For the class of diffusion processes given by Eq.

Original: (4), we have that: q t ( x t | x 0 ) =
Revised: (4), we have that: q t ( x t | x 0 ) =

Original: N (cid:0) x t ; µ = C t x 0 , Σ
Revised: N (cid:0) x t ; µ = C t x 0 , Σ

Original: = σ 2 t I (cid:1) and hence the objective becomes:
Revised: = σ 2 t I (cid:1) and hence the objective becomes:

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: As shown, the objective of the model is to predict the (normalized) difference, C t x 0 − x t , which is actually the noise, η t .
Revised: As shown, the objective of the model is to predict the (normalized) difference, C t x 0 − x t , which is actually the noise, η t .

Original: We argue that even though this objective is theoretically grounded, in many cases, it would not work in practice because we would need inﬁnite samples to actually learn the vector-ﬁeld ∇ x t log q t ( x t ) in a way that would allow sampling.
Revised: We argue that even though this objective is theoretically grounded, in many cases, it would not work in practice because we would need inﬁnite samples to actually learn the vector-ﬁeld ∇ x t log q t ( x t ) in a way that would allow sampling.

Original: Assume that the corruption process is blurring (at different levels) paired with additive noise of small magnitude.
Revised: Assume that the corruption process is blurring (at different levels) paired with additive noise of small magnitude.

Original: The objective written in Eq.
Revised: The objective written in Eq.

Original: (7) learns the distributions of blurry (and slightly noisy images) by just removing noise.
Revised: (7) learns the distributions of blurry (and slightly noisy images) by just removing noise.

Original: Hence, in practice we might only learn these distributions locally (around the blurry images) and hence we might not be able to reduce the blurriness.
Revised: Hence, in practice we might only learn these distributions locally (around the blurry images) and hence we might not be able to reduce the blurriness.

Original: This point might be better understood after we present our Sampling Method in Section 3.2.
Revised: This point might be better understood after we present our Sampling Method in Section 3.2.

Original: To account for this problem, we propose a network reparametrization which leverages that we know the linear corruption mechanism, C t .
Revised: To account for this problem, we propose a network reparametrization which leverages that we know the linear corruption mechanism, C t .

Original: Speciﬁcally, we propose the following parametrization:
Revised: Speciﬁcally, we propose the following parametrization:

Original: [Equation]
Revised: [Equation]

Original: Crucially, the network incorporates the corruption process.
Revised: Crucially, the network incorporates the corruption process.

Original: The loss becomes:
Revised: The loss becomes:

Original: [Equation]
Revised: [Equation]

Original: When C t is a blurring matrix, this loss function is the MSE between the blurred prediction of h θ and the blurred clean image.
Revised: When C t is a blurring matrix, this loss function is the MSE between the blurred prediction of h θ and the blurred clean image.

Original: Finally, as observed in previous works (Song & Ermon, 2019; Ho et al., 2020; Karras et al., 2022), the optimization landscape becomes smoother when we are predicting the residual, instead of the clean image directly.
Revised: Finally, as observed in previous works (Song & Ermon, 2019; Ho et al., 2020; Karras et al., 2022), the optimization landscape becomes smoother when we are predicting the residual, instead of the clean image directly.

Original: This corresponds to the additional reparametrization:
Revised: This corresponds to the additional reparametrization:

Original: [Equation]
Revised: [Equation]

Original: which leads to the ﬁnal form of our loss function:
Revised: which leads to the ﬁnal form of our loss function:

Original: [Equation]
Revised: [Equation]

Original: (11) where r t is the residual with respect to the clean image, i.e. r t = x 0 − x t .
Revised: (11) where r t is the residual with respect to the clean image, i.e. r t = x 0 − x t .

Original: Following prior work, we use a single network conditioned on time t that is optimized for all L ( t ) .
Revised: Following prior work, we use a single network conditioned on time t that is optimized for all L ( t ) .

Original: Hence, the total loss is:
Revised: Hence, the total loss is:

Original: [Equation]
Revised: [Equation]

Original: where the weights are usually chosen to be 1 or 1 /σ 2 t (Karras et al., 2022; Kingma et al., 2021).
Revised: where the weights are usually chosen to be 1 or 1 /σ 2 t (Karras et al., 2022; Kingma et al., 2021).

Original: We call our training objective Soft Score Matching .
Revised: We call our training objective Soft Score Matching .

Original: The name is inspired from “soft ﬁltering” a term used in photography to denote an image ﬁlter that removes ﬁne details (e.g., blur, fading, etc).
Revised: The name is inspired from “soft ﬁltering” a term used in photography to denote an image ﬁlter that removes ﬁne details (e.g., blur, fading, etc).

Original: As in the Denoising Score Matching, the network is essentially trained to predict the residual to the clean image, but in our case, the loss is in the ﬁltered space.
Revised: As in the Denoising Score Matching, the network is essentially trained to predict the residual to the clean image, but in our case, the loss is in the ﬁltered space.

Original: When there is no ﬁltering matrix, i.e.
Revised: When there is no ﬁltering matrix, i.e.

Original: C t =
Revised: C t =

Original: I , we recover the DSM objective used in (Song & Ermon, 2019; 2020; Song et al., 2021b).
Revised: I , we recover the DSM objective used in (Song & Ermon, 2019; 2020; Song et al., 2021b).

Original: Comparison with objectives used in other works.
Revised: Comparison with objectives used in other works.

Original: Previous (Anonymous, 2022) or concurrent (Bansal et al., 2022;
Revised: Previous (Anonymous, 2022) or concurrent (Bansal et al., 2022;

Original: Rissanen et al., 2022; Hoogeboom & Salimans, 2022) works that consider different degradations than Gaussian Diffusion, use the heuristic objective of predicting the clean image, i.e. they minimize: || φ θ ( x t | t ) − r t || .
Revised: Rissanen et al., 2022; Hoogeboom & Salimans, 2022) works that consider different degradations than Gaussian Diffusion, use the heuristic objective of predicting the clean image, i.e. they minimize: || φ θ ( x t | t ) − r t || .

Original: This is actually an upper-bound on our loss, i.e. || C t ( φ θ ( x t | t ) − r t ) ||
Revised: This is actually an upper-bound on our loss, i.e. || C t ( φ θ ( x t | t ) − r t ) ||

Original: ≤ || C t |||| φ θ ( x t | t ) − r t || .
Revised: ≤ || C t |||| φ θ ( x t | t ) − r t || .

Original: Since the spectral norm, || C t || , is ﬁxed, one can optimize for the upper-bound by minimizing || φ θ ( x t | t ) − r t || .
Revised: Since the spectral norm, || C t || , is ﬁxed, one can optimize for the upper-bound by minimizing || φ θ ( x t | t ) − r t || .

Original: Instead, Soft Score Matching optimizes directly for learning the score.
Revised: Instead, Soft Score Matching optimizes directly for learning the score.

Original: Experimentally, Soft Score Matching outperforms (ours
Revised: Experimentally, Soft Score Matching outperforms (ours

Original: FID: 1 . 85 , theirs: 5 . 91 ), under the exact same setting, this simple baseline (see Experiments).
Revised: FID: 1 . 85 , theirs: 5 . 91 ), under the exact same setting, this simple baseline (see Experiments).

Original: 3. S AMPLING
Revised: 3. S AMPLING

Original: Require: p 1 , φ θ , C t , σ t , ∆ t x 1 ∼ p 1 ( x ) for t = 1 to 0 with step − ∆ t do ˆ x 0 =
Revised: Require: p 1 , φ θ , C t , σ t , ∆ t x 1 ∼ p 1 ( x ) for t = 1 to 0 with step − ∆ t do ˆ x 0 =

Original: φ θ ( x t | t ) + x t η t ∼ N ( 0 , I ) x t − ∆ t ← C t − ∆ t ˆ x 0 + σ t − ∆ t η t end for return x
Revised: φ θ ( x t | t ) + x t η t ∼ N ( 0 , I ) x t − ∆ t ← C t − ∆ t ˆ x 0 + σ t − ∆ t η t end for return x

Original: Naive Sampler.
Revised: Naive Sampler.

Original: Once the model is trained, we need a way to generate samples.
Revised: Once the model is trained, we need a way to generate samples.

Original: The simplest idea is to use recursively our model, φ θ ( x t | t ) , to get estimates of the clean image, ˆ x 0 .
Revised: The simplest idea is to use recursively our model, φ θ ( x t | t ) , to get estimates of the clean image, ˆ x 0 .

Original: To move from corruption level t to corruption level t − ∆ t , we feed the image x t to the model to get an estimate of the clean image, ˆ x 0 , and then corrupt back to level t − ∆ t .
Revised: To move from corruption level t to corruption level t − ∆ t , we feed the image x t to the model to get an estimate of the clean image, ˆ x 0 , and then corrupt back to level t − ∆ t .

Original: This idea is shown in Algorithm 1.
Revised: This idea is shown in Algorithm 1.

Original: Momentum Sampler.
Revised: Momentum Sampler.

Original: As we will show in the Experiments section, the naive sampler we presented above leads to generated images that lack diversity.
Revised: As we will show in the Experiments section, the naive sampler we presented above leads to generated images that lack diversity.

Original: We propose an simple, yet novel, alternative method for sampling from the general linear diffusion model presented in Eq.
Revised: We propose an simple, yet novel, alternative method for sampling from the general linear diffusion model presented in Eq.

Original: Our method is inspired by the continuous formulation of diffusion models that is introduced in Song et al.
Revised: Our method is inspired by the continuous formulation of diffusion models that is introduced in Song et al.

Original: (2021b).
Revised: (2021b).

Original: The ﬁrst step is to ﬁnd a Markovian stochastic process that is “close” to the non-Markovian corruption process of Eq.
Revised: The ﬁrst step is to ﬁnd a Markovian stochastic process that is “close” to the non-Markovian corruption process of Eq.

Original: Consider the following SDE:
Revised: Consider the following SDE:

Original: [Equation]
Revised: [Equation]

Original: (13) where w is the standard Wiener process.
Revised: (13) where w is the standard Wiener process.

Original: This is a special case of the It ˆ o SDE: d x = f ( x , t )d t + g ( t )d w , that appears in Song et al.
Revised: This is a special case of the It ˆ o SDE: d x = f ( x , t )d t + g ( t )d w , that appears in Song et al.

Original: (2021b), i.e. f ( x , t ) = ˙
Revised: (2021b), i.e. f ( x , t ) = ˙

Original: C t E [ x 0 | x t ] and g ( t ) =
Revised: C t E [ x 0 | x t ] and g ( t ) =

Original: (cid:113)
Revised: (cid:113)

Original: d( σ 2 t ) d t .
Revised: d( σ 2 t ) d t .

Original: Crucially, E [ x 0 | x t ] does not depend on previous values of x t and our SDE is indeed an Ito SDE.
Revised: Crucially, E [ x 0 | x t ] does not depend on previous values of x t and our SDE is indeed an Ito SDE.

Original: To build some intuition, it is useful to think of the toy setting where the dataset contains one image, α ∈ R n .
Revised: To build some intuition, it is useful to think of the toy setting where the dataset contains one image, α ∈ R n .

Original: Under this setting, the Markovian corruption described by the SDE of Eq.
Revised: Under this setting, the Markovian corruption described by the SDE of Eq.

Original: (13) has the same marginal distributions as the process of Eq.
Revised: (13) has the same marginal distributions as the process of Eq.

Original: As we explain in the Appendix (Section E), in the general case, the SDE (13) introduces an approximation error with respect to the corruption process of Eq.
Revised: As we explain in the Appendix (Section E), in the general case, the SDE (13) introduces an approximation error with respect to the corruption process of Eq.

Original: (4) – the former uses the conditional expectation, E [ x 0 | x t ] , while the latter corrupts directly x 0 .
Revised: (4) – the former uses the conditional expectation, E [ x 0 | x t ] , while the latter corrupts directly x 0 .

Original: Nevertheless, the approximation error for the considered corruptions seems to be small, given the experimental success of the derived sampler.
Revised: Nevertheless, the approximation error for the considered corruptions seems to be small, given the experimental success of the derived sampler.

Original: Intuitively, this is because for low corruption levels, the conditional expectation and x 0 are close, while for high corruption levels, the distance is contracted by the multiplication with the corruption matrix C t (see also Appendix, Section E).
Revised: Intuitively, this is because for low corruption levels, the conditional expectation and x 0 are close, while for high corruption levels, the distance is contracted by the multiplication with the corruption matrix C t (see also Appendix, Section E).

Original: Eq.
Revised: Eq.

Original: (13) describes a reversible diffusion process (Anderson, 1982).
Revised: (13) describes a reversible diffusion process (Anderson, 1982).

Original: The reverse is also an Ito SDE:
Revised: The reverse is also an Ito SDE:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ¯ w is a standard Wiener process when time ﬂows backwards from t = 1 to t = 0 .
Revised: where ¯ w is a standard Wiener process when time ﬂows backwards from t = 1 to t = 0 .

Original: In practice, to solve Eq.
Revised: In practice, to solve Eq.

Original: (14), we discretize the SDE (i.e., apply Euler-Maruyama, and approximate the function derivatives with ﬁnite differences).
Revised: (14), we discretize the SDE (i.e., apply Euler-Maruyama, and approximate the function derivatives with ﬁnite differences).

Original: The Euler-Maruyama discretization is given below:
Revised: The Euler-Maruyama discretization is given below:

Original: [Equation]
Revised: [Equation]

Original: where η ∼ N ( 0 , I )
Revised: where η ∼ N ( 0 , I )

Original: In this update Equation, there are two unknowns: the conditional expectation, E [ x 0 | x t ] , and the score function, ∇ x t log q t ( x t ) .
Revised: In this update Equation, there are two unknowns: the conditional expectation, E [ x 0 | x t ] , and the score function, ∇ x t log q t ( x t ) .

Original: We show that these are actually connected, through the Tweedie’s (Efron, 2011; Robbins, 1956; Stein, 1981) formula.
Revised: We show that these are actually connected, through the Tweedie’s (Efron, 2011; Robbins, 1956; Stein, 1981) formula.

Original: Speciﬁcally, it holds that:
Revised: Speciﬁcally, it holds that:

Original: [Equation]
Revised: [Equation]

Original: The proof is given for completeness in the Appendix, Lemma A.1.
Revised: The proof is given for completeness in the Appendix, Lemma A.1.

Original: To estimate ∇ x t log q t ( x t ) we use our model that provably learns the score according to Theorem 3.1.
Revised: To estimate ∇ x t log q t ( x t ) we use our model that provably learns the score according to Theorem 3.1.

Original: Putting everything together:
Revised: Putting everything together:

Original: [Equation]
Revised: [Equation]

Original: Our sampler is summarized in Algorithm 2.
Revised: Our sampler is summarized in Algorithm 2.

Original: Essentially, there is one update for deblurring and one for denoising.
Revised: Essentially, there is one update for deblurring and one for denoising.

Original: At the core of this update equation, is the prediction of the clean image, ˆ x 0 .
Revised: At the core of this update equation, is the prediction of the clean image, ˆ x 0 .

Original: Once
Revised: Once

Original: Require: p 1 , φ θ , C t , σ t , ∆ t x 1 ∼ p 1 ( x ) for t = 1 to 0 with step − ∆ t do ˆ x 0 =
Revised: Require: p 1 , φ θ , C t , σ t , ∆ t x 1 ∼ p 1 ( x ) for t = 1 to 0 with step − ∆ t do ˆ x 0 =

Original: φ θ ( x t | t ) + x t clean image.
Revised: φ θ ( x t | t ) + x t clean image.

Original: ˆ y t ← C t ˆ x at
Revised: ˆ y t ← C t ˆ x at

Original: η t ∼ N ( 0 , I ) ˆ (cid:15) t ← ˆ y t − x t z t − ∆ t ← x t − ( σ 2 t − ∆ t − σ 2 t ) σ 2 t ˆ (cid:15) t + (cid:113) σ 2 t − σ 2 t − ∆ t η t t −
Revised: η t ∼ N ( 0 , I ) ˆ (cid:15) t ← ˆ y t − x t z t − ∆ t ← x t − ( σ 2 t − ∆ t − σ 2 t ) σ 2 t ˆ (cid:15) t + (cid:113) σ 2 t − σ 2 t − ∆ t η t t −

Original: ˆ y t − ∆ t ← C t − ∆ t ˆ x
Revised: ˆ y t − ∆ t ← C t − ∆ t ˆ x

Original: −
Revised: −

Original: x t − ∆ t ← z t − ∆ t + (ˆ y t − ∆ t − ˆ y t ) end return
Revised: x t − ∆ t ← z t − ∆ t + (ˆ y t − ∆ t − ˆ y t ) end return

Original: the clean image is predicted, we blur it back to two different corruption levels, t and t − ∆ t .
Revised: the clean image is predicted, we blur it back to two different corruption levels, t and t − ∆ t .

Original: The deblurring gradient is the residual between the blurred images at levels t − ∆ t and t .
Revised: The deblurring gradient is the residual between the blurred images at levels t − ∆ t and t .

Original: Interestingly, the denoising update is the same as the one used in typical score-based models (that only use additive noise).
Revised: Interestingly, the denoising update is the same as the one used in typical score-based models (that only use additive noise).

Original: In fact, if there is no blur ( C t = I ), our sampler becomes exactly the sampler used for the
Revised: In fact, if there is no blur ( C t = I ), our sampler becomes exactly the sampler used for the

Original: Variance Exploding (VE) SDE in Song et al.
Revised: Variance Exploding (VE) SDE in Song et al.

Original: (2021b).
Revised: (2021b).

Original: We call our sampler Momentum Sampler because we can think of it as a generalization of the update of the Naive Sampler, where there is a momentum term.
Revised: We call our sampler Momentum Sampler because we can think of it as a generalization of the update of the Naive Sampler, where there is a momentum term.

Original: To understand this better, we look at the setting where there is no noise.
Revised: To understand this better, we look at the setting where there is no noise.

Original: Then, the update rule of the Momentum Sampler is:
Revised: Then, the update rule of the Momentum Sampler is:

Original: [Equation]
Revised: [Equation]

Original: As seen, the ﬁrst term is what the Naive Sampler would use to update the image at level t and the second term is what the Naive Sampler would use to update the image at level t − ∆ t .
Revised: As seen, the ﬁrst term is what the Naive Sampler would use to update the image at level t and the second term is what the Naive Sampler would use to update the image at level t − ∆ t .

Original: If these two directions are aligned, then the gradient ∆ x t is small.
Revised: If these two directions are aligned, then the gradient ∆ x t is small.

Original: Hence, there is a notion of momentum, analogous to how the term is used in classical optimization.
Revised: Hence, there is a notion of momentum, analogous to how the term is used in classical optimization.

Original: In the Appendix, Section B.2, we also present a DDIM-type (Song et al., 2021a) sampler for which the momentum term also appears.
Revised: In the Appendix, Section B.2, we also present a DDIM-type (Song et al., 2021a) sampler for which the momentum term also appears.

Original: Probability Flow Momentum Sampler.
Revised: Probability Flow Momentum Sampler.

Original: The update rule of our sampler was derived using the discretization of the backward SDE of our corruption, given in Eq.
Revised: The update rule of our sampler was derived using the discretization of the backward SDE of our corruption, given in Eq.

Original: Similarly to Song et al.
Revised: Similarly to Song et al.

Original: (2021b), we can also consider the Ordinary Differential Equation (ODE) associated with this SDE:
Revised: (2021b), we can also consider the Ordinary Differential Equation (ODE) associated with this SDE:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Again, we can approximate ˙
Revised: Again, we can approximate ˙

Original: C t E [ x 0 | x t ] , ∇ x t log q t ( x t ) with our trained network and get a deterministic version of the Momentum Sampler, which we call Probability Flow Momentum Sampler, as in Song et al.
Revised: C t E [ x 0 | x t ] , ∇ x t log q t ( x t ) with our trained network and get a deterministic version of the Momentum Sampler, which we call Probability Flow Momentum Sampler, as in Song et al.

Original: (2021b).
Revised: (2021b).

Original: We detail our derivations in the Appendix, Section B.1.
Revised: We detail our derivations in the Appendix, Section B.1.

Original: 3. S CHEDULING
Revised: 3. S CHEDULING

Original: The last piece of our framework is how to choose the corruption levels, i.e. the scheduling of the diffusion.
Revised: The last piece of our framework is how to choose the corruption levels, i.e. the scheduling of the diffusion.

Original: For example, for Gaussian Blur, the scheduling decides how much time is spent in the diffusion in the very blurry, somewhat blurry and almost no blurry regimes.
Revised: For example, for Gaussian Blur, the scheduling decides how much time is spent in the diffusion in the very blurry, somewhat blurry and almost no blurry regimes.

Original: We provide a principled way of choosing the corruption levels for arbitrary corruption processes.
Revised: We provide a principled way of choosing the corruption levels for arbitrary corruption processes.

Original: Let D 0 the distribution of real images and D 1 be a known distribution that we know how to sample from, e.g. a Normal Distribution.
Revised: Let D 0 the distribution of real images and D 1 be a known distribution that we know how to sample from, e.g. a Normal Distribution.

Original: In the design phase of score-based modeling, the goal is to choose a set of intermediate distributions, {D t } , that smoothly transform images from D 0 to samples from the distribution D 1 .
Revised: In the design phase of score-based modeling, the goal is to choose a set of intermediate distributions, {D t } , that smoothly transform images from D 0 to samples from the distribution D 1 .

Original: Let Θ = { θ 1 , θ 2 , ..., θ k , ... } be the space of diffusion parameters, i.e. each θ i corresponds to a distribution D θ i .
Revised: Let Θ = { θ 1 , θ 2 , ..., θ k , ... } be the space of diffusion parameters, i.e. each θ i corresponds to a distribution D θ i .

Original: In the case of blur for example, θ i controls how much we blur the image.
Revised: In the case of blur for example, θ i controls how much we blur the image.

Original: Let also M : X × X
Revised: Let also M : X × X

Original: → R be a metric that measures distances between distributions, e.g.
Revised: → R be a metric that measures distances between distributions, e.g.

Original: M might be the Wasserstein Distance of distributions with support X .
Revised: M might be the Wasserstein Distance of distributions with support X .

Original: We construct a weighted graph G (cid:15) with the nodes being the distributions and the weights given by:
Revised: We construct a weighted graph G (cid:15) with the nodes being the distributions and the weights given by:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For a ﬁxed (cid:15) , we choose the distributions that minimize the cost of the path between D 0 and D 1 .
Revised: For a ﬁxed (cid:15) , we choose the distributions that minimize the cost of the path between D 0 and D 1 .

Original: The parameter (cid:15) expresses the power of the best neural network we can train to reverse one step of the diffusion.
Revised: The parameter (cid:15) expresses the power of the best neural network we can train to reverse one step of the diffusion.

Original: If (cid:15) = ∞ , then for any metric M , the shortest path is to go directly from D 0 to D 1 .
Revised: If (cid:15) = ∞ , then for any metric M , the shortest path is to go directly from D 0 to D 1 .

Original: However, it is impossible to denoise in one step a super noisy image.
Revised: However, it is impossible to denoise in one step a super noisy image.

Original: Hence, we need to go through many intermediate steps, which is forced by setting (cid:15) to a smaller value.
Revised: Hence, we need to go through many intermediate steps, which is forced by setting (cid:15) to a smaller value.

Original: As we increase the number of the candidate distributions we are getting closer to ﬁnding the geodesic between D 0 and D 1 .
Revised: As we increase the number of the candidate distributions we are getting closer to ﬁnding the geodesic between D 0 and D 1 .

Original: However, the computational cost of the method increases since we need to estimate all the pairwise distances M
Revised: However, the computational cost of the method increases since we need to estimate all the pairwise distances M

Original: ( D θ i , D θ j ) .
Revised: ( D θ i , D θ j ) .

Original: In practice, we use a relatively small number of candidate distributions, e.g. T = 256 and once the path is found, we do linear interpolation to extend to the continuous case.
Revised: In practice, we use a relatively small number of candidate distributions, e.g. T = 256 and once the path is found, we do linear interpolation to extend to the continuous case.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our method in CelebA- 64 and CIFAR-10.
Revised: We evaluate our method in CelebA- 64 and CIFAR-10.

Original: We show that by just changing the corruption mechanism (and using our framework for scheduling, learning and sampling)
Revised: We show that by just changing the corruption mechanism (and using our framework for scheduling, learning and sampling)

Original: we signiﬁcantly improve the FID score and reduce the sampling time.
Revised: we signiﬁcantly improve the FID score and reduce the sampling time.

Original: We use the architecture and the training hyperparameters from Song et al.
Revised: We use the architecture and the training hyperparameters from Song et al.

Original: (2021b) (full details can be found in the Appendix).
Revised: (2021b) (full details can be found in the Appendix).

Original: For most of our experiments, we use Gaussian Blur as our primary corruption mechanism.
Revised: For most of our experiments, we use Gaussian Blur as our primary corruption mechanism.

Original: To illustrate that our method works more generally, we also show results with masking which is discussed separately later.
Revised: To illustrate that our method works more generally, we also show results with masking which is discussed separately later.

Original: Consistent with the description of our method, our deterministic corruptions are also paired with additive low magnitude noise.
Revised: Consistent with the description of our method, our deterministic corruptions are also paired with additive low magnitude noise.

Original: This is required by our theoretical results, otherwise the conditional log-likelihood, log q ( x t | x 0 ) would be undeﬁned.
Revised: This is required by our theoretical results, otherwise the conditional log-likelihood, log q ( x t | x 0 ) would be undeﬁned.

Original: We also ﬁnd the addition of noise beneﬁcial in practice (see Appendix F.1.1 for ablation studies on the role of noise).
Revised: We also ﬁnd the addition of noise beneﬁcial in practice (see Appendix F.1.1 for ablation studies on the role of noise).

Original: Scheduling.
Revised: Scheduling.

Original: We use the standard geometric scheduling for the noise (Song & Ermon, 2019; 2020;
Revised: We use the standard geometric scheduling for the noise (Song & Ermon, 2019; 2020;

Original: Song et al., 2021b)
Revised: Song et al., 2021b)

Original: and use the methodology described in Section 3.3 to select the blur levels.
Revised: and use the methodology described in Section 3.3 to select the blur levels.

Original: We use the Wasserstein distance as the metric M to measure how close are the different distributions.
Revised: We use the Wasserstein distance as the metric M to measure how close are the different distributions.

Original: To clearly illustrate the switch to a different corruption, our diffusion has an initial stage that increases the noise (with no blur) and then we ﬁx the noise (to a small value, e.g. σ = 0 . 1 ) and change (using our scheduling framework) the amount of blur.
Revised: To clearly illustrate the switch to a different corruption, our diffusion has an initial stage that increases the noise (with no blur) and then we ﬁx the noise (to a small value, e.g. σ = 0 . 1 ) and change (using our scheduling framework) the amount of blur.

Original: Our diffusion spends less than 20% of the total time in the initial stage that increases the noise.
Revised: Our diffusion spends less than 20% of the total time in the initial stage that increases the noise.

Original: We ablate those choices in Section F.1 of the Appendix.
Revised: We ablate those choices in Section F.1 of the Appendix.

Original: One important property of our framework is that the scheduling is dataset speciﬁc.
Revised: One important property of our framework is that the scheduling is dataset speciﬁc.

Original: Intuitively, the way we corrupt depends on the nature of the data we are modelling.
Revised: Intuitively, the way we corrupt depends on the nature of the data we are modelling.

Original: The interested reader can ﬁnd the found schedulings for each dataset in Figure 6 of the Appendix.
Revised: The interested reader can ﬁnd the found schedulings for each dataset in Figure 6 of the Appendix.

Original: Results.
Revised: Results.

Original: We train our networks on CelebA- 64 and CIFAR-10 using the found schedulings and the training objective of Eq.
Revised: We train our networks on CelebA- 64 and CIFAR-10 using the found schedulings and the training objective of Eq.

Original: We start by showing uncurated samples of our models in Figure 2.
Revised: We start by showing uncurated samples of our models in Figure 2.

Original: The generated images have high diversity and ﬁdelity in both datasets.
Revised: The generated images have high diversity and ﬁdelity in both datasets.

Original: We compare the FID (Heusel et al., 2017) obtained by our method on CelebA with many natural baselines that use any of the VE,
Revised: We compare the FID (Heusel et al., 2017) obtained by our method on CelebA with many natural baselines that use any of the VE,

Original: VP or subVP SDEs.
Revised: VP or subVP SDEs.

Original: Speciﬁcally, we compare against DDPM (Ho et al., 2020) that uses the VP SDE,
Revised: Speciﬁcally, we compare against DDPM (Ho et al., 2020) that uses the VP SDE,

Original: DDIM (Song et al., 2021a) that uses the same model but with a different sampler, DDPM++ (Kim et al., 2022b) that is the state-of-the-art model for VP-SDE and the NCSN++ models (Song et al., 2021b) trained with the VE and subVP-SDEs.
Revised: DDIM (Song et al., 2021a) that uses the same model but with a different sampler, DDPM++ (Kim et al., 2022b) that is the state-of-the-art model for VP-SDE and the NCSN++ models (Song et al., 2021b) trained with the VE and subVP-SDEs.

Original: For a fair comparison, we only use reported numbers in published papers for the baselines and we do not rerun them ourselves.
Revised: For a fair comparison, we only use reported numbers in published papers for the baselines and we do not rerun them ourselves.

Original: Our model achieves state-of-the-art FID score, 1 . 85 , in CelebA, outperforming all the other methods.
Revised: Our model achieves state-of-the-art FID score, 1 . 85 , in CelebA, outperforming all the other methods.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: For CIFAR-10, we obtain FID score 3 .
Revised: For CIFAR-10, we obtain FID score 3 .

Original: 86 with our Probability Flow Momentum Sampler and 3 .
Revised: 86 with our Probability Flow Momentum Sampler and 3 .

Original: 91 with our Momentum Sampler.
Revised: 91 with our Momentum Sampler.

Original: We summarize the results in Table 2.
Revised: We summarize the results in Table 2.

Original: Our best FID is competitive with similar samplers to similar methods, e.g. with NCSN++ (VE SDE) with
Revised: Our best FID is competitive with similar samplers to similar methods, e.g. with NCSN++ (VE SDE) with

Original: Reverse SDE sampling.
Revised: Reverse SDE sampling.

Original: However, it is behind other state-of-the-art models such as LSGM (Vahdat et al., 2021) and DDPM (Ho et al., 2020).
Revised: However, it is behind other state-of-the-art models such as LSGM (Vahdat et al., 2021) and DDPM (Ho et al., 2020).

Original: We believe that this performance gap can be decreased in the future by further research in the area of diffusion models with general corruptions.
Revised: We believe that this performance gap can be decreased in the future by further research in the area of diffusion models with general corruptions.

Original: Our method is superior in sampling time , for both CIFAR-10 and CelebA. Figure 3 shows how FID changes based on the Number of Function Evaluations (NFEs).
Revised: Our method is superior in sampling time , for both CIFAR-10 and CelebA. Figure 3 shows how FID changes based on the Number of Function Evaluations (NFEs).

Original: Our method requires signiﬁcant less steps to achieve the same or better quality than NCSN++ (VE SDE)
Revised: Our method requires signiﬁcant less steps to achieve the same or better quality than NCSN++ (VE SDE)

Original: (Song et al., 2021b), using the same architecture and training hyperparameters (FID values taken from (Ma et al., 2022)).
Revised: (Song et al., 2021b), using the same architecture and training hyperparameters (FID values taken from (Ma et al., 2022)).

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Ablation Study for Sampling.
Revised: Ablation Study for Sampling.

Original: For all the results we presented so far, we used the Momentum
Revised: For all the results we presented so far, we used the Momentum

Original: Sampler that we introduced in Section 3.2 and Algorithm 2.
Revised: Sampler that we introduced in Section 3.2 and Algorithm 2.

Original: In this section, we ablate the choice of the sampler.
Revised: In this section, we ablate the choice of the sampler.

Original: Speciﬁcally, we compare with the intuitive Naive Sampler described in Algorithm 1.
Revised: Speciﬁcally, we compare with the intuitive Naive Sampler described in Algorithm 1.

Original: We show that the choice of the sampler has a dramatic effect in the quality and the diversity of the generated images.
Revised: We show that the choice of the sampler has a dramatic effect in the quality and the diversity of the generated images.

Original: Results are shown in Figure 4.
Revised: Results are shown in Figure 4.

Original: The images from the Naive Sampler seem repetitive and lack details.
Revised: The images from the Naive Sampler seem repetitive and lack details.

Original: This is reﬂected in the poor FID score.
Revised: This is reﬂected in the poor FID score.

Original: The Momentum Sampler leads to images with greater variety and detail, dramatically improving FID from 27 .
Revised: The Momentum Sampler leads to images with greater variety and detail, dramatically improving FID from 27 .

Original: 82 to 1 .
Revised: 82 to 1 .

Original: [Figure]
Revised: [Figure]

Original: Training Objective.
Revised: Training Objective.

Original: Concurrent works (Hoogeboom & Salimans, 2022; Bansal et al., 2022) that also consider different corruption processes have used as their loss a simple objective: given an input image, they predict the residual to the clean.
Revised: Concurrent works (Hoogeboom & Salimans, 2022; Bansal et al., 2022) that also consider different corruption processes have used as their loss a simple objective: given an input image, they predict the residual to the clean.

Original: As explained in Section 3.1, this is equivalent to optimizing for an upper-bound of the Soft Score Matching objective (which is minimizing the error to the score-function, see Theorem 3.1).
Revised: As explained in Section 3.1, this is equivalent to optimizing for an upper-bound of the Soft Score Matching objective (which is minimizing the error to the score-function, see Theorem 3.1).

Original: We show that if we use our exact pipeline (model, scheduling, sampler, training hyperparameters and so on) and we replace our Soft Score matching
Revised: We show that if we use our exact pipeline (model, scheduling, sampler, training hyperparameters and so on) and we replace our Soft Score matching

Original: Loss with the loss introduced in Bansal et al.
Revised: Loss with the loss introduced in Bansal et al.

Original: (2022), FID score increases from 1 . 85 to 5 .
Revised: (2022), FID score increases from 1 . 85 to 5 .

Original: This experiment shows the effectiveness of the Soft Score Matching objective.
Revised: This experiment shows the effectiveness of the Soft Score Matching objective.

Original: Ablation Studies for Scheduling.
Revised: Ablation Studies for Scheduling.

Original: We perform extensive ablations on the scheduling to understand the role of noise in the framework and whether our scheme outperforms other natural baselines.
Revised: We perform extensive ablations on the scheduling to understand the role of noise in the framework and whether our scheme outperforms other natural baselines.

Original: The results are detailed in the Appendix, Section F.1.
Revised: The results are detailed in the Appendix, Section F.1.

Original: Our main ﬁndings are: i) the Momentum Sampler works even if noise and blur are changing simultaneously (i.e. for schedulings with non-ﬁxed noise), ii) lowering the maximum value of noise leads to important performance degradation – for very small noise the method completely fails and iii) our found scheduling outperforms signiﬁcantly (baseline FID: 8 . 35 , ours: 1 . 85 ) a natural baseline that sets blur parameters such that MSE between the corrupted and the clean
Revised: Our main ﬁndings are: i) the Momentum Sampler works even if noise and blur are changing simultaneously (i.e. for schedulings with non-ﬁxed noise), ii) lowering the maximum value of noise leads to important performance degradation – for very small noise the method completely fails and iii) our found scheduling outperforms signiﬁcantly (baseline FID: 8 . 35 , ours: 1 . 85 ) a natural baseline that sets blur parameters such that MSE between the corrupted and the clean

Original: Masking Diffusion Models.
Revised: Masking Diffusion Models.

Original: To show the generality of our framework, we also train models with (discrete) masking diffusion paired with noise.
Revised: To show the generality of our framework, we also train models with (discrete) masking diffusion paired with noise.

Original: Figure 1 (top 2 rows) shows the forward and the (learned) reverse process for blur on the left and masking on the right.
Revised: Figure 1 (top 2 rows) shows the forward and the (learned) reverse process for blur on the left and masking on the right.

Original: We train the model with our Soft Score Matching objective on CelebA. Unconditional samples from the model trained with masking can be found in Figure 13 of the Appendix.
Revised: We train the model with our Soft Score Matching objective on CelebA. Unconditional samples from the model trained with masking can be found in Figure 13 of the Appendix.

Original: In Figure 5, we show the predictions of our two trained models (blur and masking) for the conditional mean, E
Revised: In Figure 5, we show the predictions of our two trained models (blur and masking) for the conditional mean, E

Original: [ x 0 | x t ] , at different times of the diffusion.
Revised: [ x 0 | x t ] , at different times of the diffusion.

Original: Soft Score Matching trains the model to make a prediction that matches the real images in the ﬁltered space.
Revised: Soft Score Matching trains the model to make a prediction that matches the real images in the ﬁltered space.

Original: Hence, given masked images the masking model is incentived to predict right only the observed noisy region.
Revised: Hence, given masked images the masking model is incentived to predict right only the observed noisy region.

Original: As diffusion time t becomes smaller (cleaner images), the observed region grows and the model predicts bigger windows.
Revised: As diffusion time t becomes smaller (cleaner images), the observed region grows and the model predicts bigger windows.

Original: Although it is interesting that we can train Masking Diffusion models, there are several limitations compared to Blur Diffusion (and even Gaussian Diffusion).
Revised: Although it is interesting that we can train Masking Diffusion models, there are several limitations compared to Blur Diffusion (and even Gaussian Diffusion).

Original: We observe that these models are very slow to sample from: withsampling steps, FID is 30 .
Revised: We observe that these models are very slow to sample from: withsampling steps, FID is 30 .

Original: 92 while with 4096 , FID improves to 12 . 37 .
Revised: 92 while with 4096 , FID improves to 12 . 37 .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We showed that by just changing the corruption mechanism, we observe important computational beneﬁts.
Revised: We showed that by just changing the corruption mechanism, we observe important computational beneﬁts.

Original: Reducing the number of function evaluations for sample generation with diffusion models is an active area of research.
Revised: Reducing the number of function evaluations for sample generation with diffusion models is an active area of research.

Original: Jolicoeur-Martineau et al.
Revised: Jolicoeur-Martineau et al.

Original: Liu et al.
Revised: Liu et al.

Original: ; Song et al.
Revised: ; Song et al.

Original: (2021a) propose more efﬁcient samplers for diffusion models.
Revised: (2021a) propose more efﬁcient samplers for diffusion models.

Original: Rombach et al.
Revised: Rombach et al.

Original: ; Daras et al.
Revised: ; Daras et al.

Original: (2022) train diffusion models on low-dimensional latent spaces.
Revised: (2022) train diffusion models on low-dimensional latent spaces.

Original: Xiao et al.
Revised: Xiao et al.

Original: (2022) combine diffusion models with Generative Adversarial Networks (GANs) to allow for bigger denoising steps.
Revised: (2022) combine diffusion models with Generative Adversarial Networks (GANs) to allow for bigger denoising steps.

Original: Ryu & Ye (2022); Ho et al.
Revised: Ryu & Ye (2022); Ho et al.

Original: (2022a) generate high-resolution images progressively, from coarse to ﬁne quality.
Revised: (2022a) generate high-resolution images progressively, from coarse to ﬁne quality.

Original: Salimans & Ho (2022) train progressively a student network that mimics the teacher diffusion model with fewer sampling steps.
Revised: Salimans & Ho (2022) train progressively a student network that mimics the teacher diffusion model with fewer sampling steps.

Original: We note that all these works are orthogonal to ours and therefore can be used in combination with our framework for even faster sampling.
Revised: We note that all these works are orthogonal to ours and therefore can be used in combination with our framework for even faster sampling.

Original: On scheduling, there is closely related work by Bao et al.
Revised: On scheduling, there is closely related work by Bao et al.

Original: The authors ﬁnd a closed form solution (w.r.t.
Revised: The authors ﬁnd a closed form solution (w.r.t.

Original: the score function) for the variance of the reverse SDE for Gaussian Diffusion.
Revised: the score function) for the variance of the reverse SDE for Gaussian Diffusion.

Original: Then, they select a noise scheduling that minimizes the KL along the path from the initial to the ﬁnal distribution.
Revised: Then, they select a noise scheduling that minimizes the KL along the path from the initial to the ﬁnal distribution.

Original: In our work, we use Wasserstein distances and consider general corruption processes for which it is unclear whether such a closed form solution exists.
Revised: In our work, we use Wasserstein distances and consider general corruption processes for which it is unclear whether such a closed form solution exists.

Original: Instead, we estimate the distances in a data-driven way.
Revised: Instead, we estimate the distances in a data-driven way.

Original: This allows us to schedule arbitrary diffusion processes in a principled way.
Revised: This allows us to schedule arbitrary diffusion processes in a principled way.

Original: There is signiﬁcant recent (Anonymous, 2022) and concurrent work (Rissanen et al., 2022; Bansal et al., 2022; Hoogeboom & Salimans, 2022) that proposes diffusion with other degradations.
Revised: There is signiﬁcant recent (Anonymous, 2022) and concurrent work (Rissanen et al., 2022; Bansal et al., 2022; Hoogeboom & Salimans, 2022) that proposes diffusion with other degradations.

Original: These works have signiﬁcant differences since they use different loss functions and sampling methods.
Revised: These works have signiﬁcant differences since they use different loss functions and sampling methods.

Original: Soft
Revised: Soft

Original: Score Matching experimentally outperforms (under the exact same setting) the loss functions used in the concurrent works (ours FID: 1 . 85 , theirs: 5 . 91 ).
Revised: Score Matching experimentally outperforms (under the exact same setting) the loss functions used in the concurrent works (ours FID: 1 . 85 , theirs: 5 . 91 ).

Original: Our (blur) models obtain state-of-the-art FID for linear diffusion models on CelebA (FID 1.85).
Revised: Our (blur) models obtain state-of-the-art FID for linear diffusion models on CelebA (FID 1.85).

Original: For CIFAR10, we outperform (FID: 3.86) Gaussian diffusion with Variance Exploding SDE (Song et al., 2021b).
Revised: For CIFAR10, we outperform (FID: 3.86) Gaussian diffusion with Variance Exploding SDE (Song et al., 2021b).

Original: Hoogeboom & Salimans (2022) also use blurring (but with Variance Preserving SDE) to further push the CIFAR-10 performance to FID 3.17.
Revised: Hoogeboom & Salimans (2022) also use blurring (but with Variance Preserving SDE) to further push the CIFAR-10 performance to FID 3.17.

Original: These advancements show that there are multiple diffusions with promising potential.
Revised: These advancements show that there are multiple diffusions with promising potential.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: We presented a framework to train and sample from diffusion models that reverse general linear corruptions.
Revised: We presented a framework to train and sample from diffusion models that reverse general linear corruptions.

Original: We showed that by changing the corruption process, we can get signiﬁcant sample quality improvements and computational beneﬁts.
Revised: We showed that by changing the corruption process, we can get signiﬁcant sample quality improvements and computational beneﬁts.

Original: This work opens several future research directions.
Revised: This work opens several future research directions.

Original: For example, it is possible to optimize or learn the corruption process for solving a speciﬁc type of inverse problem.
Revised: For example, it is possible to optimize or learn the corruption process for solving a speciﬁc type of inverse problem.

Original: It is also worth exploring if mixing different corruptions (blur, noise, masking, etc.)
Revised: It is also worth exploring if mixing different corruptions (blur, noise, masking, etc.)

Original: improves performance.
Revised: improves performance.

Original: Our work could also be extended to the non-linear case, leveraging the techniques introduced in (Rombach et al., 2022; Kim et al., 2022a; Wang et al., 2022).
Revised: Our work could also be extended to the non-linear case, leveraging the techniques introduced in (Rombach et al., 2022; Kim et al., 2022a; Wang et al., 2022).

Original: Finally, it is important to understand the role of noise, from both a theoretical and practical standpoint.
Revised: Finally, it is important to understand the role of noise, from both a theoretical and practical standpoint.

Original: A A PPENDIX A. P ROOFS
Revised: A A PPENDIX A. P ROOFS

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Let q 0 , q t be two distributions in R n .
Revised: Let q 0 , q t be two distributions in R n .

Original: Assume that all the conditional distributions, q t ( x t | x 0 ) , are supported and differentiable in R n .
Revised: Assume that all the conditional distributions, q t ( x t | x 0 ) , are supported and differentiable in R n .

Original: Let:
Revised: Let:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then, there is a universal constant C (that does not depend on θ ) such that: J 1 ( θ ) =
Revised: Then, there is a universal constant C (that does not depend on θ ) such that: J 1 ( θ ) =

Original: J 2 ( θ ) + C .
Revised: J 2 ( θ ) + C .

Original: The proof of this Theorem is following the calculations of Vincent (2011).
Revised: The proof of this Theorem is following the calculations of Vincent (2011).

Original: We observe that as long as the technical conditions listed are satisﬁed, the proof holds independent of the corruption type.
Revised: We observe that as long as the technical conditions listed are satisﬁed, the proof holds independent of the corruption type.

Original: We provide the proof below for the shake of completeness.
Revised: We provide the proof below for the shake of completeness.

Original: Proof of Theorem 3.1.
Revised: Proof of Theorem 3.1.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: It sufﬁces to show that:
Revised: It sufﬁces to show that:

Original: [Equation]
Revised: [Equation]

Original: We start with the second term.
Revised: We start with the second term.

Original: [Equation]
Revised: [Equation]

Original: Lemma A.1 (Tweedie’s formula) .
Revised: Lemma A.1 (Tweedie’s formula) .

Original: Consider the corruption process: x t =
Revised: Consider the corruption process: x t =

Original: C t x 0 + σ t η , where η ∼ N ( 0 , I ) .
Revised: C t x 0 + σ t η , where η ∼ N ( 0 , I ) .

Original: Denote with q t ( x t ) the density of x t and assume that log q t ( x t ) is differentiable everywhere.
Revised: Denote with q t ( x t ) the density of x t and assume that log q t ( x t ) is differentiable everywhere.

Original: Then, for the Minimum Mean-Square Estimation (MMSE) of x 0 given x t , it holds that:
Revised: Then, for the Minimum Mean-Square Estimation (MMSE) of x 0 given x t , it holds that:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We know that: q t ( x t | x 0 ) =
Revised: We know that: q t ( x t | x 0 ) =

Original: N ( x t ; µ = C t x 0 , Σ
Revised: N ( x t ; µ = C t x 0 , Σ

Original: = σ 2 t I ) .
Revised: = σ 2 t I ) .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: B D ETERMINISTIC S AMPLERS B. P ROBABILITY F LOW ODE
Revised: B D ETERMINISTIC S AMPLERS B. P ROBABILITY F LOW ODE

Original: In the main text
Revised: In the main text

Original: [Equation]
Revised: [Equation]

Original: we derived our Momentum Sampler by analyzing the Backward SDE associated with our corruption process.
Revised: we derived our Momentum Sampler by analyzing the Backward SDE associated with our corruption process.

Original: Inspired by the works of Song et al.
Revised: Inspired by the works of Song et al.

Original: (2021b)
Revised: (2021b)

Original: ; Maoutsa et al.
Revised: ; Maoutsa et al.

Original: , we also consider deterministic sampling that is derived by looking at the ODE that describes our diffusion.
Revised: , we also consider deterministic sampling that is derived by looking at the ODE that describes our diffusion.

Original: Particularly, the ODE:
Revised: Particularly, the ODE:

Original: [Equation]
Revised: [Equation]

Original: (47) has the same marginal distributions (Anderson, 1982;
Revised: (47) has the same marginal distributions (Anderson, 1982;

Original: Maoutsa et al., 2020; Song et al., 2021b; Chen et al., 2018) with the Backward SDE:
Revised: Maoutsa et al., 2020; Song et al., 2021b; Chen et al., 2018) with the Backward SDE:

Original: [Equation]
Revised: [Equation]

Original: For our case, Eq.
Revised: For our case, Eq.

Original: (47) becomes:
Revised: (47) becomes:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst-order discretization of this ODE is given below:
Revised: The ﬁrst-order discretization of this ODE is given below:

Original: [Equation]
Revised: [Equation]

Original: We estimate C t E
Revised: We estimate C t E

Original: [ x 0 | x t ] and ∇ x t log q t ( x t ) with our neural network and we get the Neural
Revised: [ x 0 | x t ] and ∇ x t log q t ( x t ) with our neural network and we get the Neural

Original: ODE (Chen et al., 2018):
Revised: ODE (Chen et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: which is the update rule of our Probability Flow ODE Momentum Sampler.
Revised: which is the update rule of our Probability Flow ODE Momentum Sampler.

Original: We note that our simple discretization is not the only way to solve the ODE of Eq.
Revised: We note that our simple discretization is not the only way to solve the ODE of Eq.

Original: We can use more sophisticated, e.g. see Dormand & Prince (1980).
Revised: We can use more sophisticated, e.g. see Dormand & Prince (1980).

Original: B. DDIM
Revised: B. DDIM

Original: A popular sampling scheme is the DDIM method, introduced in Song et al.
Revised: A popular sampling scheme is the DDIM method, introduced in Song et al.

Original: (2021a).
Revised: (2021a).

Original: The idea is to use the network to predict ˆ x 0 from x t and then use the forward model to move from ˆ x 0 to x t − ∆ t .
Revised: The idea is to use the network to predict ˆ x 0 from x t and then use the forward model to move from ˆ x 0 to x t − ∆ t .

Original: This is same with the Naive Sampler we introduced in the main paper but the difference in DDIM is that part of the stochasticity is replaced by “simulated” noise.
Revised: This is same with the Naive Sampler we introduced in the main paper but the difference in DDIM is that part of the stochasticity is replaced by “simulated” noise.

Original: The main trick to simulate the noise is to observe that once we know ˆ x 0 and x t , we can once again use the forward model to estimate the noise.
Revised: The main trick to simulate the noise is to observe that once we know ˆ x 0 and x t , we can once again use the forward model to estimate the noise.

Original: We can use the same idea for Soft Diffusion sampling.
Revised: We can use the same idea for Soft Diffusion sampling.

Original: Speciﬁcally, we propose the following
Revised: Speciﬁcally, we propose the following

Original: DDIM-type sampler:
Revised: DDIM-type sampler:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: This Equation is very similar to Equation 12, page 5 in the DDIM (Song et al., 2021a) paper.
Revised: This Equation is very similar to Equation 12, page 5 in the DDIM (Song et al., 2021a) paper.

Original: The parameter k controls the amount of stochasticity, i.e. how much of the noise is simulated.
Revised: The parameter k controls the amount of stochasticity, i.e. how much of the noise is simulated.

Original: For k = 0 , we get a deterministic sampler.
Revised: For k = 0 , we get a deterministic sampler.

Original: Experiments with the deterministic DDIM-type sampler can be found in Section F.3.
Revised: Experiments with the deterministic DDIM-type sampler can be found in Section F.3.

Original: C S CHEDULINGS
Revised: C S CHEDULINGS

Original: We use our framework to select the blur levels in an unsupervised way.
Revised: We use our framework to select the blur levels in an unsupervised way.

Original: For solving the optimal transport problem, we use the Sinkhorn Distances by Cuturi (2013) that has been extensively used for image experiments.
Revised: For solving the optimal transport problem, we use the Sinkhorn Distances by Cuturi (2013) that has been extensively used for image experiments.

Original: We use the implementation of this paper from the software package ott-jax (Cuturi et al., 2022).
Revised: We use the implementation of this paper from the software package ott-jax (Cuturi et al., 2022).

Original: We experimented with both using the whole dataset and using slices.
Revised: We experimented with both using the whole dataset and using slices.

Original: As expected, dataset slices lead to higher estimation errors, but we observe that the found scheduling doesn’t change much, i.e. the estimation error increases approximately uniformly for all the pairs for reasonably sized dataset slices.
Revised: As expected, dataset slices lead to higher estimation errors, but we observe that the found scheduling doesn’t change much, i.e. the estimation error increases approximately uniformly for all the pairs for reasonably sized dataset slices.

Original: For the schedules found in the paper, we used slices of 10% of the dataset.
Revised: For the schedules found in the paper, we used slices of 10% of the dataset.

Original: We start with 256 different blur levels and we tune (cid:15) such that the shortest path contains 32 distributions.
Revised: We start with 256 different blur levels and we tune (cid:15) such that the shortest path contains 32 distributions.

Original: We then use linear interpolation to extend to the continuous case.
Revised: We then use linear interpolation to extend to the continuous case.

Original: Full experimental details can be found in the Appendix.
Revised: Full experimental details can be found in the Appendix.

Original: These choices seem to work well in practice, but further optimization could be made in future work.
Revised: These choices seem to work well in practice, but further optimization could be made in future work.

Original: Figure 6 shows the found schedulings for the CelebA and the CIFAR-10 datasets.
Revised: Figure 6 shows the found schedulings for the CelebA and the CIFAR-10 datasets.

Original: Notice that the scheduling is slightly different between the two datasets – the diffusion depends on the nature of the data we are trying to model.
Revised: Notice that the scheduling is slightly different between the two datasets – the diffusion depends on the nature of the data we are trying to model.

Original: We underline that the parameters for the blur are selected without any supervision, by solving the optimization problem we deﬁned in Section 3.3.
Revised: We underline that the parameters for the blur are selected without any supervision, by solving the optimization problem we deﬁned in Section 3.3.

Original: [Figure]
Revised: [Figure]

Original: D T RAINING D ETAILS
Revised: D T RAINING D ETAILS

Original: Hyperparameters.
Revised: Hyperparameters.

Original: For our trainings, we use Adam optimizer with learning rate 2 e − 4 , β 1 = 0 .
Revised: For our trainings, we use Adam optimizer with learning rate 2 e − 4 , β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , (cid:15) = 1 e − 8 .
Revised: 999 , (cid:15) = 1 e − 8 .

Original: We additionally use gradient clipping for gradient norms bigger than 1 .
Revised: We additionally use gradient clipping for gradient norms bigger than 1 .

Original: For the learning rate scheduling, we use 5000 steps of linear warmup.
Revised: For the learning rate scheduling, we use 5000 steps of linear warmup.

Original: We use batch size 

Revised: We use batch size

Original: and we train for 1 − 2 M iterations (based on observed FID performance).
Revised: and we train for 1 − 2 M iterations (based on observed FID performance).

Original: Blur parameters.
Revised: Blur parameters.

Original: For the blurring operator, we use Gaussian blur with ﬁxed kernel size and we vary the variance of the kernel.
Revised: For the blurring operator, we use Gaussian blur with ﬁxed kernel size and we vary the variance of the kernel.

Original: For CelebA- 64 , we keep the kernel half size ﬁxed to 80 and we vary the standard deviation from 0 .
Revised: For CelebA- 64 , we keep the kernel half size ﬁxed to 80 and we vary the standard deviation from 0 .

Original: 01 to 23 .
Revised: 01 to 23 .

Original: For CIFAR-10, we keep the kernel half size ﬁxed to 32 and we vary the standard deviation from 0 .
Revised: For CIFAR-10, we keep the kernel half size ﬁxed to 32 and we vary the standard deviation from 0 .

Original: 01 to 18 .
Revised: 01 to 18 .

Original: For both datasets, we implement blur with zero-padding.
Revised: For both datasets, we implement blur with zero-padding.

Original: We chose the ﬁnal blur level such that the ﬁnal distribution is easy to sample from.
Revised: We chose the ﬁnal blur level such that the ﬁnal distribution is easy to sample from.

Original: In both cases, the ﬁnal distribution becomes noise on top of (almost) a single color.
Revised: In both cases, the ﬁnal distribution becomes noise on top of (almost) a single color.

Original: Final distribution For the blurring models, the ﬁnal distribution is the distribution of very blurry images with additive Gaussian noise (we ﬁrst blur, then add noise).
Revised: Final distribution For the blurring models, the ﬁnal distribution is the distribution of very blurry images with additive Gaussian noise (we ﬁrst blur, then add noise).

Original: At the limit, each blurry image becomes a constant image having a single color (i.e., the average color of the image).
Revised: At the limit, each blurry image becomes a constant image having a single color (i.e., the average color of the image).

Original: Hence, to sample from this ﬁnal distribution, we ﬁrst have to sample a single color (from the distribution of average colors in the dataset), generate a constant image having that color, and then add little noise to the constant image.
Revised: Hence, to sample from this ﬁnal distribution, we ﬁrst have to sample a single color (from the distribution of average colors in the dataset), generate a constant image having that color, and then add little noise to the constant image.

Original: The distribution of average colors for all the considered datasets is very simple and we model it with a Gaussian distribution.
Revised: The distribution of average colors for all the considered datasets is very simple and we model it with a Gaussian distribution.

Original: Speciﬁcally, we ﬁt a three-dimensional gaussian distribution (one dimension for each color channel, diagonal covariance) to the average color of the dataset.
Revised: Speciﬁcally, we ﬁt a three-dimensional gaussian distribution (one dimension for each color channel, diagonal covariance) to the average color of the dataset.

Original: To begin the inference process, we sample one color from this distribution, we create an image where every pixel has this color and then we add Gaussian noise.
Revised: To begin the inference process, we sample one color from this distribution, we create an image where every pixel has this color and then we add Gaussian noise.

Original: Architecture.
Revised: Architecture.

Original: We use the architecture of Song et al.
Revised: We use the architecture of Song et al.

Original: (2021b) without any changes.
Revised: (2021b) without any changes.

Original: Training Objective For all our experiments, we scale the loss at level t with w ( t )
Revised: Training Objective For all our experiments, we scale the loss at level t with w ( t )

Original: = 

Revised: =

Original: /σ 2 t as in
Revised: /σ 2 t as in

Original: Song & Ermon (2019; 2020); Song et al.
Revised: Song & Ermon (2019; 2020); Song et al.

Original: (2021b).
Revised: (2021b).

Original: Compute and Training Time We train our models on 16 v2-TPUs.
Revised: Compute and Training Time We train our models on 16 v2-TPUs.

Original: Our blur models on CelebA had an average speed of 6 iterations per second.
Revised: Our blur models on CelebA had an average speed of 6 iterations per second.

Original: For CIFAR-10, the average speed was 11 iterations per second
Revised: For CIFAR-10, the average speed was 11 iterations per second

Original: We note that there is no overhead over the NCSN++ paper other than projecting to the measurements space, which can be done very efﬁciently for both blur and masking.
Revised: We note that there is no overhead over the NCSN++ paper other than projecting to the measurements space, which can be done very efﬁciently for both blur and masking.

Original: Evaluation We keep one checkpoint every 10000 steps and we keep the best model among the kept checkpoints based on the obtained FID score.
Revised: Evaluation We keep one checkpoint every 10000 steps and we keep the best model among the kept checkpoints based on the obtained FID score.

Original: We use 50000 samples to evaluate the FID, as it is typically done in prior work.
Revised: We use 50000 samples to evaluate the FID, as it is typically done in prior work.

Original: Regarding the number of steps, we selected the best FID among models evaluated with steps ranging from 200 to 1000 with a step size of 80 ( 10 experiments total).
Revised: Regarding the number of steps, we selected the best FID among models evaluated with steps ranging from 200 to 1000 with a step size of 80 ( 10 experiments total).

Original: The Momentum Sampler seems to have a U-shaped performance plot, i.e. there is a sweet spot in the number of function evaluations that gives the lowest FID score.
Revised: The Momentum Sampler seems to have a U-shaped performance plot, i.e. there is a sweet spot in the number of function evaluations that gives the lowest FID score.

Original: We observed this in both CelebA and CIFAR-10.
Revised: We observed this in both CelebA and CIFAR-10.

Original: However, this is not unique to our sampler – the general belief that sample quality improves with the number of steps has been called into question by recent works.
Revised: However, this is not unique to our sampler – the general belief that sample quality improves with the number of steps has been called into question by recent works.

Original: Speciﬁcally, Karras et.
Revised: Speciﬁcally, Karras et.

Original: al.
Revised: al.

Original: do an extensive evaluation of the role of NFEs for different samplers
Revised: do an extensive evaluation of the role of NFEs for different samplers

Original: and they ﬁnd that many stochastic samplers behave similarly to our Momentum Sampler.
Revised: and they ﬁnd that many stochastic samplers behave similarly to our Momentum Sampler.

Original: For example, we refer the interested reader to
Revised: For example, we refer the interested reader to

Original: Figure 4, page 8 of the paper Elucidating the Design Space of Diffusion-Based Generative Models that clearly shows that for many stochastic samplers performance slightly deteriorates after a certain threshold of function evaluations and the optimum is at some intermediate point.
Revised: Figure 4, page 8 of the paper Elucidating the Design Space of Diffusion-Based Generative Models that clearly shows that for many stochastic samplers performance slightly deteriorates after a certain threshold of function evaluations and the optimum is at some intermediate point.

Original: We emphasize, however, that the deterioration we observed in CelebA is bigger compared to other stochastic samplers and therefore selecting the correct number of NFEs is important for our method.
Revised: We emphasize, however, that the deterioration we observed in CelebA is bigger compared to other stochastic samplers and therefore selecting the correct number of NFEs is important for our method.

Original: Consistent with the observations by Karras et al.
Revised: Consistent with the observations by Karras et al.

Original: (e.g. Figure 2, page 4), for deterministic samplers the performance ﬂattens (DDIM) or deteriorates slightly (Probability Flow Momentum Sampler) in the regime of very high Number of Function Evaluations (NFEs).
Revised: (e.g. Figure 2, page 4), for deterministic samplers the performance ﬂattens (DDIM) or deteriorates slightly (Probability Flow Momentum Sampler) in the regime of very high Number of Function Evaluations (NFEs).

Original: We also want to note that selecting the optimal number of function evaluations for reporting FID is not uncommon in the literature.
Revised: We also want to note that selecting the optimal number of function evaluations for reporting FID is not uncommon in the literature.

Original: It is done in both the landmark papers Elucidating the Design Space
Revised: It is done in both the landmark papers Elucidating the Design Space

Original: E L IMITATIONS AND THINGS THAT DID NOT WORK
Revised: E L IMITATIONS AND THINGS THAT DID NOT WORK

Original: Our method has several limitations.
Revised: Our method has several limitations.

Original: First, it requires the diffusion operator to be known.
Revised: First, it requires the diffusion operator to be known.

Original: This is not always the case, e.g. see Peebles et al.
Revised: This is not always the case, e.g. see Peebles et al.

Original: (2022) where diffusion is applied to checkpoints of different models.
Revised: (2022) where diffusion is applied to checkpoints of different models.

Original: Another limitation is that our framework does not offer any guidance on which diffusion operators are actually more or less useful for learning the data distribution.
Revised: Another limitation is that our framework does not offer any guidance on which diffusion operators are actually more or less useful for learning the data distribution.

Original: Particularly, we already showed that blurring is a much more powerful diffusion mechanism than masking, in the sense that it leads to better FID scores and faster generations.
Revised: Particularly, we already showed that blurring is a much more powerful diffusion mechanism than masking, in the sense that it leads to better FID scores and faster generations.

Original: It is yet to be seen whether blurring is going to be outperformed by some other corruption method.
Revised: It is yet to be seen whether blurring is going to be outperformed by some other corruption method.

Original: Our method also only concerns linear diffusions (however, the extension to non-linear is relatively straightforward).
Revised: Our method also only concerns linear diffusions (however, the extension to non-linear is relatively straightforward).

Original: On the theoretical side, our method has also some shortcomings.
Revised: On the theoretical side, our method has also some shortcomings.

Original: First, it only intuitively explains why the reparametrization to the Denoising Score Matching is needed.
Revised: First, it only intuitively explains why the reparametrization to the Denoising Score Matching is needed.

Original: Second, since our method is based on the Denoising Score Matching, it only works when the conditional log-likelihood is deﬁned everywhere.
Revised: Second, since our method is based on the Denoising Score Matching, it only works when the conditional log-likelihood is deﬁned everywhere.

Original: There are distributions for which such condition is not satisﬁed, but still, can be learned (to some extent) with heuristic methods (Bansal et al., 2022).
Revised: There are distributions for which such condition is not satisﬁed, but still, can be learned (to some extent) with heuristic methods (Bansal et al., 2022).

Original: Another limitation of our work is the derivation of the Momentum Sampler and its sampling distribution.
Revised: Another limitation of our work is the derivation of the Momentum Sampler and its sampling distribution.

Original: Consider the forward process:
Revised: Consider the forward process:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This process is described by an Ito SDE, i.e. an SDE of the form:
Revised: This process is described by an Ito SDE, i.e. an SDE of the form:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where f ( x t , t ) = ˙
Revised: where f ( x t , t ) = ˙

Original: C t E x 0 ∼ p 

Revised: C t E x 0 ∼ p

Original: [ x 0 | x t ] .
Revised: [ x 0 | x t ] .

Original: The sampling process we write in the main body of the paper (that leads to the Momentum Sampler) is exactly the reverse of this forward process, where E x 0 ∼ p 

Revised: The sampling process we write in the main body of the paper (that leads to the Momentum Sampler) is exactly the reverse of this forward process, where E x 0 ∼ p

Original: [ x 0 | x t ] is approximated by the neural network.
Revised: [ x 0 | x t ] is approximated by the neural network.

Original: Hence, the Fokker-Planck equations hold and we are guaranteed that (as long as the approximation of the conditional expectation and the approximation of the score function are accurate), we are sampling from the correct distribution (Song et al., 2021b).
Revised: Hence, the Fokker-Planck equations hold and we are guaranteed that (as long as the approximation of the conditional expectation and the approximation of the score function are accurate), we are sampling from the correct distribution (Song et al., 2021b).

Original: During training, we cannot use this forward process because we do not have the conditional expectation.
Revised: During training, we cannot use this forward process because we do not have the conditional expectation.

Original: Instead, we replace the conditional expectation of x 0 with x 0 itself.
Revised: Instead, we replace the conditional expectation of x 0 with x 0 itself.

Original: The mismatch between x 0 and the conditional expectation of x 0 leads to an additional approximation error in the learning of the score.
Revised: The mismatch between x 0 and the conditional expectation of x 0 leads to an additional approximation error in the learning of the score.

Original: For small t , the conditional expectation will be very close to x 0 and hence this approximation error is small.
Revised: For small t , the conditional expectation will be very close to x 0 and hence this approximation error is small.

Original: For large values of t , the distance between the conditional expectation and x 0 is bigger.
Revised: For large values of t , the distance between the conditional expectation and x 0 is bigger.

Original: However, we are multiplying with the corruption matrix C t , which is removing information as t grows.
Revised: However, we are multiplying with the corruption matrix C t , which is removing information as t grows.

Original: Therefore the distance of the corrupted conditional expectation and the corrupted x 0 is also expected to be small.
Revised: Therefore the distance of the corrupted conditional expectation and the corrupted x 0 is also expected to be small.

Original: Our training process learns the correct score for the corruption process applied directly to x 0 .
Revised: Our training process learns the correct score for the corruption process applied directly to x 0 .

Original: Our sampler, however, assumes that we have learned the score using the conditional expectation of x 0 .
Revised: Our sampler, however, assumes that we have learned the score using the conditional expectation of x 0 .

Original: We intuitively expect that these two are not far away (for all t ) as we explained.
Revised: We intuitively expect that these two are not far away (for all t ) as we explained.

Original: However, precisely characterizing this mismatch remains open.
Revised: However, precisely characterizing this mismatch remains open.

Original: On the practical side, we believe that our objective, Soft Score Matching, sometimes leads to slower sampling compared to the simpler objective of predicting the clean image.
Revised: On the practical side, we believe that our objective, Soft Score Matching, sometimes leads to slower sampling compared to the simpler objective of predicting the clean image.

Original: For example, for masking, since the model is only penalized in the observed region, there is no incentive in expanding this region.
Revised: For example, for masking, since the model is only penalized in the observed region, there is no incentive in expanding this region.

Original: Hence, to achieve smooth transition between different masking levels we need to run many steps.
Revised: Hence, to achieve smooth transition between different masking levels we need to run many steps.

Original: Experimentally, we tried using our framework with even less stochasticity, but it did not work, e.g. see 7.
Revised: Experimentally, we tried using our framework with even less stochasticity, but it did not work, e.g. see 7.

Original: It would be interesting to understand better what is causing the failure and also what is the proper amount of randomness required at each diffusion step.
Revised: It would be interesting to understand better what is causing the failure and also what is the proper amount of randomness required at each diffusion step.

Original: [Figure]
Revised: [Figure]

Original: F A DDITIONAL R ESULTS F. S CHEDULING A BLATIONS F.1. A BLATION S TUDIES FOR NOISE
Revised: F A DDITIONAL R ESULTS F. S CHEDULING A BLATIONS F.1. A BLATION S TUDIES FOR NOISE

Original: In the experiments of the main paper, our diffusion involves an initial stage where only noise is added.
Revised: In the experiments of the main paper, our diffusion involves an initial stage where only noise is added.

Original: Then noise is ﬁxed and the images are getting corrupted by the deterministic operator (e.g. blur or masking).
Revised: Then noise is ﬁxed and the images are getting corrupted by the deterministic operator (e.g. blur or masking).

Original: In this section, we show two ablations regarding the noise.
Revised: In this section, we show two ablations regarding the noise.

Original: Magnitude of noise.
Revised: Magnitude of noise.

Original: In this ablation study, we still keep the noise ﬁxed for a signiﬁcant part of the diffusion, but we ablate the magnitude of the noise.
Revised: In this ablation study, we still keep the noise ﬁxed for a signiﬁcant part of the diffusion, but we ablate the magnitude of the noise.

Original: Speciﬁcally, we attempt to study to what extent noise is needed in order to learn to reverse corruption processes with Soft Score Matching.
Revised: Speciﬁcally, we attempt to study to what extent noise is needed in order to learn to reverse corruption processes with Soft Score Matching.

Original: We train two additional models on CelebA-64 where we attempt to decrease the maximum noise to a lower value.
Revised: We train two additional models on CelebA-64 where we attempt to decrease the maximum noise to a lower value.

Original: The corruptions for both models involves an initial stage where noise grows geometrically rate from the initial value ( 0 . 01 ) to the maximum value.
Revised: The corruptions for both models involves an initial stage where noise grows geometrically rate from the initial value ( 0 . 01 ) to the maximum value.

Original: Then, all the models keep the noise ﬁxed at their maximum value for the rest of the diffusion.
Revised: Then, all the models keep the noise ﬁxed at their maximum value for the rest of the diffusion.

Original: We use the following maximum values: i) σ max = 0 .(model used in the paper), ii) σ max = 0 . 05 , and iii) σ max = 0 . 025 .
Revised: We use the following maximum values: i) σ max = 0 .(model used in the paper), ii) σ max = 0 . 05 , and iii) σ max = 0 . 025 .

Original: Unconditional samples from the two ablation models are shown in Figure 7.
Revised: Unconditional samples from the two ablation models are shown in Figure 7.

Original: As shown, both models fail to produce realistic samples.
Revised: As shown, both models fail to produce realistic samples.

Original: The quality of samples deteriorates signiﬁcantly as the noise decreases – the samples from the ablation models are signiﬁcantly worse than the ones produced by our state-of-the-art model (see
Revised: The quality of samples deteriorates signiﬁcantly as the noise decreases – the samples from the ablation models are signiﬁcantly worse than the ones produced by our state-of-the-art model (see

Original: Figure 2 (right)).
Revised: Figure 2 (right)).

Original: We want to underline that this is not a conclusive study.
Revised: We want to underline that this is not a conclusive study.

Original: It might be the case that with different hyperparameters one can make Soft Score Matching work with lower values of noise.
Revised: It might be the case that with different hyperparameters one can make Soft Score Matching work with lower values of noise.

Original: For example, we might need to tune the weights w ( t ) since for the ablations (and the state-of-the-art model), we use w ( t )
Revised: For example, we might need to tune the weights w ( t ) since for the ablations (and the state-of-the-art model), we use w ( t )

Original: =
Revised: =

Original: /σ 2 t (as in Song & Ermon (2019; 2020);
Revised: /σ 2 t (as in Song & Ermon (2019; 2020);

Original: Song et al.
Revised: Song et al.

Original: (2021b)) which might be causing instabilities for low values of noise (Nichol & Dhariwal, 2021).
Revised: (2021b)) which might be causing instabilities for low values of noise (Nichol & Dhariwal, 2021).

Original: Noise changing throughout the diffusion.
Revised: Noise changing throughout the diffusion.

Original: We also train a model where noise and blur are changing simultaneously throughout the diffusion.
Revised: We also train a model where noise and blur are changing simultaneously throughout the diffusion.

Original: This is a sanity check to verify that our framework (learning and sampling) still works when the model needs to deblur and denoise at the same time.
Revised: This is a sanity check to verify that our framework (learning and sampling) still works when the model needs to deblur and denoise at the same time.

Original: For the noise scheduling, we simply use geometric scheduling from 0 .
Revised: For the noise scheduling, we simply use geometric scheduling from 0 .

Original: 1 to 0 .
Revised: 1 to 0 .

Original: We keep the blur parameters the same with the state-of-the-art model, i.e. we use the blur parameters shown in Figure 6.
Revised: We keep the blur parameters the same with the state-of-the-art model, i.e. we use the blur parameters shown in Figure 6.

Original: This model achieves a competitive FID score, 4 . 31 .
Revised: This model achieves a competitive FID score, 4 . 31 .

Original: This score could probably be further improved (by jointly selecting the blur and the noise scheduling with our framework), but this is beyond the scope of this ablation.
Revised: This score could probably be further improved (by jointly selecting the blur and the noise scheduling with our framework), but this is beyond the scope of this ablation.

Original: F.1. A BLATION S TUDY FOR B LUR
Revised: F.1. A BLATION S TUDY FOR B LUR

Original: For all our experiments so far, we chose the blur corruption levels based on the scheduling method we described in Section 3.3.
Revised: For all our experiments so far, we chose the blur corruption levels based on the scheduling method we described in Section 3.3.

Original: We show the beneﬁts of our approach by comparing to a natural baseline for selecting the diffusion levels.
Revised: We show the beneﬁts of our approach by comparing to a natural baseline for selecting the diffusion levels.

Original: For this natural baseline, we use the scheduling of Variance
Revised: For this natural baseline, we use the scheduling of Variance

Original: Exploding (VE) as guidance.
Revised: Exploding (VE) as guidance.

Original: Speciﬁcally, we choose the blur parameters such that the MSE between the corrupted image and the clean image decays with the same rate for the Gaussian Denoising Diffusion and our Soft Diffusion (blur and low magnitude noise).
Revised: Speciﬁcally, we choose the blur parameters such that the MSE between the corrupted image and the clean image decays with the same rate for the Gaussian Denoising Diffusion and our Soft Diffusion (blur and low magnitude noise).

Original: Formally, let { q (cid:48) t } 1 t =

Revised: Formally, let { q (cid:48) t } 1 t =

Original: be the (noisy) distributions used in Song et al.
Revised: be the (noisy) distributions used in Song et al.

Original: (2021b)
Revised: (2021b)

Original: for the Variance Exploding (VE) SDE and let { q t } 1 t =

Revised: for the Variance Exploding (VE) SDE and let { q t } 1 t =

Original: the blurry (and noisy) distributions we want to select.
Revised: the blurry (and noisy) distributions we want to select.

Original: At level t , we choose the blur parameters such that:
Revised: At level t , we choose the blur parameters such that:

Original: [Equation]
Revised: [Equation]

Original: We retrain on CelebA using this natural baseline method for selecting the diffusion parameters.
Revised: We retrain on CelebA using this natural baseline method for selecting the diffusion parameters.

Original: For a fair comparison, we keep the architecture and all the hyperparameters the same and we only ablate the scheduling of the blur.
Revised: For a fair comparison, we keep the architecture and all the hyperparameters the same and we only ablate the scheduling of the blur.

Original: We measure FID for both the trained model with the baseline scheduling and we observe it increases from 1 . 85 to 8 .
Revised: We measure FID for both the trained model with the baseline scheduling and we observe it increases from 1 . 85 to 8 .

Original: Apart from this large deterioration in performance, the baseline model obtains its best FID score after 2000 steps, while with our scheduling we only need 280 steps to obtain the best FID.
Revised: Apart from this large deterioration in performance, the baseline model obtains its best FID score after 2000 steps, while with our scheduling we only need 280 steps to obtain the best FID.

Original: This experiment shows that the choice of scheduling is really important for the model performance but for also the computational requirements of the sampling.
Revised: This experiment shows that the choice of scheduling is really important for the model performance but for also the computational requirements of the sampling.

Original: F. CIFAR-10: A DDITIONAL R ESULTS
Revised: F. CIFAR-10: A DDITIONAL R ESULTS

Original: We report how FID is changing based on the Number of Function Evaluations for CIFAR-10 with
Revised: We report how FID is changing based on the Number of Function Evaluations for CIFAR-10 with

Original: Momentum Sampling in Figure 8.
Revised: Momentum Sampling in Figure 8.

Original: The baseline model we are comparing against is NCSN++ trained with VE SDE, since our model also uses a VE SDE.
Revised: The baseline model we are comparing against is NCSN++ trained with VE SDE, since our model also uses a VE SDE.

Original: For the baseline, we use the Reverse SDE sampler with Euler–Maruyama discretization (similar to our Momentum Sampler).
Revised: For the baseline, we use the Reverse SDE sampler with Euler–Maruyama discretization (similar to our Momentum Sampler).

Original: The numbers are taken directly from Ma et al.
Revised: The numbers are taken directly from Ma et al.

Original: [Figure]
Revised: [Figure]

Original: how FID changes based on Number of Function Evaluations (NFEs) for CIFAR-10 for our (blur) model on CIFAR-10 with momentum sampling.
Revised: how FID changes based on Number of Function Evaluations (NFEs) for CIFAR-10 for our (blur) model on CIFAR-10 with momentum sampling.

Original: Our model offers signiﬁcant performance beneﬁts for low number of function evaluations.
Revised: Our model offers signiﬁcant performance beneﬁts for low number of function evaluations.

Original: We also show visual samples from our blurring model, NCSN++ (VE SDE) and DDPM++(VP
Revised: We also show visual samples from our blurring model, NCSN++ (VE SDE) and DDPM++ (VP

Original: SDE) in Figure 9.
Revised: SDE) in Figure 9.

Original: For all the models we ﬁx the NFEs to 200 and we show samples generated with the Euler-Maruyama discretization of the associated reverse ODE.
Revised: For all the models we ﬁx the NFEs to 200 and we show samples generated with the Euler-Maruyama discretization of the associated reverse ODE.

Original: As shown, the baseline models generate samples with artifacts for low NFEs while our model leads to images of superior visual quality.
Revised: As shown, the baseline models generate samples with artifacts for low NFEs while our model leads to images of superior visual quality.

Original: We note that different samplers can be used to accelerate sampling for all models, e.g. samplers from Karras et al.
Revised: We note that different samplers can be used to accelerate sampling for all models, e.g. samplers from Karras et al.

Original: or DDIM-type samplers (Song et al., 2021a), as we show in
Revised: or DDIM-type samplers (Song et al., 2021a), as we show in

Original: Section B.2.
Revised: Section B.2.

Original: We also report best FID performance for different samplers for ours and competing methods in Table 2.
Revised: We also report best FID performance for different samplers for ours and competing methods in Table 2.

Original: Our best FID is competitive with similar samplers to similar methods, e.g. with NCSN++ trained with VE SDE and using the Reverse SDE sampler.
Revised: Our best FID is competitive with similar samplers to similar methods, e.g. with NCSN++ trained with VE SDE and using the Reverse SDE sampler.

Original: However, it is signiﬁcantly behind the state-ofthe-art LSGM (Vahdat et al., 2021) model.
Revised: However, it is signiﬁcantly behind the state-ofthe-art LSGM (Vahdat et al., 2021) model.

Original: We believe that this performance gap can be decreased in the future by further research in the area of diffusion models with general corruptions.
Revised: We believe that this performance gap can be decreased in the future by further research in the area of diffusion models with general corruptions.

Original: F. S AMPLING A BLATIONS
Revised: F. S AMPLING A BLATIONS

Original: We perform several ablations regarding the sampling method, additional to the results mentioned in the paper.
Revised: We perform several ablations regarding the sampling method, additional to the results mentioned in the paper.

Original: DDIM We experiment with the deterministic DDIM-type sampler and we report our results in the
Revised: DDIM We experiment with the deterministic DDIM-type sampler and we report our results in the

Original: Table 3 for CelebA-64.
Revised: Table 3 for CelebA-64.

Original: Our DDIM-type sampler is very effective when the number of function evaluations is very low – it achieves FID 5 .
Revised: Our DDIM-type sampler is very effective when the number of function evaluations is very low – it achieves FID 5 .

Original: with only 50 steps while in comparison, the Momentum
Revised: with only 50 steps while in comparison, the Momentum

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Sampler, achieves FID 30 .
Revised: Sampler, achieves FID 30 .

Original: 31 with 50 steps.
Revised: 31 with 50 steps.

Original: The efﬁciency of DDIM is consistent with what has been observed in (Song et al., 2021a; Karras et al., 2022).
Revised: The efﬁciency of DDIM is consistent with what has been observed in (Song et al., 2021a; Karras et al., 2022).

Original: DDIM’s performance is also not getting worse as we increase the number of steps.
Revised: DDIM’s performance is also not getting worse as we increase the number of steps.

Original: Instead, the performance of the Momentum Sampler, as observed in Figure 3 has a U-shape: there is an intermediate optimal number of steps to achieve the best FID score.
Revised: Instead, the performance of the Momentum Sampler, as observed in Figure 3 has a U-shape: there is an intermediate optimal number of steps to achieve the best FID score.

Original: We believe this could be related to the fact that Momentum Sampler is not deterministic.
Revised: We believe this could be related to the fact that Momentum Sampler is not deterministic.

Original: Speciﬁcally, in Karras et al.
Revised: Speciﬁcally, in Karras et al.

Original: (2022) it is observed that for deterministic samplers performance usually ﬂattens after a point (e.g. see Figure 2, page 4) while for some stochastic samplers performance goes again up after a certain number of function evaluations (e.g. see Figure 4, page 8).
Revised: (2022) it is observed that for deterministic samplers performance usually ﬂattens after a point (e.g. see Figure 2, page 4) while for some stochastic samplers performance goes again up after a certain number of function evaluations (e.g. see Figure 4, page 8).

Original: [Table]
Revised: [Table]

Original: Probability Flow Momentum Sampler For CelebA- 64 , we reported results only for the (stochastic) Momentum Sampler in the main body of the paper (see Figure 3).
Revised: Probability Flow Momentum Sampler For CelebA- 64 , we reported results only for the (stochastic) Momentum Sampler in the main body of the paper (see Figure 3).

Original: We report here results for the deterministic counterpart of this sampler, Probability Flow Momentum Sampler, summarized in Table 4.
Revised: We report here results for the deterministic counterpart of this sampler, Probability Flow Momentum Sampler, summarized in Table 4.

Original: The deterministic version of the Momentum Sampler seems to be performing slightly worse than the stochastic version – FID jumps from 1 . 85 to 2 .
Revised: The deterministic version of the Momentum Sampler seems to be performing slightly worse than the stochastic version – FID jumps from 1 . 85 to 2 .

Original: However, this sampler, similar to the DDIM-type sampler, maintains more performance as we increase the number of steps.
Revised: However, this sampler, similar to the DDIM-type sampler, maintains more performance as we increase the number of steps.

Original: This is an important advantage for the deterministic sampling methods since practitioners need to put less effort into tuning the NFEs to get the best result.
Revised: This is an important advantage for the deterministic sampling methods since practitioners need to put less effort into tuning the NFEs to get the best result.

Original: [Table]
Revised: [Table]

Original: Predictor-Corrector Samplers
Revised: Predictor-Corrector Samplers

Original: Finally, we perform experiments with Predictor-Corrector samplers, as proposed in Song et al.
Revised: Finally, we perform experiments with Predictor-Corrector samplers, as proposed in Song et al.

Original: (2021b).
Revised: (2021b).

Original: The idea is that we are alternating at each diffusion step between two different samplers.
Revised: The idea is that we are alternating at each diffusion step between two different samplers.

Original: We experiment with a DDIM Predictor and a Probability Flow Momentum Sampler corrector.
Revised: We experiment with a DDIM Predictor and a Probability Flow Momentum Sampler corrector.

Original: The results are summarized in Table 5.
Revised: The results are summarized in Table 5.

Original: The Predictor-Corrector sampler maintains some of the beneﬁts of both samplers.
Revised: The Predictor-Corrector sampler maintains some of the beneﬁts of both samplers.

Original: Namely, for low number of function evaluations it has better performance than the Probability Flow Momentum Sampler (beneﬁt coming from the
Revised: Namely, for low number of function evaluations it has better performance than the Probability Flow Momentum Sampler (beneﬁt coming from the

Original: DDIM sampler) and for higher number of function evaluations performance is better than the DDIM sampler (beneﬁt coming from the Probability Flow Momentum Sampler).
Revised: DDIM sampler) and for higher number of function evaluations performance is better than the DDIM sampler (beneﬁt coming from the Probability Flow Momentum Sampler).

Original: There is a spot, at
Revised: There is a spot, at

Original: NFEs, where the Predictor-Corrector sampler is better than both the Predictor and the Corrector.
Revised: NFEs, where the Predictor-Corrector sampler is better than both the Predictor and the Corrector.

Original: We encourage future research in identifying even better pairs of Predictor-Correctors that might outperform both the Predictor and the Corrector in some regime.
Revised: We encourage future research in identifying even better pairs of Predictor-Correctors that might outperform both the Predictor and the Corrector in some regime.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: . N EAREST S AMPLES IN T RAINING D ATA
Revised: . N EAREST S AMPLES IN T RAINING D ATA

Original: To verify that our model does not simply memorize the training dataset, we present generated images from our model and their nearest neighbor (L2 pixel distance) from the dataset.
Revised: To verify that our model does not simply memorize the training dataset, we present generated images from our model and their nearest neighbor (L2 pixel distance) from the dataset.

Original: The results are shown in Figure 10.
Revised: The results are shown in Figure 10.

Original: [Figure]
Revised: [Figure]

Original: F. U NCURATED SAMPLES
Revised: F. U NCURATED SAMPLES

Original: Figures 11 and 12 show more uncurated samples from our trained models on CelebA and CIFAR-10.
Revised: Figures 11 and 12 show more uncurated samples from our trained models on CelebA and CIFAR-10.

Original: G I MPLEMENTATION OF D EGRADATION O PERATORS
Revised: G I MPLEMENTATION OF D EGRADATION O PERATORS

Original: Blurring.
Revised: Blurring.

Original: The blurring operator is implemented as a convolution with a Gaussian kernel.
Revised: The blurring operator is implemented as a convolution with a Gaussian kernel.

Original: The Gaussian kernel is truncated to have ﬁxed support size ( 161 × 161 pixels), and normalized to have area one.
Revised: The Gaussian kernel is truncated to have ﬁxed support size ( 161 × 161 pixels), and normalized to have area one.

Original: The standard deviation of the Gaussian kernel sigma blur (parameter controlling the strength of the blur) is computed using the optimal transport optimization for the scheduling.
Revised: The standard deviation of the Gaussian kernel sigma blur (parameter controlling the strength of the blur) is computed using the optimal transport optimization for the scheduling.

Original: The obtained schedule for the parameter sigma blur in CIFAR10 can be accessed from the following anonymous url: https://drive.google.com/file/d/ 192kdbj9oq1EGCm7KY52QZj--x0g5Elgs .
Revised: The obtained schedule for the parameter sigma blur in CIFAR10 can be accessed from the following anonymous url: https://drive.google.com/file/d/ 192kdbj9oq1EGCm7KY52QZj--x0g5Elgs .

Original: def generate_gaussian_kernel(sigma_blur, half_size=80): v = np.arange(-half_size, half_size + 1) x, y =
Revised: def generate_gaussian_kernel(sigma_blur, half_size=80): v = np.arange(-half_size, half_size + 1) x, y =

Original: np.meshgrid(v, v) k
Revised: np.meshgrid(v, v) k

Original: = np.exp(-(x**2 + y**2) / (2*sigma_blur**2))
Revised: = np.exp(-(x**2 + y**2) / (2*sigma_blur**2))

Original: k
Revised: k

Original: = k / np.sum(k) return k
Revised: = k / np.sum(k) return k

Original: Masking.
Revised: Masking.

Original: The masking operator is implemented using a centered binary mask that sets to zero some percentage of pixels in the image.
Revised: The masking operator is implemented using a centered binary mask that sets to zero some percentage of pixels in the image.

Original: The scheduling with the masking operators can be accessed from the following anonymous url: https://drive.google.com/file/d/ 1YjzYKgivhvbHOzTABOYDiUoHbzq60Ozy .
Revised: The scheduling with the masking operators can be accessed from the following anonymous url: https://drive.google.com/file/d/ 1YjzYKgivhvbHOzTABOYDiUoHbzq60Ozy .

Original: Noise.
Revised: Noise.

Original: The noise degradation is implemented by adding to the blurred or masked image white Gaussian noise having standard deviation sigma noise.
Revised: The noise degradation is implemented by adding to the blurred or masked image white Gaussian noise having standard deviation sigma noise.

Original: The scheduling of the noise can be accessed from the following anonymous url: https://drive.google.com/file/d/ 17UwFlJyp4euQeKVAVmRRHg8HL52anapfk .
Revised: The scheduling of the noise can be accessed from the following anonymous url: https://drive.google.com/file/d/ 17UwFlJyp4euQeKVAVmRRHg8HL52anapfk .