Original: S OFT D IFFUSION
Revised: S OFT D IFFUSION

Original: S CORE M ATCHING F OR G ENERAL C ORRUPTIONS
Revised: S CORE M ATCHING F OR G ENERAL C ORRUPTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We deﬁne a broader family of corruption processes that generalizes previously known diffusion models.
Revised: We deﬁne a broader family of corruption processes that generalizes previously known diffusion models.

Original: To reverse these general diffusions, we propose a new objective called Soft Score Matching that provably learns the score function for any linear corruption process and yields state of the art results for CelebA. Soft
Revised: To reverse these general diffusions, we propose a new objective called Soft Score Matching that provably learns the score function for any linear corruption process and yields state of the art results for CelebA. Soft

Original: Score Matching incorporates the degradation process in the network.
Revised: Score Matching incorporates the degradation process in the network.

Original: Our new loss trains the model to predict a clean image, that after corruption , matches the diffused observation.
Revised: Our new loss trains the model to predict a clean image, that after corruption , matches the diffused observation.

Original: We show that our objective learns the gradient of the likelihood under suitable regularity conditions for a family of corruption processes.
Revised: We show that our objective learns the gradient of the likelihood under suitable regularity conditions for a family of corruption processes.

Original: We further develop a principled way to select the corruption levels for general diffusion processes and a novel sampling method that we call Momentum Sampler.
Revised: We further develop a principled way to select the corruption levels for general diffusion processes and a novel sampling method that we call Momentum Sampler.

Original: We show experimentally that our framework works for general linear corruption processes, such as Gaussian blur and masking.
Revised: We show experimentally that our framework works for general linear corruption processes, such as Gaussian blur and masking.

Original: We achieve state-of-the-art FID score 1 . 85 on
Revised: We achieve state-of-the-art FID score 1 . 85 on

Original: CelebA-64, outperforming all previous linear diffusion models.
Revised: CelebA-64, outperforming all previous linear diffusion models.

Original: We also show signiﬁcant computational beneﬁts compared to vanilla denoising diffusion.
Revised: We also show signiﬁcant computational beneﬁts compared to vanilla denoising diffusion.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Score-based models (Song & Ermon, 2019; 2020; Song et al., 2021b) and Denoising Diffusion
Revised: Score-based models (Song & Ermon, 2019; 2020; Song et al., 2021b) and Denoising Diffusion

Original: Probabilistic Models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021a) are two powerful classes of generative models that produce samples by inverting a diffusion process.
Revised: Probabilistic Models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021a) are two powerful classes of generative models that produce samples by inverting a diffusion process.

Original: These two classes have been uniﬁed under a single framework (Song et al., 2021b) and are widely known as diffusion models.
Revised: These two classes have been uniﬁed under a single framework (Song et al., 2021b) and are widely known as diffusion models.

Original: Diffusion modeling has found great success in a wide range of applications (Croitoru et al., 2022;
Revised: Diffusion modeling has found great success in a wide range of applications (Croitoru et al., 2022;

Original: Yang et al., 2022), including image (Saharia et al., 2022a; Ramesh et al., 2022; Rombach et al., 2022; Dhariwal & Nichol, 2021), audio (Kong et al., 2021; Richter et al., 2022; Serr ` a et al., 2022), video generation (Ho et al., 2022b), as well as solving inverse problems (Daras et al., 2022; Kadkhodaie & Simoncelli, 2021; Kawar et al., 2022; 2021; Jalal et al., 2021; Saharia et al., 2022b; Laumont et al., 2022; Whang et al., 2022; Chung et al., 2022).
Revised: Yang et al., 2022), including image (Saharia et al., 2022a; Ramesh et al., 2022; Rombach et al., 2022; Dhariwal & Nichol, 2021), audio (Kong et al., 2021; Richter et al., 2022; Serr ` a et al., 2022), video generation (Ho et al., 2022b), as well as solving inverse problems (Daras et al., 2022; Kadkhodaie & Simoncelli, 2021; Kawar et al., 2022; 2021; Jalal et al., 2021; Saharia et al., 2022b; Laumont et al., 2022; Whang et al., 2022; Chung et al., 2022).

Original: Karras et al.
Revised: Karras et al.

Original: (2022) analyze the design space of diffusion models.
Revised: (2022) analyze the design space of diffusion models.

Original: The authors identify three stages: i) the noise scheduling, ii) the network parametrization (each one leads to a different loss function), iii) the sampling algorithm.
Revised: The authors identify three stages: i) the noise scheduling, ii) the network parametrization (each one leads to a different loss function), iii) the sampling algorithm.

Original: We argue that there is one more important step: choosing how to corrupt .
Revised: We argue that there is one more important step: choosing how to corrupt .

Original: Typically, the diffusion is additive noise of different magnitudes (and sometimes input rescalings).
Revised: Typically, the diffusion is additive noise of different magnitudes (and sometimes input rescalings).

Original: There have been a few recent attempts to use different corruptions (Deasy et al., 2021;
Revised: There have been a few recent attempts to use different corruptions (Deasy et al., 2021;

Original: Hoogeboom et al., 2022a;b; Avrahami et al., 2022; Nachmani et al., 2021; Johnson et al., 2021;
Revised: Hoogeboom et al., 2022a;b; Avrahami et al., 2022; Nachmani et al., 2021; Johnson et al., 2021;

Original: Lee et al., 2022;
Revised: Lee et al., 2022;

Original: Ye et al., 2022), but the results are usually inferior to diffusion with additive noise.
Revised: Ye et al., 2022), but the results are usually inferior to diffusion with additive noise.

Original: Also, a common framework on how to properly design general corruption processes is missing.
Revised: Also, a common framework on how to properly design general corruption processes is missing.

Original: We present such a principled framework for learning to invert a general class of corruption processes.
Revised: We present such a principled framework for learning to invert a general class of corruption processes.

Original: We propose a new objective called Soft Score Matching that provably learns the score for any regular linear corruption process.
Revised: We propose a new objective called Soft Score Matching that provably learns the score for any regular linear corruption process.

Original: Soft Score Matching incorporates the ﬁltering process in the network and trains the model to predict a clean image that after corruption matches the diffused observation .
Revised: Soft Score Matching incorporates the ﬁltering process in the network and trains the model to predict a clean image that after corruption matches the diffused observation .

Original: Our theoretical results show that Soft Score Matching learns the score (i.e. likelihood gradients) for corruption processes that satisfy a regularity condition that we identify: the diffusion must transform any image into any other image with nonzero likelihood.
Revised: Our theoretical results show that Soft Score Matching learns the score (i.e. likelihood gradients) for corruption processes that satisfy a regularity condition that we identify: the diffusion must transform any image into any other image with nonzero likelihood.

Original: Using our method and Gaussian Blur paired with little noise as the diffusion mechanism, we achieve state-of-the-art FID on CelebA (FID 1.85).
Revised: Using our method and Gaussian Blur paired with little noise as the diffusion mechanism, we achieve state-of-the-art FID on CelebA (FID 1.85).

Original: We also show that our corruption process leads to generative models that are faster compared to vanilla Gaussian denoising diffusion.
Revised: We also show that our corruption process leads to generative models that are faster compared to vanilla Gaussian denoising diffusion.

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t= 
Revised: t= t

Original: [Figure]
Revised: [Figure]

Original: t

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Our contributions: a) We propose a learning objective that: i) provably learns the score for a wide family of regular diffusion processes and ii) enables learning under limited randomness in the diffusion.
Revised: Our contributions: a) We propose a learning objective that: i) provably learns the score for a wide family of regular diffusion processes and ii) enables learning under limited randomness in the diffusion.

Original: b) We present a principled way to select the intermediate distributions.
Revised: b) We present a principled way to select the intermediate distributions.

Original: Our method minimizes the Wasserstein distance along the path from the initial to the ﬁnal distribution.
Revised: Our method minimizes the Wasserstein distance along the path from the initial to the ﬁnal distribution.

Original: c)
Revised: c)

Original: We propose a novel sampling method that we call Momentum Sampler: our sampler uses a convex combination of corruptions at different diffusion levels and is inspired by momentum methods in optimization.
Revised: We propose a novel sampling method that we call Momentum Sampler: our sampler uses a convex combination of corruptions at different diffusion levels and is inspired by momentum methods in optimization.

Original: d) We train models on CelebA and CIFAR-10.
Revised: d) We train models on CelebA and CIFAR-10.

Original: Our trained models on CelebA achieve a new state-of-the-art FID score of 1 . 85 for linear diffusion models while being signiﬁcantly faster compared to models trained with vanilla Gaussian denoising diffusion.
Revised: Our trained models on CelebA achieve a new state-of-the-art FID score of 1 . 85 for linear diffusion models while being signiﬁcantly faster compared to models trained with vanilla Gaussian denoising diffusion.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Diffusion models are generative models that produce samples by inverting a corruption process.
Revised: Diffusion models are generative models that produce samples by inverting a corruption process.

Original: The corruption level is typically indexed by a time t , with t = 0 corresponding to clean and t = 1 to fully corrupted images.
Revised: The corruption level is typically indexed by a time t , with t = 0 corresponding to clean and t = 1 to fully corrupted images.

Original: The diffusion process can be discrete or continuous.
Revised: The diffusion process can be discrete or continuous.

Original: The two general classes of diffusion models are Score-Based Models (Song & Ermon, 2019; 2020; Song et al., 2021b) and
Revised: The two general classes of diffusion models are Score-Based Models (Song & Ermon, 2019; 2020; Song et al., 2021b) and

Original: Denoising Diffusion Probabilistic Models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020).
Revised: Denoising Diffusion Probabilistic Models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020).

Original: The typical diffusion in score-based modeling is additive noise of increasing magnitude.
Revised: The typical diffusion in score-based modeling is additive noise of increasing magnitude.

Original: The perturbation kernel at time t is: q t ( x t | x 0 ) =
Revised: The perturbation kernel at time t is: q t ( x t | x 0 ) =

Original: N ( x t | µ = x 0 , Σ
Revised: N ( x t | µ = x 0 , Σ

Original: = σ 2 t I ) , where x 0 ∼ q 0 is a clean image.
Revised: = σ 2 t I ) , where x 0 ∼ q 0 is a clean image.

Original: Score models are trained with the Denoising Score Matching (DSM) objective:
Revised: Score models are trained with the Denoising Score Matching (DSM) objective:

Original: [Equation]
Revised: [Equation]

Original: where w t scales differently the weights of the inner objectives.
Revised: where w t scales differently the weights of the inner objectives.

Original: If we train for each noise level t independently, given enough data and model capacity, the network is guaranteed to recover the gradient of the log likelihood (Vincent, 2011), known as the score function.
Revised: If we train for each noise level t independently, given enough data and model capacity, the network is guaranteed to recover the gradient of the log likelihood (Vincent, 2011), known as the score function.

Original: In other words, the model s θ ( x t | t ) is trained such that: s θ ( x t | t )
Revised: In other words, the model s θ ( x t | t ) is trained such that: s θ ( x t | t )

Original: ≈ ∇ x t log q t ( x t ) .
Revised: ≈ ∇ x t log q t ( x t ) .

Original: In practice, we use parameter sharing and conditioning on time t to learn all the scores.
Revised: In practice, we use parameter sharing and conditioning on time t to learn all the scores.

Original: Once the model is trained, we start from a sample of the ﬁnal distribution, q 1 , and then use the learned score to gradually denoise it (Song &
Revised: Once the model is trained, we start from a sample of the ﬁnal distribution, q 1 , and then use the learned score to gradually denoise it (Song &

Original: Ermon, 2019; 2020).
Revised: Ermon, 2019; 2020).

Original: The ﬁnal variance σ 21 is selected to be very large such that the distribution qis approximately Gaussian, i.e. the signal to noise ratio tends to 0 .
Revised: The ﬁnal variance σ 21 is selected to be very large such that the distribution qis approximately Gaussian, i.e. the signal to noise ratio tends to 0 .

Original: DDPMs corrupt by rescaling the input images and by adding noise.
Revised: DDPMs corrupt by rescaling the input images and by adding noise.

Original: The corruption can be modelled with a Markov chain with perturbation kernel q t ( x t | x t − ∆ t ) =
Revised: The corruption can be modelled with a Markov chain with perturbation kernel q t ( x t | x t − ∆ t ) =

Original: N ( x t | µ = √ 1 − β t x t − ∆ t , Σ = β t I ) .
Revised: N ( x t | µ = √ 1 − β t x t − ∆ t , Σ = β t I ) .

Original: Typically, β 1 = 1 and hence q 1 = N (0 , I ) .
Revised: Typically, β 1 = 1 and hence q 1 = N (0 , I ) .

Original: DDPMs are also trained with the DSM objective which is derived by minimizing an evidence lower bound (ELBO)
Revised: DDPMs are also trained with the DSM objective which is derived by minimizing an evidence lower bound (ELBO)

Original: (Ho et al., 2020).
Revised: (Ho et al., 2020).

Original: In their seminal work, Song et al.
Revised: In their seminal work, Song et al.

Original: (2021b) observe that the diffusions of both Score-Based models and DDPMs can be expressed as solutions of Stochastic Differential Equations (SDEs) of the form:
Revised: (2021b) observe that the diffusions of both Score-Based models and DDPMs can be expressed as solutions of Stochastic Differential Equations (SDEs) of the form:

Original: [Equation]
Revised: [Equation]

Original: where w is the standard Wiener process.
Revised: where w is the standard Wiener process.

Original: Particularly, Score-Based models use: f ( x , t ) = 0 , g ( t ) =
Revised: Particularly, Score-Based models use: f ( x , t ) = 0 , g ( t ) =

Original: (cid:113)
Revised: (cid:113)

Original: d σ 2 t d t and DDPMs use: f ( x , t ) =
Revised: d σ 2 t d t and DDPMs use: f ( x , t ) =

Original: − 12 β t x , g ( t ) = √ β t .
Revised: − 12 β t x , g ( t ) = √ β t .

Original: As explained earlier, for Score-Based models we need large noise at the end for the ﬁnal distribution to approximate a normal distribution.
Revised: As explained earlier, for Score-Based models we need large noise at the end for the ﬁnal distribution to approximate a normal distribution.

Original: Hence, the corresponding SDE is named V ariance E xploding (VE) SDE (Song et al., 2021b).
Revised: Hence, the corresponding SDE is named V ariance E xploding (VE) SDE (Song et al., 2021b).

Original: DDPMs usually have a ﬁnal distribution of unit variance and hence their SDE is known as the
Revised: DDPMs usually have a ﬁnal distribution of unit variance and hence their SDE is known as the

Original: V ariance P reserving (VP) SDE (Song et al., 2021b).
Revised: V ariance P reserving (VP) SDE (Song et al., 2021b).

Original: Song et al.
Revised: Song et al.

Original: (2021b) also propose another SDE with bounded variance, the subVP-SDE, that experimentally yields better likelihoods.
Revised: (2021b) also propose another SDE with bounded variance, the subVP-SDE, that experimentally yields better likelihoods.

Original: For both Score-Based models and DDPMs, Eq.
Revised: For both Score-Based models and DDPMs, Eq.

Original: (2) is known as the Forward SDE.
Revised: (2) is known as the Forward SDE.

Original: This SDE is reversible Anderson (1982) and the Reverse SDE is given below:
Revised: This SDE is reversible Anderson (1982) and the Reverse SDE is given below:

Original: [Equation]
Revised: [Equation]

Original: where ¯ w is the standard Wiener process when the time ﬂows in the reverse direction.
Revised: where ¯ w is the standard Wiener process when the time ﬂows in the reverse direction.

Original: Typically, ∇ x log q t ( x ) is approximated by s θ ( x t | t ) and samples are generated by solving the Reverse
Revised: Typically, ∇ x log q t ( x ) is approximated by s θ ( x t | t ) and samples are generated by solving the Reverse

Original: SDE (Song et al., 2021b).
Revised: SDE (Song et al., 2021b).

Original:  M ETHOD
Revised:  M ETHOD

Original: Our framework for training diffusion models with more general corruptions includes three components: i) the training objective, ii) the sampling, iii) the scheduling of the corruption mechanism.
Revised: Our framework for training diffusion models with more general corruptions includes three components: i) the training objective, ii) the sampling, iii) the scheduling of the corruption mechanism.

Original: 3. T RAINING O BJECTIVE
Revised: 3. T RAINING O BJECTIVE

Original: We study corruption processes of the form:
Revised: We study corruption processes of the form:

Original: [Equation]
Revised: [Equation]

Original: where C t : R n → R n is a deterministic linear operator, η t is a Wiener process, and s t is a nonnegative scalar controlling the noise level at time t , and x 0 ∼ q 0 ( x ) .
Revised: where C t : R n → R n is a deterministic linear operator, η t is a Wiener process, and s t is a nonnegative scalar controlling the noise level at time t , and x 0 ∼ q 0 ( x ) .

Original: We further denote with σ 2 t the variance of the noise at level t and we assume that it is a non-decreasing function of t .
Revised: We further denote with σ 2 t the variance of the noise at level t and we assume that it is a non-decreasing function of t .

Original: Unless stated otherwise, we assume that time is continuous and runs from t = 0 to t = 1 .
Revised: Unless stated otherwise, we assume that time is continuous and runs from t = 0 to t = 1 .

Original: Additionally, we assume that at t = 0 , we have C 

Revised: Additionally, we assume that at t = 0 , we have C

Original: = I n × n and σ 0 = 0 , i.e. t = 0 corresponds to natural images.
Revised: = I n × n and σ 0 = 0 , i.e. t = 0 corresponds to natural images.

Original: We also assume that recovering x 0 from x t is harder as t gets larger (i.e., entropy of q t ( x 0 | x t ) increases with t ).
Revised: We also assume that recovering x 0 from x t is harder as t gets larger (i.e., entropy of q t ( x 0 | x t ) increases with t ).

Original: Eq.
Revised: Eq.

Original: (4) deﬁnes a general class of diffusion processes, that includes (as special cases) the VE,
Revised: (4) deﬁnes a general class of diffusion processes, that includes (as special cases) the VE,

Original: VP and subVP SDEs used in Song et al.
Revised: VP and subVP SDEs used in Song et al.

Original: (2021b).
Revised: (2021b).

Original: Our diffusion is the sum of a deterministic linear corruption of x 0 and a stochastic part that progressively adds noise.
Revised: Our diffusion is the sum of a deterministic linear corruption of x 0 and a stochastic part that progressively adds noise.

Original: For any corruption process of this family, we are interested in learning the scores, i.e. ∇ x t log q t ( x t ) for all t .
Revised: For any corruption process of this family, we are interested in learning the scores, i.e. ∇ x t log q t ( x t ) for all t .

Original: For the vanilla Gaussian denoising diffusion, the celebrated result of Vincent (2011) shows that we only need access to the gradient of the conditional log-likelihood, ∇ x t log q t ( x t | x 0 ) , in order to learn the score, ∇ x t log q t ( x t ) .
Revised: For the vanilla Gaussian denoising diffusion, the celebrated result of Vincent (2011) shows that we only need access to the gradient of the conditional log-likelihood, ∇ x t log q t ( x t | x 0 ) , in order to learn the score, ∇ x t log q t ( x t ) .

Original: By revisiting the proof of Vincent (2011), we ﬁnd that this is actually true for a wide set of corruption processes, as long as some mild technical conditions are satisﬁed.
Revised: By revisiting the proof of Vincent (2011), we ﬁnd that this is actually true for a wide set of corruption processes, as long as some mild technical conditions are satisﬁed.

Original: In fact, the following general Theorem holds:
Revised: In fact, the following general Theorem holds:

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Let q 0 , q t be two distributions in R n .
Revised: Let q 0 , q t be two distributions in R n .

Original: Assume that all conditional distributions, q t ( x t | x 0 ) , are fully supported and differentiable in R n .
Revised: Assume that all conditional distributions, q t ( x t | x 0 ) , are fully supported and differentiable in R n .

Original: Let:
Revised: Let:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then, there is a universal constant C (that does not depend on θ ) such that: J 1 ( θ ) =
Revised: Then, there is a universal constant C (that does not depend on θ ) such that: J 1 ( θ ) =

Original: J 2 ( θ ) + C .
Revised: J 2 ( θ ) + C .

Original: Theorem 3.1 implies that minimizing the second function is equivalent to minimizing the ﬁrst one.
Revised: Theorem 3.1 implies that minimizing the second function is equivalent to minimizing the ﬁrst one.

Original: The second function is nothing else than the DSM objective.
Revised: The second function is nothing else than the DSM objective.

Original: Our main observation is that noise is not always necessary for learning the score using the DSM objective.
Revised: Our main observation is that noise is not always necessary for learning the score using the DSM objective.

Original: A necessary condition is that the corruption process gives non-zero probability to all x t for any image x 0 .
Revised: A necessary condition is that the corruption process gives non-zero probability to all x t for any image x 0 .

Original: This is easily achieved by adding Gaussian noise, but this is not the only option.
Revised: This is easily achieved by adding Gaussian noise, but this is not the only option.

Original: The proof of this Theorem is deferred in the Appendix and it is following the calculations of Vincent (2011).
Revised: The proof of this Theorem is deferred in the Appendix and it is following the calculations of Vincent (2011).

Original: Network parametrization.
Revised: Network parametrization.

Original: For the class of diffusion processes given by Eq.
Revised: For the class of diffusion processes given by Eq.

Original: (4), we have that: q t ( x t | x 0 ) =
Revised: (4), we have that: q t ( x t | x 0 ) =

Original: N (cid:0) x t ; µ = C t x 0 , Σ
Revised: N (cid:0) x t ; µ = C t x 0 , Σ

Original: = σ 2 t I (cid:1) and hence the objective becomes:
Revised: = σ 2 t I (cid:1) and hence the objective becomes:

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: As shown, the objective of the model is to predict the (normalized) difference, C t x 0 − x t , which is actually the noise, η t .
Revised: As shown, the objective of the model is to predict the (normalized) difference, C t x 0 − x t , which is actually the noise, η t .

Original: We argue that even though this objective is theoretically grounded, in many cases, it would not work in practice because we would need inﬁnite samples to actually learn the vector-ﬁeld ∇ x t log q t ( x t ) in a way that would allow sampling.
Revised: We argue that even though this objective is theoretically grounded, in many cases, it would not work in practice because we would need inﬁnite samples to actually learn the vector-ﬁeld ∇ x t log q t ( x t ) in a way that would allow sampling.

Original: Assume that the corruption process is blurring (at different levels) paired with additive noise of small magnitude.
Revised: Assume that the corruption process is blurring (at different levels) paired with additive noise of small magnitude.

Original: The objective written in Eq.
Revised: The objective written in Eq.

Original: (7) learns the distributions of blurry (and slightly noisy images) by just removing noise.
Revised: (7) learns the distributions of blurry (and slightly noisy images) by just removing noise.

Original: Hence, in practice we might only learn these distributions locally (around the blurry images) and hence we might not be able to reduce the blurriness.
Revised: Hence, in practice we might only learn these distributions locally (around the blurry images) and hence we might not be able to reduce the blurriness.

Original: This point might be better understood after we present our Sampling Method in Section 3.2.
Revised: This point might be better understood after we present our Sampling Method in Section 3.2.

Original: To account for this problem, we propose a network reparametrization which leverages that we know the linear corruption mechanism, C t .
Revised: To account for this problem, we propose a network reparametrization which leverages that we know the linear corruption mechanism, C t .

Original: Speciﬁcally, we propose the following parametrization:
Revised: Speciﬁcally, we propose the following parametrization:

Original: [Equation]
Revised: [Equation]

Original: Crucially, the network incorporates the corruption process.
Revised: Crucially, the network incorporates the corruption process.

Original: The loss becomes:
Revised: The loss becomes:

Original: [Equation]
Revised: [Equation]

Original: When C t is a blurring matrix, this loss function is the MSE between the blurred prediction of h θ and the blurred clean image.
Revised: When C t is a blurring matrix, this loss function is the MSE between the blurred prediction of h θ and the blurred clean image.

Original: Finally, as observed in previous works (Song & Ermon, 2019; Ho et al., 2020; Karras et al., 2022), the optimization landscape becomes smoother when we are predicting the residual, instead of the clean image directly.
Revised: Finally, as observed in previous works (Song & Ermon, 2019; Ho et al., 2020; Karras et al., 2022), the optimization landscape becomes smoother when we are predicting the residual, instead of the clean image directly.

Original: This corresponds to the additional reparametrization:
Revised: This corresponds to the additional reparametrization:

Original: [Equation]
Revised: [Equation]

Original: which leads to the ﬁnal form of our loss function:
Revised: which leads to the ﬁnal form of our loss function:

Original: [Equation]
Revised: [Equation]

Original: (11) where r t is the residual with respect to the clean image, i.e. r t = x 0 − x t .
Revised: (11) where r t is the residual with respect to the clean image, i.e. r t = x 0 − x t .

Original: Following prior work, we use a single network conditioned on time t that is optimized for all L ( t ) .
Revised: Following prior work, we use a single network conditioned on time t that is optimized for all L ( t ) .

Original: Hence, the total loss is:
Revised: Hence, the total loss is:

Original: [Equation]
Revised: [Equation]

Original: where the weights are usually chosen to be 1 or 1 /σ 2 t (Karras et al., 2022; Kingma et al., 2021).
Revised: where the weights are usually chosen to be 1 or 1 /σ 2 t (Karras et al., 2022; Kingma et al., 2021).

Original: We call our training objective Soft Score Matching .
Revised: We call our training objective Soft Score Matching .

Original: The name is inspired from “soft ﬁltering” a term used in photography to denote an image ﬁlter that removes ﬁne details (e.g., blur, fading, etc).
Revised: The name is inspired from “soft ﬁltering” a term used in photography to denote an image ﬁlter that removes ﬁne details (e.g., blur, fading, etc).

Original: As in the Denoising Score Matching, the network is essentially trained to predict the residual to the clean image, but in our case, the loss is in the ﬁltered space.
Revised: As in the Denoising Score Matching, the network is essentially trained to predict the residual to the clean image, but in our case, the loss is in the ﬁltered space.

Original: When there is no ﬁltering matrix, i.e.
Revised: When there is no ﬁltering matrix, i.e.

Original: C t =
Revised: C t =

Original: I , we recover the DSM objective used in (Song & Ermon, 2019; 2020; Song et al., 2021b).
Revised: I , we recover the DSM objective used in (Song & Ermon, 2019; 2020; Song et al., 2021b).

Original: Comparison with objectives used in other works.
Revised: Comparison with objectives used in other works.

Original: Previous (Anonymous, 2022) or concurrent (Bansal et al., 2022;
Revised: Previous (Anonymous, 2022) or concurrent (Bansal et al., 2022;

Original: Rissanen et al., 2022; Hoogeboom & Salimans, 2022) works that consider different degradations than Gaussian Diffusion, use the heuristic objective of predicting the clean image, i.e. they minimize: || φ θ ( x t | t ) − r t || .
Revised: Rissanen et al., 2022; Hoogeboom & Salimans, 2022) works that consider different degradations than Gaussian Diffusion, use the heuristic objective of predicting the clean image, i.e. they minimize: || φ θ ( x t | t ) − r t || .

Original: This is actually an upper-bound on our loss, i.e. || C t ( φ θ ( x t | t ) − r t ) ||
Revised: This is actually an upper-bound on our loss, i.e. || C t ( φ θ ( x t | t ) − r t ) ||

Original: ≤ || C t |||| φ θ ( x t | t ) − r t || .
Revised: ≤ || C t |||| φ θ ( x t | t ) − r t || .

Original: Since the spectral norm, || C t || , is ﬁxed, one can optimize for the upper-bound by minimizing || φ θ ( x t | t ) − r t || .
Revised: Since the spectral norm, || C t || , is ﬁxed, one can optimize for the upper-bound by minimizing || φ θ ( x t | t ) − r t || .

Original: Instead, Soft Score Matching optimizes directly for learning the score.
Revised: Instead, Soft Score Matching optimizes directly for learning the score.

Original: Experimentally, Soft Score Matching outperforms (ours
Revised: Experimentally, Soft Score Matching outperforms (ours

Original: FID: 1 . 85 , theirs: 5 . 91 ), under the exact same setting, this simple baseline (see Experiments).
Revised: FID: 1 . 85 , theirs: 5 . 91 ), under the exact same setting, this simple baseline (see Experiments).

Original: 3. S AMPLING
Revised: 3. S AMPLING

Original: Require: p 1 , φ θ , C t , σ t , ∆ t x 1 ∼ p 1 ( x ) for t = 1 to 0 with step − ∆ t do ˆ x 0 =
Revised: Require: p 1 , φ θ , C t , σ t , ∆ t x 1 ∼ p 1 ( x ) for t = 1 to 0 with step − ∆ t do ˆ x 0 =

Original: φ θ ( x t | t ) + x t η t ∼ N ( 0 , I ) x t − ∆ t ← C t − ∆ t ˆ x 0 + σ t − ∆ t η t end for return x
Revised: φ θ ( x t | t ) + x t η t ∼ N ( 0 , I ) x t − ∆ t ← C t − ∆ t ˆ x 0 + σ t − ∆ t η t end for return x

Original: Naive Sampler.
Revised: Naive Sampler.

Original: Once the model is trained, we need a way to generate samples.
Revised: Once the model is trained, we need a way to generate samples.

Original: The simplest idea is take advantage of the fact that our trained model, φ θ ( x t | t ) , can be used to get an estimate of the clean image, ˆ x 0 .
Revised: The simplest idea is take advantage of the fact that our trained model, φ θ ( x t | t ) , can be used to get an estimate of the clean image, ˆ x 0 .

Original: Hence, whenever we want to move from a corruption level t to a corruption level t − ∆ t , we can feed the image x t to the model to get an estimate of the clean image, ˆ x 0 , and then corrupt back to level t − ∆ t .
Revised: Hence, whenever we want to move from a corruption level t to a corruption level t − ∆ t , we can feed the image x t to the model to get an estimate of the clean image, ˆ x 0 , and then corrupt back to level t − ∆ t .

Original: This idea is summarized in Algorithm 1.
Revised: This idea is summarized in Algorithm 1.

Original: Momentum Sampler.
Revised: Momentum Sampler.

Original: As we will show in the Experiments section, the naive sampler we presented above leads to generated images that lack diversity.
Revised: As we will show in the Experiments section, the naive sampler we presented above leads to generated images that lack diversity.

Original: We propose an simple, yet novel, alternative method for sampling from the general linear diffusion model presented in Eq.
Revised: We propose an simple, yet novel, alternative method for sampling from the general linear diffusion model presented in Eq.

Original: Our method is inspired by the continuous formulation of diffusion models that is introduced in Song et al.
Revised: Our method is inspired by the continuous formulation of diffusion models that is introduced in Song et al.

Original: (2021b).
Revised: (2021b).

Original: To develop our idea, we start with the toy setting where the dataset contains one single image, α ∈ R n .
Revised: To develop our idea, we start with the toy setting where the dataset contains one single image, α ∈ R n .

Original: Then the corruption of Eq.
Revised: Then the corruption of Eq.

Original: (4) can be seen as the solution to the following SDE:
Revised: (4) can be seen as the solution to the following SDE:

Original: [Equation]
Revised: [Equation]

Original: (13) where w is the standard Wiener process.
Revised: (13) where w is the standard Wiener process.

Original: This is a special case of the It ˆ o SDE: d x = f ( x , t )d t + g ( t )d w , that appears in Song et al.
Revised: This is a special case of the It ˆ o SDE: d x = f ( x , t )d t + g ( t )d w , that appears in Song et al.

Original: (2021b).
Revised: (2021b).

Original: Particularly, f ( x , t ) = ˙
Revised: Particularly, f ( x , t ) = ˙

Original: C t α and g ( t ) = (cid:113) d( σ 2 t ) d t .
Revised: C t α and g ( t ) = (cid:113) d( σ 2 t ) d t .

Original: We note that crucially we treat α as a constant and hence f ( x , t ) does not depend on previous values of x .
Revised: We note that crucially we treat α as a constant and hence f ( x , t ) does not depend on previous values of x .

Original: Under this setting, this SDE describes a diffusion process that is reversible Anderson (1982).
Revised: Under this setting, this SDE describes a diffusion process that is reversible Anderson (1982).

Original: The reverse is also a diffusion process and it is given below:
Revised: The reverse is also a diffusion process and it is given below:

Original: [Equation]
Revised: [Equation]

Original: where ¯ w is a standard Wiener process when time ﬂows backwards from t = 1 to t = 0 .
Revised: where ¯ w is a standard Wiener process when time ﬂows backwards from t = 1 to t = 0 .

Original: In practice, to solve Eq.
Revised: In practice, to solve Eq.

Original: (14), we would discretize the SDE (i.e., apply Euler-Maruyama, and approximate the function derivatives with ﬁnite differences).
Revised: (14), we would discretize the SDE (i.e., apply Euler-Maruyama, and approximate the function derivatives with ﬁnite differences).

Original: We use step size ∆ t > 0 ,
Revised: We use step size ∆ t > 0 ,

Original: [Equation]
Revised: [Equation]

Original: where η ∼ N ( 0 , I )
Revised: where η ∼ N ( 0 , I )

Original: So far we assumed that the dataset has one single image α while α is actually sampled from a distribution, q 0 .
Revised: So far we assumed that the dataset has one single image α while α is actually sampled from a distribution, q 0 .

Original: For each α ∼ q 0 , there is a different SDE describing the corruption process.
Revised: For each α ∼ q 0 , there is a different SDE describing the corruption process.

Original: The key idea is that at each timestep t , we use the trained network to predict C t α
Revised: The key idea is that at each timestep t , we use the trained network to predict C t α

Original: and then we solve the associated reverse SDE.
Revised: and then we solve the associated reverse SDE.

Original: Remember from Eq.
Revised: Remember from Eq.

Original: (9) that the network
Revised: (9) that the network

Original: h θ was trained such that h θ ( x t | t )
Revised: h θ was trained such that h θ ( x t | t )

Original: ≈ C t x 0 .
Revised: ≈ C t x 0 .

Original: Hence, we can use our trained network to estimate at each time t the ﬁltered version, C t α , of the image α that it is to be generated.
Revised: Hence, we can use our trained network to estimate at each time t the ﬁltered version, C t α , of the image α that it is to be generated.

Original: For the estimation of ∇ x t log q t ( x t ) we also use our model that provably learns the score according to Theorem 3.1.
Revised: For the estimation of ∇ x t log q t ( x t ) we also use our model that provably learns the score according to Theorem 3.1.

Original: Putting everything together:
Revised: Putting everything together:

Original: [Equation]
Revised: [Equation]

Original: Our sampler is summarized in Algorithm 2.
Revised: Our sampler is summarized in Algorithm 2.

Original: It is interesting to understand what this sampler is doing.
Revised: It is interesting to understand what this sampler is doing.

Original: Essentially, there are two updates: one for deblurring and one for denoising.
Revised: Essentially, there are two updates: one for deblurring and one for denoising.

Original: At the core of this update equation, is the prediction of the clean image, ˆ x 0 .
Revised: At the core of this update equation, is the prediction of the clean image, ˆ x 0 .

Original: Once the clean image is predicted, we blur it back to two different corruption levels, t and t − ∆ t .
Revised: Once the clean image is predicted, we blur it back to two different corruption levels, t and t − ∆ t .

Original: The deblurring gradient is the residual between the blurred image at level t − ∆ t and the blurred image at level t .
Revised: The deblurring gradient is the residual between the blurred image at level t − ∆ t and the blurred image at level t .

Original: Interestingly, the denoising update is the same as the one used in typical score-based models (that only use additive noise).
Revised: Interestingly, the denoising update is the same as the one used in typical score-based models (that only use additive noise).

Original: In fact, if there is no blur ( C t = I ), our sampler becomes exactly the sampler used for the Variance Exploding (VE) SDE in Song et al.
Revised: In fact, if there is no blur ( C t = I ), our sampler becomes exactly the sampler used for the Variance Exploding (VE) SDE in Song et al.

Original: (2021b).
Revised: (2021b).

Original: Require: p 1 , φ θ , C t , σ t , ∆ t x 1 ∼ p x ) for
Revised: Require: p 1 , φ θ , C t , σ t , ∆ t x 1 ∼ p x ) for

Original: = 1 to with step − ∆ t do ˆ x 0 =
Revised: = 1 to with step − ∆ t do ˆ x 0 =

Original: φ θ ( x t | t ) + x t
Revised: φ θ ( x t | t ) + x t

Original: ˆ y t ← C t ˆ x at
Revised: ˆ y t ← C t ˆ x at

Original: η t ∼ N ( 0 , I ) ˆ (cid:15) t ← ˆ y t − x t z t − ∆ t ← x t − ( σ 2 t − ∆ t − σ 2 t ) σ 2 t ˆ (cid:15) t + (cid:113) σ 2 t − σ 2 t − ∆ t η t
Revised: η t ∼ N ( 0 , I ) ˆ (cid:15) t ← ˆ y t − x t z t − ∆ t ← x t − ( σ 2 t − ∆ t − σ 2 t ) σ 2 t ˆ (cid:15) t + (cid:113) σ 2 t − σ 2 t − ∆ t η t

Original: −
Revised: −

Original: ˆ y t − ∆ t ← C t − ∆ t ˆ x
Revised: ˆ y t − ∆ t ← C t − ∆ t ˆ x

Original: − x t − ∆ t ← z t − ∆ t + (ˆ y t − ∆ t − ˆ y t )
Revised: − x t − ∆ t ← z t − ∆ t + (ˆ y t − ∆ t − ˆ y t )

Original: We call our sampler Momentum Sampler because we can think of it as a generalization of the update of the Naive Sampler, where there is a momentum term.
Revised: We call our sampler Momentum Sampler because we can think of it as a generalization of the update of the Naive Sampler, where there is a momentum term.

Original: To understand this better, we look at the setting where there is no noise.
Revised: To understand this better, we look at the setting where there is no noise.

Original: Then, the update rule of the Momentum Sampler is:
Revised: Then, the update rule of the Momentum Sampler is:

Original: [Equation]
Revised: [Equation]

Original: As seen, the ﬁrst term is what the Naive Sampler would use to update the image at level t and the second term is what the Naive Sampler would use to update the image at level t − ∆ t .
Revised: As seen, the ﬁrst term is what the Naive Sampler would use to update the image at level t and the second term is what the Naive Sampler would use to update the image at level t − ∆ t .

Original: If these two directions are aligned, then the gradient ∆ x t is small.
Revised: If these two directions are aligned, then the gradient ∆ x t is small.

Original: Hence, there is a notion of momentum, analogous to how the term is used in classical optimization.
Revised: Hence, there is a notion of momentum, analogous to how the term is used in classical optimization.

Original: Probability Flow Momentum Sampler.
Revised: Probability Flow Momentum Sampler.

Original: The update-rule of our Momentum Sampler was derived by looking at the ﬁrst-order discretization of the backward SDE associated with our corruption, given in Eq.
Revised: The update-rule of our Momentum Sampler was derived by looking at the ﬁrst-order discretization of the backward SDE associated with our corruption, given in Eq.

Original: Similarly to Song et al.
Revised: Similarly to Song et al.

Original: (2021b), we can also consider the Ordinary Differential
Revised: (2021b), we can also consider the Ordinary Differential

Original: Equation (ODE) associated with this SDE:
Revised: Equation (ODE) associated with this SDE:

Original: [Equation]
Revised: [Equation]

Original: Using the same derivations as we did before, we can approximate ˙
Revised: Using the same derivations as we did before, we can approximate ˙

Original: C t α , ∇ x t log q t ( x t ) with our trained neural network and get a deterministic version of the Momentum Sampler.
Revised: C t α , ∇ x t log q t ( x t ) with our trained neural network and get a deterministic version of the Momentum Sampler.

Original: We name this sampler Probability Flow Momentum Sampler, following the naming convention of Song et al.
Revised: We name this sampler Probability Flow Momentum Sampler, following the naming convention of Song et al.

Original: (2021b).
Revised: (2021b).

Original: We detail our derivations in Section B of the Appendix.
Revised: We detail our derivations in Section B.1 of the Appendix.

Original: 3. S CHEDULING
Revised: 3. S CHEDULING

Original: The last piece of our framework is how to choose the corruption levels, i.e. the scheduling of the diffusion.
Revised: The last piece of our framework is how to choose the corruption levels, i.e. the scheduling of the diffusion.

Original: For example, for Gaussian Blur, the scheduling decides how much time is spent in the diffusion in the very blurry, somewhat blurry and almost no blurry regimes.
Revised: For example, for Gaussian Blur, the scheduling decides how much time is spent in the diffusion in the very blurry, somewhat blurry and almost no blurry regimes.

Original: We provide a principled way of choosing the corruption levels for arbitrary corruption processes.
Revised: We provide a principled way of choosing the corruption levels for arbitrary corruption processes.

Original: Let D 0 the distribution of real images and D 1 be a known distribution that we know how to sample from, e.g. a Normal Distribution.
Revised: Let D 0 the distribution of real images and D 1 be a known distribution that we know how to sample from, e.g. a Normal Distribution.

Original: In the design phase of score-based modeling, the goal is to choose a set of intermediate distributions, {D t } , that smoothly transform images from D 0 to samples from the distribution D 1 .
Revised: In the design phase of score-based modeling, the goal is to choose a set of intermediate distributions, {D t } , that smoothly transform images from D 0 to samples from the distribution D 1 .

Original: Let Θ = { θ 1 , θ 2 , ..., θ k , ... } be the space of diffusion parameters, i.e. each θ i corresponds to a distribution D θ i .
Revised: Let Θ = { θ 1 , θ 2 , ..., θ k , ... } be the space of diffusion parameters, i.e. each θ i corresponds to a distribution D θ i .

Original: In the case of blur for example, θ i controls how much we blur the image.
Revised: In the case of blur for example, θ i controls how much we blur the image.

Original: Let also M : X × X
Revised: Let also M : X × X

Original: → R be a metric that measures distances between distributions, e.g.
Revised: → R be a metric that measures distances between distributions, e.g.

Original: M might be the Wasserstein Distance of distributions with support X .
Revised: M might be the Wasserstein Distance of distributions with support X .

Original: We construct a weighted graph G (cid:15) with the nodes being the distributions and the weights given by:
Revised: We construct a weighted graph G (cid:15) with the nodes being the distributions and the weights given by:

Original: [Equation]
Revised: [Equation]

Original: For a ﬁxed (cid:15) , we choose the distributions that minimize the cost of the path between D 0 and D 1 .
Revised: For a ﬁxed (cid:15) , we choose the distributions that minimize the cost of the path between D 0 and D 1 .

Original: The parameter (cid:15) expresses the power of the best neural network we can train to reverse one step of the diffusion.
Revised: The parameter (cid:15) expresses the power of the best neural network we can train to reverse one step of the diffusion.

Original: If (cid:15) = ∞ , then for any metric M , the shortest path is to go directly from D 0 to D 1 .
Revised: If (cid:15) = ∞ , then for any metric M , the shortest path is to go directly from D 0 to D 1 .

Original: However, it is impossible to denoise in one step a super noisy image.
Revised: However, it is impossible to denoise in one step a super noisy image.

Original: Hence, we need to go through many intermediate steps, which is forced by setting (cid:15) to a smaller value.
Revised: Hence, we need to go through many intermediate steps, which is forced by setting (cid:15) to a smaller value.

Original: As we increase the number of the candidate distributions we are getting closer to ﬁnding the geodesic between D 0 and D 1 .
Revised: As we increase the number of the candidate distributions we are getting closer to ﬁnding the geodesic between D 0 and D 1 .

Original: However, the computational cost of the method increases since we need to estimate all the pairwise distances M
Revised: However, the computational cost of the method increases since we need to estimate all the pairwise distances M

Original: ( D θ i , D θ j ) .
Revised: ( D θ i , D θ j ) .

Original: In practice, we use a relatively small number of candidate distributions, e.g. T = 256 and once the path is found, we do linear interpolation to extend to the continuous case.
Revised: In practice, we use a relatively small number of candidate distributions, e.g. T = 256 and once the path is found, we do linear interpolation to extend to the continuous case.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our method in CelebA- 64 and CIFAR-10.
Revised: We evaluate our method in CelebA- 64 and CIFAR-10.

Original: We show that by just changing the corruption mechanism (and using our framework for scheduling, learning and sampling)
Revised: We show that by just changing the corruption mechanism (and using our framework for scheduling, learning and sampling)

Original: we signiﬁcantly improve the FID score and reduce the sampling time.
Revised: we signiﬁcantly improve the FID score and reduce the sampling time.

Original: We use the architecture and the training hyperparameters from Song et al.
Revised: We use the architecture and the training hyperparameters from Song et al.

Original: (2021b) (full details can be found in the Appendix).
Revised: (2021b) (full details can be found in the Appendix).

Original: For most of our experiments, we use Gaussian Blur as our primary corruption mechanism.
Revised: For most of our experiments, we use Gaussian Blur as our primary corruption mechanism.

Original: To illustrate that our method works more generally, we also show results with masking which is discussed separately later.
Revised: To illustrate that our method works more generally, we also show results with masking which is discussed separately later.

Original: Consistent with the description of our method, our deterministic corruptions are also paired with additive low magnitude noise.
Revised: Consistent with the description of our method, our deterministic corruptions are also paired with additive low magnitude noise.

Original: This is required by our theoretical results, otherwise the conditional log-likelihood, log q ( x t | x 0 ) would be undeﬁned.
Revised: This is required by our theoretical results, otherwise the conditional log-likelihood, log q ( x t | x 0 ) would be undeﬁned.

Original: We also ﬁnd the addition of noise beneﬁcial in practice (see Appendix F.1.1 for ablation studies on the role of noise).
Revised: We also ﬁnd the addition of noise beneﬁcial in practice (see Appendix F.1.1 for ablation studies on the role of noise).

Original: Scheduling.
Revised: Scheduling.

Original: We use the standard geometric scheduling for the noise (Song & Ermon, 2019; 2020;
Revised: We use the standard geometric scheduling for the noise (Song & Ermon, 2019; 2020;

Original: Song et al., 2021b)
Revised: Song et al., 2021b)

Original: and use the methodology described in Section 3.3 to select the blur levels.
Revised: and use the methodology described in Section 3.3 to select the blur levels.

Original: We use the Wasserstein distance as the metric M to measure how close are the different distributions.
Revised: We use the Wasserstein distance as the metric M to measure how close are the different distributions.

Original: To clearly illustrate the switch to a different corruption, our diffusion has an initial stage that increases the noise (with no blur) and then we ﬁx the noise (to a small value, e.g. σ = 0 . 1 ) and change (using our scheduling framework) the amount of blur.
Revised: To clearly illustrate the switch to a different corruption, our diffusion has an initial stage that increases the noise (with no blur) and then we ﬁx the noise (to a small value, e.g. σ = 0 . 1 ) and change (using our scheduling framework) the amount of blur.

Original: Our diffusion spends less than 20% of the total time in the initial stage that increases the noise.
Revised: Our diffusion spends less than 20% of the total time in the initial stage that increases the noise.

Original: We ablate those choices in Section F.1 of the Appendix.
Revised: We ablate those choices in Section F.1 of the Appendix.

Original: One important property of our framework is that the scheduling is dataset speciﬁc.
Revised: One important property of our framework is that the scheduling is dataset speciﬁc.

Original: Intuitively, the way we corrupt should depend on the nature of the data we are modelling.
Revised: Intuitively, the way we corrupt should depend on the nature of the data we are modelling.

Original: The interested reader can ﬁnd the found schedulings for each dataset in Figure 6 of the Appendix.
Revised: The interested reader can ﬁnd the found schedulings for each dataset in Figure 6 of the Appendix.

Original: Results.
Revised: Results.

Original: We train our networks on CelebA- 64 and CIFAR-10 using the found schedulings and the training objective of Eq.
Revised: We train our networks on CelebA- 64 and CIFAR-10 using the found schedulings and the training objective of Eq.

Original: We start by showing uncurated samples of our models in Figure 2.
Revised: We start by showing uncurated samples of our models in Figure 2.

Original: The generated images have high diversity and ﬁdelity in both datasets.
Revised: The generated images have high diversity and ﬁdelity in both datasets.

Original: We compare the FID (Heusel et al., 2017) obtained by our method on CelebA with many natural baselines that use any of the VE,
Revised: We compare the FID (Heusel et al., 2017) obtained by our method on CelebA with many natural baselines that use any of the VE,

Original: VP or subVP SDEs.
Revised: VP or subVP SDEs.

Original: Speciﬁcally, we compare against DDPM (Ho et al., 2020) that uses the VP SDE,
Revised: Speciﬁcally, we compare against DDPM (Ho et al., 2020) that uses the VP SDE,

Original: DDIM (Song et al., 2021a) that uses the same model but with a different sampler, DDPM++ (Kim et al., 2022) that is the state-of-the-art model for VP-SDE and the NCSN++ models (Song et al., 2021b) trained with the VE and subVP-SDEs.
Revised: DDIM (Song et al., 2021a) that uses the same model but with a different sampler, DDPM++ (Kim et al., 2022) that is the state-of-the-art model for VP-SDE and the NCSN++ models (Song et al., 2021b) trained with the VE and subVP-SDEs.

Original: For a fair comparison, we only use reported numbers in published papers for the baselines and we do not rerun them ourselves.
Revised: For a fair comparison, we only use reported numbers in published papers for the baselines and we do not rerun them ourselves.

Original: Our model achieves state-of-the-art FID score, 1 . 85 , in CelebA, outperforming all the other methods.
Revised: Our model achieves state-of-the-art FID score, 1 . 85 , in CelebA, outperforming all the other methods.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: For CIFAR-10, we obtain FID score 3 .
Revised: For CIFAR-10, we obtain FID score 3 .

Original: 86 with our Probability Flow Momentum Sampler and 3 .
Revised: 86 with our Probability Flow Momentum Sampler and 3 .

Original: 91 with our Momentum Sampler.
Revised: 91 with our Momentum Sampler.

Original: This FID score is competitive, yet not state-of-the-art.
Revised: We summarize the results in Table 2.

Original: Speciﬁcally, we outperform the NCSN++ (VE SDE)baseline with ReverseDiffusion (similar to our SDE sampler – achieves FID 4 . 79 ) and with Probability Flow Ode sampling (theirs FID: 10 . 54 ).
Revised: Our best FID is competitive with similar samplers to similar methods, e.g. with NCSN++ (VE SDE) with Reverse SDE sampling.

Original: However, it is known (Karras et al., 2022) that  diffusion models with VP SDEs usually do better in CIFAR-10, e.g. DDPMs achieves FID 3 . 17 .
Revised: However, it is behind other state-of-the-art models such as LSGM (Vahdat et al., 2021) model and DDPM (Ho et al., 2020). We believe that this performance gap can be decreased in the future by further research in the area of diffusion models with general corruptions.

Original: Our method is superior in sampling time , for both CIFAR-10 and CelebA. Figure 3 shows how FID changes based on the Number of Function Evaluations (NFEs).
Revised: Our method is superior in sampling time , for both CIFAR-10 and CelebA. Figure 3 shows how FID changes based on the Number of Function Evaluations (NFEs).

Original: Our method requires signiﬁcant less steps to achieve the same or better quality than NCSN++ (VE SDE)
Revised: Our method requires signiﬁcant less steps to achieve the same or better quality than NCSN++ (VE SDE)

Original: (Song et al., 2021b), using the same architecture and training hyperparameters (FID values taken from (Ma et al., 2022)).
Revised: (Song et al., 2021b), using the same architecture and training hyperparameters (FID values taken from (Ma et al., 2022)).

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Ablation Study for Sampling.
Revised: Ablation Study for Sampling.

Original: For all the results we presented so far, we used the Momentum
Revised: For all the results we presented so far, we used the Momentum

Original: Sampler that we introduced in Section 3.2 and Algorithm 2.
Revised: Sampler that we introduced in Section 3.2 and Algorithm 2.

Original: In this section, we ablate the choice of the sampler.
Revised: In this section, we ablate the choice of the sampler.

Original: Speciﬁcally, we compare with the intuitive Naive Sampler described in Algorithm 1.
Revised: Speciﬁcally, we compare with the intuitive Naive Sampler described in Algorithm 1.

Original: We show that the choice of the sampler has a dramatic effect in the quality and the diversity of the generated images.
Revised: We show that the choice of the sampler has a dramatic effect in the quality and the diversity of the generated images.

Original: Results are shown in Figure 4.
Revised: Results are shown in Figure 4.

Original: The images from the Naive Sampler seem repetitive and lack details.
Revised: The images from the Naive Sampler seem repetitive and lack details.

Original: This is reﬂected in the poor FID score.
Revised: This is reﬂected in the poor FID score.

Original: The Momentum Sampler leads to images with greater variety and detail, dramatically improving FID from 27 .
Revised: The Momentum Sampler leads to images with greater variety and detail, dramatically improving FID from 27 .

Original: 82 to 1 .
Revised: 82 to 1 .

Original: [Figure]
Revised: [Figure]

Original: Training Objective.
Revised: Training Objective.

Original: Concurrent works (Hoogeboom & Salimans, 2022; Bansal et al., 2022) that also consider different corruption processes have used as their loss a simple objective: given an input image, they predict the residual to the clean.
Revised: Concurrent works (Hoogeboom & Salimans, 2022; Bansal et al., 2022) that also consider different corruption processes have used as their loss a simple objective: given an input image, they predict the residual to the clean.

Original: As explained in Section 3.1, this is equivalent to optimizing for an upper-bound of the Soft Score Matching objective (which is minimizing the error to the score-function, see Theorem 3.1).
Revised: As explained in Section 3.1, this is equivalent to optimizing for an upper-bound of the Soft Score Matching objective (which is minimizing the error to the score-function, see Theorem 3.1).

Original: We show that if we use our exact pipeline (model, scheduling, sampler, training hyperparameters and so on) and we replace our Soft Score matching
Revised: We show that if we use our exact pipeline (model, scheduling, sampler, training hyperparameters and so on) and we replace our Soft Score matching

Original: Loss with the loss introduced in Bansal et al.
Revised: Loss with the loss introduced in Bansal et al.

Original: (2022), FID score increases from 1 . 85 to 5 .
Revised: (2022), FID score increases from 1 . 85 to 5 .

Original: This experiment shows the effectiveness of the Soft Score Matching objective.
Revised: This experiment shows the effectiveness of the Soft Score Matching objective.

Original: Ablation Studies for Scheduling.
Revised: Ablation Studies for Scheduling.

Original: We perform extensive ablations on the scheduling to understand the role and importance of noise in the framework and whether our proposed scheme outperforms other natural baselines.
Revised: We perform extensive ablations on the scheduling to understand the role and importance of noise in the framework and whether our proposed scheme outperforms other natural baselines.

Original: The results are detailed in section F.1 of the Appendix.
Revised: The results are detailed in section F.1 of the Appendix.

Original: Our main ﬁndings are as follows: i) the Momentum Sampler works even if noise and blur are changing simultaneously (i.e. for schedulings with non-ﬁxed noise), ii) lowering the maximum value of noise leads to important performance degradation – for very small noise the method completely fails and iii) our found scheduling outperforms signiﬁcantly (baseline FID: 8 . 35 , ours: 1 . 85 ) a natural baseline that sets blur parameters such that MSE between the corrupted and the clean image decays in the same rate for
Revised: Our main ﬁndings are as follows: i) the Momentum Sampler works even if noise and blur are changing simultaneously (i.e. for schedulings with non-ﬁxed noise), ii) lowering the maximum value of noise leads to important performance degradation – for very small noise the method completely fails and iii) our found scheduling outperforms signiﬁcantly (baseline FID: 8 . 35 , ours: 1 . 85 ) a natural baseline that sets blur parameters such that MSE between the corrupted and the clean image decays in the same rate for

Original: Gaussian Denoising and Soft Diffusion.
Revised: Gaussian Denoising and Soft Diffusion.

Original: Masking Diffusion Models.
Revised: Masking Diffusion Models.

Original: To show the generality of our framework, we also train models with (discrete) masking diffusion paired with noise.
Revised: To show the generality of our framework, we also train models with (discrete) masking diffusion paired with noise.

Original: Figure 1 (top 2 rows) shows the forward and the (learned) reverse process for blur on the left and masking on the right.
Revised: Figure 1 (top 2 rows) shows the forward and the (learned) reverse process for blur on the left and masking on the right.

Original: We train the model with our Soft Score Matching objective on CelebA. Unconditional samples from the model trained with masking can be found in Figure 12 of the Appendix.
Revised: We train the model with our Soft Score Matching objective on CelebA. Unconditional samples from the model trained with masking can be found in Figure 12 of the Appendix.

Original: In Figure 5, we show the predictions of our two trained models (blur and masking) for the conditional mean, E
Revised: In Figure 5, we show the predictions of our two trained models (blur and masking) for the conditional mean, E

Original: [ x 0 | x t ] , at different times of the diffusion.
Revised: [ x 0 | x t ] , at different times of the diffusion.

Original: Soft Score Matching trains the model to make a prediction that matches the real images in the ﬁltered space.
Revised: Soft Score Matching trains the model to make a prediction that matches the real images in the ﬁltered space.

Original: Hence, given masked images the masking model is incentived to predict right only the observed noisy region.
Revised: Hence, given masked images the masking model is incentived to predict right only the observed noisy region.

Original: As diffusion time t becomes smaller (cleaner images), the observed region grows and the model predicts bigger windows.
Revised: As diffusion time t becomes smaller (cleaner images), the observed region grows and the model predicts bigger windows.

Original: Although it is interesting that we can train Masking Diffusion models, there are several limitations compared to Blur Diffusion (and even Gaussian Diffusion).
Revised: Although it is interesting that we can train Masking Diffusion models, there are several limitations compared to Blur Diffusion (and even Gaussian Diffusion).

Original: We observe that these models are very slow to sample from: withsampling steps, FID is 30 .
Revised: We observe that these models are very slow to sample from: withsampling steps, FID is 30 .

Original: 92 while with 4096 , FID improves to 12 . 37 .
Revised: 92 while with 4096 , FID improves to 12 . 37 .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We showed that by just changing the corruption mechanism, we observe important computational beneﬁts.
Revised: We showed that by just changing the corruption mechanism, we observe important computational beneﬁts.

Original: Reducing the number of function evaluations for sample generation with diffusion models is an active area of research.
Revised: Reducing the number of function evaluations for sample generation with diffusion models is an active area of research.

Original: Jolicoeur-Martineau et al.
Revised: Jolicoeur-Martineau et al.

Original: Liu et al.
Revised: Liu et al.

Original: ; Song et al.
Revised: ; Song et al.

Original: (2021a) propose more efﬁcient samplers for diffusion models.
Revised: (2021a) propose more efﬁcient samplers for diffusion models.

Original: Rombach et al.
Revised: Rombach et al.

Original: ; Daras et al.
Revised: ; Daras et al.

Original: (2022) train diffusion models on low-dimensional latent spaces.
Revised: (2022) train diffusion models on low-dimensional latent spaces.

Original: Xiao et al.
Revised: Xiao et al.

Original: (2022) combine diffusion models with Generative Adversarial Networks (GANs) to allow for bigger denoising steps.
Revised: (2022) combine diffusion models with Generative Adversarial Networks (GANs) to allow for bigger denoising steps.

Original: Ryu & Ye (2022); Ho et al.
Revised: Ryu & Ye (2022); Ho et al.

Original: (2022a) generate high-resolution images progressively, from coarse to ﬁne quality.
Revised: (2022a) generate high-resolution images progressively, from coarse to ﬁne quality.

Original: Salimans & Ho (2022) train progressively a student network that mimics the teacher diffusion model with fewer sampling steps.
Revised: Salimans & Ho (2022) train progressively a student network that mimics the teacher diffusion model with fewer sampling steps.

Original: We note that all these works are orthogonal to ours and therefore can be used in combination with our framework for even faster sampling.
Revised: We note that all these works are orthogonal to ours and therefore can be used in combination with our framework for even faster sampling.

Original: On scheduling, there is closely related work by Bao et al.
Revised: On scheduling, there is closely related work by Bao et al.

Original: The authors ﬁnd a closed form solution (w.r.t.
Revised: The authors ﬁnd a closed form solution (w.r.t.

Original: the score function) for the variance of the reverse SDE for Gaussian Diffusion.
Revised: the score function) for the variance of the reverse SDE for Gaussian Diffusion.

Original: Then, they select a noise scheduling that minimizes the KL along the path from the initial to the ﬁnal distribution.
Revised: Then, they select a noise scheduling that minimizes the KL along the path from the initial to the ﬁnal distribution.

Original: In our work, we use Wasserstein distances and consider general corruption processes for which it is unclear whether such a closed form solution exists.
Revised: In our work, we use Wasserstein distances and consider general corruption processes for which it is unclear whether such a closed form solution exists.

Original: Instead, we estimate the distances in a data-driven way.
Revised: Instead, we estimate the distances in a data-driven way.

Original: This allows us to schedule arbitrary diffusion processes in a principled way.
Revised: This allows us to schedule arbitrary diffusion processes in a principled way.

Original: There is signiﬁcant recent (Anonymous, 2022) and concurrent work (Rissanen et al., 2022; Bansal et al., 2022; Hoogeboom & Salimans, 2022) that proposes diffusion with other degradations.
Revised: There is signiﬁcant recent (Anonymous, 2022) and concurrent work (Rissanen et al., 2022; Bansal et al., 2022; Hoogeboom & Salimans, 2022) that proposes diffusion with other degradations.

Original: These concurrent works have signiﬁcant differences since they use different loss functions and sampling mechanisms.
Revised: These concurrent works have signiﬁcant differences since they use different loss functions and sampling mechanisms.

Original: Soft Score Matching experimentally outperforms (under the exact same setting) the loss functions used in the concurrent works (ours FID: 1 . 85 , theirs: 5 . 91 ).
Revised: Soft Score Matching experimentally outperforms (under the exact same setting) the loss functions used in the concurrent works (ours FID: 1 . 85 , theirs: 5 . 91 ).

Original: Our (blur) models obtain state-of-the-art FID for CelebA (FID 1.85).
Revised: Our (blur) models obtain state-of-the-art FID for CelebA (FID 1.85).

Original: For CIFAR10, we outperform (FID: 3.86) Gaussian diffusion with Variance Exploding SDE (Song et al., 2021b).
Revised: For CIFAR10, we outperform (FID: 3.86) Gaussian diffusion with Variance Exploding SDE (Song et al., 2021b).

Original: Hoogeboom & Salimans (2022) also use blurring (but with Variance Preserving SDE) to further push the CIFAR-10 performance to FID 3.17.
Revised: Hoogeboom & Salimans (2022) also use blurring (but with Variance Preserving SDE) to further push the CIFAR-10 performance to FID 3.17.

Original: These advancements show that there are multiple diffusions with promising potential.
Revised: These advancements show that there are multiple diffusions with promising potential.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: We presented a framework to train and sample from diffusion models that reverse general corruption processes.
Revised: We presented a framework to train and sample from diffusion models that reverse general corruption processes.

Original: We showed that by changing the corruption process, we can get signiﬁcant sample quality improvements and computational beneﬁts.
Revised: We showed that by changing the corruption process, we can get signiﬁcant sample quality improvements and computational beneﬁts.

Original: This work opens several future research directions.
Revised: This work opens several future research directions.

Original: For example, it is possible to optimize or learn the corruption process for solving a speciﬁc type of inverse problem.
Revised: For example, it is possible to optimize or learn the corruption process for solving a speciﬁc type of inverse problem.

Original: It is also worth exploring if mixing different corruptions (blur, noise, masking, etc.)
Revised: It is also worth exploring if mixing different corruptions (blur, noise, masking, etc.)

Original: improves performance or sampling quality.
Revised: improves performance or sampling quality.

Original: Finally, it is important to understand the role of noise, from both a theoretical and practical standpoint.
Revised: Finally, it is important to understand the role of noise, from both a theoretical and practical standpoint.

Original: A A PPENDIX A. P ROOFS
Revised: A A PPENDIX A. P ROOFS

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Let q 0 , q t be two distributions in R n .
Revised: Let q 0 , q t be two distributions in R n .

Original: Assume that all the conditional distributions, q t ( x t | x 0 ) , are supported and differentiable in R n .
Revised: Assume that all the conditional distributions, q t ( x t | x 0 ) , are supported and differentiable in R n .

Original: Let:
Revised: Let:

Original: [Equation]
Revised: [Equation]

Original: Then, there is a universal constant C (that does not depend on θ ) such that: J 1 ( θ ) =
Revised: Then, there is a universal constant C (that does not depend on θ ) such that: J 1 ( θ ) =

Original: J 2 ( θ ) + C .
Revised: J 2 ( θ ) + C .

Original: The proof of this Theorem is following the calculations of Vincent (2011).
Revised: The proof of this Theorem is following the calculations of Vincent (2011).

Revised: We observe that as long as the technical conditions listed as satisﬁed, the proof holds independent of the corruption type.

Revised: We provide the proof below for the shake of completeness.

Original: Proof of Theorem 3.1.
Revised: Proof of Theorem 3.1.

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: It sufﬁces to show that:
Revised: It sufﬁces to show that:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: B  P ROBABILITY F LOW ODE
Revised: B D ETERMINISTIC S AMPLERS B. P ROBABILITY F LOW ODE

Original: In the main text
Revised: In the main text

Original: [Equation]
Revised: [Equation]

Original: we derived our Momentum Sampler by analyzing the Backward SDE associated with our corruption process.
Revised: we derived our Momentum Sampler by analyzing the Backward SDE associated with our corruption process.

Original: Inspired by the works of Song et al.
Revised: Inspired by the works of Song et al.

Original: (2021b)
Revised: (2021b)

Original: ; Maoutsa et al.
Revised: ; Maoutsa et al.

Original: , we also consider deterministic sampling that is derived by looking at the ODE that describes our diffusion.
Revised: , we also consider deterministic sampling that is derived by looking at the ODE that describes our diffusion.

Original: Particularly, the ODE:
Revised: Particularly, the ODE:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  has the same marginal distributions (Anderson, 1982;
Revised: (37) has the same marginal distributions (Anderson, 1982;

Original: Maoutsa et al., 2020; Song et al., 2021b; Chen et al., 2018) with the Backward SDE:
Revised: Maoutsa et al., 2020; Song et al., 2021b; Chen et al., 2018) with the Backward SDE:

Original: [Equation]
Revised: [Equation]

Original: For our case, Eq.
Revised: For our case [Equation] Eq.

Original: (37) becomes:
Revised: (37) becomes:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst-order discretization of this ODE is given below:
Revised: The ﬁrst-order discretization of this ODE is given below:

Original: [Equation]
Revised: [Equation]

Original: We estimate C t α and ∇ x t log q t ( x t ) with our neural network and we get the Neural ODE (Chen et al., 2018):
Revised: We estimate C t α and ∇ x t log q t ( x t ) with our neural network and we get the Neural ODE (Chen et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: which is the update rule of our Probability Flow ODE Momentum Sampler.
Revised: which is the update rule of our Probability Flow ODE Momentum Sampler.

Original: We note that our simple discretization is not the only way to solve the ODE of Eq.
Revised: We note that our simple discretization is not the only way to solve the ODE of Eq.

Original: We can use more sophisticated, e.g. see Dormand & Prince (1980). 
Revised: We can use more sophisticated, e.g. see Dormand & Prince (1980). B. DDIM

Revised: A popular sampling scheme is the DDIM method, introduced in Song et al.

Revised: (2021a).

Original: C S CHEDULINGS
Revised: The idea is to use the network to predict ˆ x 0 from x t and then use the forward model to move from ˆ x 0 to x t − ∆ t . This is same with the Naive Sampler we introduced in the main paper but the difference in DDIM is that part of the stochasticity is replaced by “simulated” noise. The main trick to simulate the noise is to observe that once we know ˆ x 0 and x t , we can once again use the forward model to estimate the noise. We can use the same idea for Soft Diffusion sampling.

Revised: Speciﬁcally, we propose the following

Revised: DDIM-type sampler:

Revised: [Equation]

Revised: k 2 (cid:15).

Revised: This Equation is very similar to Equation 12, page 5 in the DDIM (Song et al., 2021a) paper.

Revised: The parameter k controls the amount of stochasticity, i.e. how much of the noise is simulated.

Revised: For k = 0 , we get a deterministic sampler.

Revised: Experiments with the deterministic DDIM-type sampler can be found in Section F.3.

Revised: C S CHEDULINGS

Original: We use our framework to select the blur levels in an unsupervised way.
Revised: We use our framework to select the blur levels in an unsupervised way.

Original: For the estimation of Wasserstein distances, we use the ott-jax (Cuturi et al., 2022) software package.
Revised: For the estimation of Wasserstein distances, we use the ott-jax (Cuturi et al., 2022) software package.

Original: We start withdifferent blur levels and we tune (cid:15) such that the shortest path contains 32 distributions.
Revised: We start withdifferent blur levels and we tune (cid:15) such that the shortest path contains 32 distributions.

Original: We then use linear interpolation to extend to the continuous case.
Revised: We then use linear interpolation to extend to the continuous case.

Original: Full experimental details can be found in the
Revised: Full experimental details can be found in the

Original: Appendix.
Revised: Appendix.

Original: These choices seem to work well in practice, but further optimization could be made in future work.
Revised: These choices seem to work well in practice, but further optimization could be made in future work.

Original: Figure 6 shows the found schedulings for the CelebA and the CIFAR-10 datasets.
Revised: Figure 6 shows the found schedulings for the CelebA and the CIFAR-10 datasets.

Original: Notice that the scheduling is slightly different between the two datasets – the diffusion depends on the nature of the data we are trying to model.
Revised: Notice that the scheduling is slightly different between the two datasets – the diffusion depends on the nature of the data we are trying to model.

Original: We underline that the parameters for the blur are selected without any supervision, by solving the optimization problem we deﬁned in Section 3.3.[Figure]
Revised: We underline that the parameters for the blur are selected without any supervision, by solving the optimization problem we deﬁned in Section 3.3.

Original: D T RAINING D ETAILS
Revised: D T RAINING D ETAILS

Original: Hyperparameters.
Revised: Hyperparameters.

Original: For our trainings, we use Adam optimizer with learning rate 2 e − 4 , β 1 = 0 .
Revised: For our trainings, we use Adam optimizer with learning rate 2 e − 4 , β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , (cid:15) = 1 e − 8 .
Revised: 999 , (cid:15) = 1 e − 8 .

Original: We additionally use gradient clipping for gradient norms bigger than 1 .
Revised: We additionally use gradient clipping for gradient norms bigger than 1 .

Original: For the learning rate scheduling, we use 5000 steps of linear warmup.
Revised: For the learning rate scheduling, we use 5000 steps of linear warmup.

Original: We use batch size 

Revised: We use batch size

Original: and we train for 1 − 2 M iterations (based on observed FID performance).
Revised: and we train for 1 − 2 M iterations (based on observed FID performance).

Original: Blur parameters.
Revised: Blur parameters.

Original: For the blurring operator, we use Gaussian blur with ﬁxed kernel size and we vary the variance of the kernel.
Revised: For the blurring operator, we use Gaussian blur with ﬁxed kernel size and we vary the variance of the kernel.

Original: For CelebA- 64 , we keep the kernel half size ﬁxed to 80 and we vary the standard deviation from 0 .
Revised: For CelebA- 64 , we keep the kernel half size ﬁxed to 80 and we vary the standard deviation from 0 .

Original: 01 to 23 .
Revised: 01 to 23 .

Original: For CIFAR-10, we keep the kernel half size ﬁxed to 32 and we vary the standard deviation from 0 .
Revised: For CIFAR-10, we keep the kernel half size ﬁxed to 32 and we vary the standard deviation from 0 .

Original: 01 to 18 .
Revised: 01 to 18 .

Original: For both datasets, we implement blur with zero-padding.
Revised: For both datasets, we implement blur with zero-padding.

Original: We chose the ﬁnal blur level such that the ﬁnal distribution is easy to sample from.
Revised: We chose the ﬁnal blur level such that the ﬁnal distribution is easy to sample from.

Original: In both cases, the ﬁnal distribution becomes noise on top of (almost) a single color. 
Revised: In both cases, the ﬁnal distribution becomes noise on top of (almost) a single color. [Figure]

Original: Architecture.
Revised: Architecture.

Original: We use the architecture of Song et al.
Revised: We use the architecture of Song et al.

Original: (2021b) without any changes.
Revised: (2021b) without any changes.

Original: Training Objective For all our experiments, we scale the loss at level t with w ( t )
Revised: Training Objective For all our experiments, we scale the loss at level t with w ( t )

Original: = 

Revised: =

Original: /σ 2 t as in
Revised: /σ 2 t as in

Original: Song & Ermon (2019; 2020); Song et al.
Revised: Song & Ermon (2019; 2020); Song et al.

Original: (2021b).
Revised: (2021b).

Original: Compute and Training Time We train our models on 16 v2-TPUs.
Revised: Compute and Training Time We train our models on 16 v2-TPUs.

Original: Our blur models on CelebA had an average speed of 6 iterations per second.
Revised: Our blur models on CelebA had an average speed of 6 iterations per second.

Original: For CIFAR-10, the average speed was 11 iterations per second
Revised: For CIFAR-10, the average speed was 11 iterations per second

Original: We note that there is no overhead over the NCSN++ paper other than projecting to the measurements space, which can be done very efﬁciently for both blur and masking.
Revised: We note that there is no overhead over the NCSN++ paper other than projecting to the measurements space, which can be done very efﬁciently for both blur and masking.

Original: Evaluation We keep one checkpoint every 10000 steps and we keep the best model among the kept checkpoints based on the obtained FID score.
Revised: Evaluation We keep one checkpoint every 10000 steps and we keep the best model among the kept checkpoints based on the obtained FID score.

Original: We use 50000 samples to evaluate the FID, as it is typically done in prior work.
Revised: We use 50000 samples to evaluate the FID, as it is typically done in prior work.

Original: E L IMITATIONS AND THINGS THAT DID NOT WORK
Revised: E L IMITATIONS AND THINGS THAT DID NOT WORK

Original: Our method has several limitations.
Revised: Our method has several limitations.

Original: First, it requires the diffusion operator to be known.
Revised: First, it requires the diffusion operator to be known.

Original: This is not always the case, e.g. see Peebles et al.
Revised: This is not always the case, e.g. see Peebles et al.

Original: (2022) where diffusion is applied to checkpoints of different models.
Revised: (2022) where diffusion is applied to checkpoints of different models.

Original: Another limitation is that our framework does not offer any guidance on which diffusion operators are actually more or less useful for learning the data distribution.
Revised: Another limitation is that our framework does not offer any guidance on which diffusion operators are actually more or less useful for learning the data distribution.

Original: Particularly, we already showed that blurring is a much more powerful diffusion mechanism than masking, in the sense that it leads to better FID scores and faster generations.
Revised: Particularly, we already showed that blurring is a much more powerful diffusion mechanism than masking, in the sense that it leads to better FID scores and faster generations.

Original: It is yet to be seen whether blurring is going to be outperformed by some other corruption method.
Revised: It is yet to be seen whether blurring is going to be outperformed by some other corruption method.

Original: Our method also only concerns linear diffusions (however, the extension to non-linear is relatively straightforward).
Revised: Our method also only concerns linear diffusions (however, the extension to non-linear is relatively straightforward).

Original: On the theoretical side, our method has also some shortcomings.
Revised: On the theoretical side, our method has also some shortcomings.

Original: First, it only intuitively explains why the reparametrization to the Denoising Score Matching is needed.
Revised: First, it only intuitively explains why the reparametrization to the Denoising Score Matching is needed.

Original: Second, since our method is based on the Denoising Score Matching, it only works when the conditional log-likelihood is deﬁned everywhere.
Revised: Second, since our method is based on the Denoising Score Matching, it only works when the conditional log-likelihood is deﬁned everywhere.

Original: There are distributions for which such condition is not satisﬁed, but still, can be learned (to some extent) with heuristic methods (Bansal et al., 2022).
Revised: There are distributions for which such condition is not satisﬁed, but still, can be learned (to some extent) with heuristic methods (Bansal et al., 2022).

Original: On the practical side, we believe that our objective, Soft Score Matching, sometimes leads to slower sampling compared to the simpler objective of predicting the clean image.
Revised: On the practical side, we believe that our objective, Soft Score Matching, sometimes leads to slower sampling compared to the simpler objective of predicting the clean image.

Original: For example, for masking, since the model is only penalized in the observed region, there is no incentive in expanding this region.
Revised: For example, for masking, since the model is only penalized in the observed region, there is no incentive in expanding this region.

Original: Hence, to achieve smooth transition between different masking levels we need to run many steps.
Revised: Hence, to achieve smooth transition between different masking levels we need to run many steps.

Original: Experimentally, we tried using our framework with even less stochasticity, but it did not work, e.g. see 7.
Revised: Experimentally, we tried using our framework with even less stochasticity, but it did not work, e.g. see 7.

Original: It would be interesting to understand better what is causing the failure and also what is the proper amount of randomness required at each diffusion step. 
Revised: It would be interesting to understand better what is causing the failure and also what is the proper amount of randomness required at each diffusion step. [Figure]

Original: F A DDITIONAL R ESULTS F. S CHEDULING A BLATIONS F.1. A BLATION S TUDIES FOR NOISE
Revised: F A DDITIONAL R ESULTS F. S CHEDULING A BLATIONS F.1. A BLATION S TUDIES FOR NOISE

Original: In the experiments of the main paper, our diffusion involves an initial stage where only noise is added.
Revised: In the experiments of the main paper, our diffusion involves an initial stage where only noise is added.

Original: Then noise is ﬁxed and the images are getting corrupted by the deterministic operator (e.g. blur or masking).
Revised: Then noise is ﬁxed and the images are getting corrupted by the deterministic operator (e.g. blur or masking).

Original: In this section, we show two ablations regarding the noise.
Revised: In this section, we show two ablations regarding the noise.

Original: Magnitude of noise.
Revised: Magnitude of noise.

Original: In this ablation study, we still keep the noise ﬁxed for a signiﬁcant part of the diffusion, but we ablate the magnitude of the noise.
Revised: In this ablation study, we still keep the noise ﬁxed for a signiﬁcant part of the diffusion, but we ablate the magnitude of the noise.

Original: Speciﬁcally, we attempt to study to what extent noise is needed in order to learn to reverse corruption processes with Soft Score Matching.
Revised: Speciﬁcally, we attempt to study to what extent noise is needed in order to learn to reverse corruption processes with Soft Score Matching.

Original: We train two additional models on CelebA-64 where we attempt to decrease the maximum noise to a lower value.
Revised: We train two additional models on CelebA-64 where we attempt to decrease the maximum noise to a lower value.

Original: The corruptions for both models involves an initial stage where noise grows geometrically rate from the initial value ( 0 . 01 ) to the maximum value.
Revised: The corruptions for both models involves an initial stage where noise grows geometrically rate from the initial value ( 0 . 01 ) to the maximum value.

Original: Then, all the models keep the noise ﬁxed at their maximum value for the rest of the diffusion.
Revised: Then, all the models keep the noise ﬁxed at their maximum value for the rest of the diffusion.

Original: We use the following maximum values: i) σ max = 0 .(model used in the paper), ii) σ max = 0 . 05 , and iii) σ max = 0 . 025 .
Revised: We use the following maximum values: i) σ max = 0 .(model used in the paper), ii) σ max = 0 . 05 , and iii) σ max = 0 . 025 .

Original: Unconditional samples from the two ablation models are shown in Figure 7.
Revised: Unconditional samples from the two ablation models are shown in Figure 7.

Original: As shown, both models fail to produce realistic samples.
Revised: As shown, both models fail to produce realistic samples.

Original: The quality of samples deteriorates signiﬁcantly as the noise decreases – the samples from[Figure] the ablation models are signiﬁcantly worse than the ones produced by our state-of-the-art model (see
Revised: The quality of samples deteriorates signiﬁcantly as the noise decreases – the samples from the ablation models are signiﬁcantly worse than the ones produced by our state-of-the-art model (see

Original: Figure 2 (right)).
Revised: Figure 2 (right)).

Original: We want to underline that this is not a conclusive study.
Revised: We want to underline that this is not a conclusive study.

Original: It might be the case that with different hyperparameters one can make Soft Score Matching work with lower values of noise.
Revised: It might be the case that with different hyperparameters one can make Soft Score Matching work with lower values of noise.

Original: For example, we might need to tune the weights w ( t ) since for the ablations (and the state-of-the-art model), we use w ( t )
Revised: For example, we might need to tune the weights w ( t ) since for the ablations (and the state-of-the-art model), we use w ( t )

Original: =
Revised: =

Original: /σ 2 t (as in Song & Ermon (2019; 2020);
Revised: /σ 2 t (as in Song & Ermon (2019; 2020);

Original: Song et al.
Revised: Song et al.

Original: (2021b)) which might be causing instabilities for low values of noise (Nichol & Dhariwal, 2021).
Revised: (2021b)) which might be causing instabilities for low values of noise (Nichol & Dhariwal, 2021).

Original: Noise changing throughout the diffusion.
Revised: Noise changing throughout the diffusion.

Original: We also train a model where noise and blur are changing simultaneously throughout the diffusion.
Revised: We also train a model where noise and blur are changing simultaneously throughout the diffusion.

Original: This is a sanity check to verify that our framework (learning and sampling) still works when the model needs to deblur and denoise at the same time.
Revised: This is a sanity check to verify that our framework (learning and sampling) still works when the model needs to deblur and denoise at the same time.

Original: For the noise scheduling, we simply use geometric scheduling from 0 .
Revised: For the noise scheduling, we simply use geometric scheduling from 0 .

Original: 1 to 0 .
Revised: 1 to 0 .

Original: We keep the blur parameters the same with the state-of-the-art model, i.e. we use the blur parameters shown in Figure 6.
Revised: We keep the blur parameters the same with the state-of-the-art model, i.e. we use the blur parameters shown in Figure 6.

Original: This model achieves a competitive FID score, 4 . 31 .
Revised: This model achieves a competitive FID score, 4 . 31 .

Original: This score could probably be further improved (by jointly selecting the blur and the noise scheduling with our framework), but this is beyond the scope of this ablation.
Revised: This score could probably be further improved (by jointly selecting the blur and the noise scheduling with our framework), but this is beyond the scope of this ablation.

Original: F.1. A BLATION S TUDY FOR B LUR
Revised: F.1. A BLATION S TUDY FOR B LUR

Original: For all our experiments so far, we chose the blur corruption levels based on the scheduling method we described in Section 3.3.
Revised: For all our experiments so far, we chose the blur corruption levels based on the scheduling method we described in Section 3.3.

Original: We show the beneﬁts of our approach by comparing to a natural baseline for selecting the diffusion levels.
Revised: We show the beneﬁts of our approach by comparing to a natural baseline for selecting the diffusion levels.

Original: For this natural baseline, we use the scheduling of Variance
Revised: For this natural baseline, we use the scheduling of Variance

Original: Exploding (VE) as guidance.
Revised: Exploding (VE) as guidance.

Original: Speciﬁcally, we choose the blur parameters such that the MSE between the corrupted image and the clean image decays with the same rate for the Gaussian Denoising Diffusion and our Soft Diffusion (blur and low magnitude noise).
Revised: Speciﬁcally, we choose the blur parameters such that the MSE between the corrupted image and the clean image decays with the same rate for the Gaussian Denoising Diffusion and our Soft Diffusion (blur and low magnitude noise).

Original: Formally, let { q (cid:48) t } 1 t =

Revised: Formally, let { q (cid:48) t } 1 t =

Original: be the (noisy) distributions used in Song et al.
Revised: be the (noisy) distributions used in Song et al.

Original: (2021b)
Revised: (2021b)

Original: for the Variance Exploding (VE) SDE and let { q t } 1 t =

Revised: for the Variance Exploding (VE) SDE and let { q t } 1 t =

Original: the blurry (and noisy) distributions we want to select.
Revised: the blurry (and noisy) distributions we want to select.

Original: At level t , we choose the blur parameters such that:
Revised: At level t , we choose the blur parameters such that:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We retrain on CelebA using this natural baseline method for selecting the diffusion parameters.
Revised: We retrain on CelebA using this natural baseline method for selecting the diffusion parameters.

Original: For a fair comparison, we keep the architecture and all the hyperparameters the same and we only ablate the scheduling of the blur.
Revised: For a fair comparison, we keep the architecture and all the hyperparameters the same and we only ablate the scheduling of the blur.

Original: We measure FID for both the trained model with the baseline scheduling and we observe it increases from 1 . 85 to 8 .
Revised: We measure FID for both the trained model with the baseline scheduling and we observe it increases from 1 . 85 to 8 .

Original: Apart from this large deterioration in performance, the baseline model obtains its best FID score after 2000 steps, while with our scheduling we only need 280 steps to obtain the best FID.
Revised: Apart from this large deterioration in performance, the baseline model obtains its best FID score after 2000 steps, while with our scheduling we only need 280 steps to obtain the best FID.

Original: This experiment shows that the choice of scheduling is really important for the model performance but for also the computational requirements of the sampling.
Revised: This experiment shows that the choice of scheduling is really important for the model performance but for also the computational requirements of the sampling.

Original: F. CIFAR-10 S PEED P LOT F. N EAREST S AMPLES IN T RAINING D ATA
Revised: F. CIFAR-10: A DDITIONAL R ESULTS

Revised: We report how FID is changing based on the Number of Function Evaluations for CIFAR-10 with Momentum Sampling in Figure 8.

Original: To verify that our model does not simply memorize the training dataset, we present generated images from our model and their nearest neighbor (L2 pixel distance) from the dataset.
Revised: The baseline model we are comparing against is NCSN++ trained with VE SDE, since our model also uses a VE SDE. For the baseline, we use the Reverse SDE sampler with Euler–Maruyama discretization (similar to our Momentum Sampler). The numbers are taken directly from Ma et al. [Figure]

Revised: how FID changes based on Number of Function Evaluations (NFEs) for CIFAR-10 for our (blur) model on CIFAR-10 with momentum sampling.

Revised: Our model offers signiﬁcant performance beneﬁts for low number of function evaluations.

Revised: We also report best FID performance for different samplers for ours and competing methods in Table 2.

Revised: Our best FID is competitive with similar samplers to similar methods, e.g. with NCSN++ trained with VE SDE and using the Reverse SDE sampler.

Revised: However, it is signiﬁcantly behind the state-ofthe-art LSGM (Vahdat et al., 2021) model.

Revised: We believe that this performance gap can be decreased in the future by further research in the area of diffusion models with general corruptions.

Revised: [Table]

Revised: F. S AMPLING A BLATIONS

Revised: We perform several ablations regarding the sampling method, additional to the results mentioned in the paper.

Revised: DDIM We experiment with the deterministic DDIM-type sampler and we report our results in the

Revised: Table 3 for CelebA-64.

Revised: Our DDIM-type sampler is very effective when the number of function evaluations is very low – it achieves FID 2 . 79 with only 25 steps while in comparison, the Momentum

Revised: Sampler, achieves FID 30 .

Revised: 31 with 50 steps.

Revised: The efﬁciency of DDIM is consistent with what has been observed in (Song et al., 2021a; Karras et al., 2022).

Revised: DDIM’s performance is also not getting worse as we increase the number of steps.

Revised: Instead, the performance of the Momentum Sampler, as observed in Figure 3 has a U-shape: there is an intermediate optimal number of steps to achieve the best FID score.

Revised: We believe this could be related to the fact that Momentum Sampler is not deterministic.

Revised: Speciﬁcally, in Karras et al.

Revised: (2022) it is observed that for deterministic samplers performance usually ﬂattens after a point (e.g. see Figure 2, page 4) while for some stochastic samplers performance goes again up after a certain number of function evaluations (e.g. see Figure 4, page 8).

Revised: [Table]

Revised: Probability Flow Momentum Sampler For CelebA- 64 , we reported results only for the (stochastic) Momentum Sampler in the main body of the paper (see Figure 3).

Revised: We report here results for the deterministic counterpart of this sampler, Probability Flow Momentum Sampler, summarized in Table 4.

Revised: The deterministic version of the Momentum Sampler seems to be performing slightly worse than the stochastic version – FID jumps from 1 . 85 to 2 .

Revised: However, this sampler, similar to the DDIM-type sampler, maintains more performance as we increase the number of steps.

Revised: This is an important advantage for the deterministic sampling methods since practitioners need to put less effort into tuning the NFEs to get the best result.

Revised: [Table]

Revised: Predictor-Corrector Samplers

Revised: Finally, we perform experiments with Predictor-Corrector samplers, as proposed in Song et al.

Revised: (2021b).

Revised: The idea is that we are alternating at each diffusion step between two different samplers.

Revised: We experiment with a DDIM Predictor and a Probability Flow Momentum Sampler corrector.

Original: The results are shown in Figure 9.
Revised: The results are summarized in Table 5. The Predictor-Corrector sampler maintains some of the beneﬁts of both samplers. Namely, for low number of function evaluations it has better performance than the Probability Flow Momentum Sampler (beneﬁt coming from the DDIM sampler) and for higher number of function evaluations performance is better than the DDIM sampler (beneﬁt coming from the Probability Flow Momentum Sampler).

Revised: That said, there is no spot in which the Predictor-Correct performs better than both DDIM and the Probability Flow Momentum Sampler.

Revised: We encourage future research in identifying better pairs of Predictor-Correctors that might outperform both the Predictor and the Corrector in some regime.

Original: [Figure]
Revised: [Table]

Revised: F. N EAREST S AMPLES IN T RAINING D ATA

Revised: To verify that our model does not simply memorize the training dataset, we present generated images from our model and their nearest neighbor (L2 pixel distance) from the dataset.

Revised: The results are shown in Figure 9.

Original: [Figure]
Revised: [Figure]

Original: F. U NCURATED SAMPLES
Revised: F. U NCURATED SAMPLES

Original: Figures 10 and 11 show more uncurated samples from our trained models on CelebA and CIFAR-10.
Revised: Figures 10 and 11 show more uncurated samples from our trained models on CelebA and CIFAR-