From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper challenges the current mainstream direction that for diffusion-based generative learning we need noise injection to corrupt the data. Instead, it proposes using a series of deterministic processes that corrupt the input data via operations such as blurring, downsampling, masking, etc. For such processes, the paper also presents a novel sampling mechanism that improves results by a large margin. Experimentally, proof of concept experiments are presented but in most cases, the obtained results are much worse than mainstream diffusion models even on small datasets.

On the positive side, this paper goes out of the current school of thought that we need noise injection for learning diffusion models and provides very preliminary results that one can train these models on image datasets. However, it falls short in two aspects: It does not provide a theoretical guarantee that one can obtain a well-defined generative model using deterministic processes (similar to how diffusion models such as DDPMs are derived).  In the absence of such theoretical discussion, one would expect strong empirical results showing that deterministic process can in fact obtain generative results as good as current diffusion models with noise-injected corruption. Unfortunately, in the current form, the paper does not successfully create a compelling case that noise injection is not needed in diffusion models. Given this, the paper does not appear to be ready for publication at ICLR.

On a minor note, as the authors observed in sec 5.2, it is not straightforward to define a generative model with a forward blurring process with no noise injection. In fact, the blurring process will map all the images with the same average pixel value to the same delta distribution. Reversing this process with no noise injection will simply generate the pixel-wise average image of all those images that resulted in the same delta distribution. As you observed, noise injection is in fact necessary for breaking symmetries in the de-blurring process, however, if the generative model is not trained with any noise injection, it may not know how to use that in the reverse process. 
justification_for_why_not_higher_score: All the reviewers agreed that the paper is not ready for acceptance at ICLR. 
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: 3 out of 4 reviewers attended the AC-reviewer meeting including 8WRE, h4aq, and BSRf. 

h4aq summarized the work as:
- This submission shows that we can potentially define deterministic processes in diffusion models (+)
- It obtains okay results while other previous works add noise (+)
- Results are not convincing, much worse than concurrent works like generative learning with inverse heat dissipation, and much worse than current diffusion models (-)
- Confused about theoretical analysis, assuming that the degradation operation is linear (-)
- Most other concerns are addressed to some degree during the rebuttal (+)

h4aq is still leaning towards reject.

8WRE summarized their thoughts as:
- This paper is a good proof of concept (+)
- The theoretical part showing that you can form a well-defined generative model using deterministic processes is missing (-)
- The reviewer is strongly concerned regarding the poor performance (-)
-  Motivation and validation on why deterministic processes can do well in generation is also missing (-) 

8WRE was originally leaning towards weak accept, but after the discussion, they also leaned towards reject.

BSRf  summarized their thoughts as:
- The topic of this paper is very interesting. They would consider it as a good analysis paper going out of the current thinking box (+)
- The paper is very easy to approach and read (+)
- The empirical results are not convincing at all (-)
- The prior is not easily defined as a standard Gaussian distribution and is data dependent (-)
- You cannot generate more images than images in the dataset with animorphosis (-)
- There were issues with Taylor expansion that was confusing at the beginning but they are resolved now (+)

BSRf saw value in this work in terms of thinking outside the current mainstream of diffusion models, however, they agreed with other reviewers that the empirical results are not creating a compelling case to use deterministic diffusion models in practice, and there is no theoretical justification that these models can form well-defined generative models. 

The final conclusion of the AC-reviewer meeting was that the paper is not ready for publication at ICLR which is in line with the recommendation of the absent reviewer.

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: Follow-up: 8WRE
comment: Dear reviewer 8WRE,\
Thank you again for your thoughtful review. We hope our response addresses your questions. \
We would be more than happy to address further questions.

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: Follow-up: h4aq
comment: Dear reviewer h4aq, \
Thank you again for your thoughtful review. We hope our response addresses your questions.
We would be more than happy to address further questions.

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: Thanks for your response
comment: We thank the reviewer for their response and for engaging in the discussion. \
Yes, in this paper we do not show that any of these transformations (like blur, masking, pixelation, animorphs, etc) have better results than the well-researched and highly engineered SOTA Gaussian diffusion models. We in fact show this clearly for celebA dataset in table 5 under the hot diffusion column. The point of our work is to challenge the intuition that noise is a **critical** component for learning the training data distribution and effective sampling. We believe our experiments effectively demonstrate that even entirely noise-free schemes (blur, inpainting, super-resolution) can still work. This will not only be of theoretical interest, but it will move the community to investigate other components that could improve diffusion such as model architecture or training setup.

From: ICLR.cc/2023/Conference/Paper5035/Reviewer_wT7w
title: Thanks for the reply
comment: Thanks for these detailed reply. It seems that samples in Figure 21 are smooth, which do not fully match the data distribution. So I still think currently the Gaussian noise is a preferable choice.

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: Author Response to wT7w - 2
comment: Thank you for your detailed reply. We'll provide further clarification and additional experimental results below:
It seems that Q6 and Q7 are missing.
>Q6 : It seems that TACoS sometimes performs worse than direct reconstruction. For example, in Figure 6, the samples of TACoS have more artifacts than the direct reconstruction. In Table 4, the TACoS quantative result is missing.
>
Yes, we agree with you that the TACoS does not perform as well for the snow operation as other operations. We think this is interesting and theorize that this is because the snow degradation is different from other degradations, as 
1) it is random and not deterministic degradation and 
2) the extreme case of this degradation is simply a white image and will thus produce same image if used for image generation.
We present snow degradation to show that TACoS is not unstable or produce artifacts like Algorithm 1 even for unconventional non-deterministic degradations.




>Q7: In Section 3.3, the author uses the first order Taylar approximation for arbitary s. However, this approximation only holds for small s.
>
Our analysis for small $t$ is not meant to be a complete convergence theory – our goal is to highlight one desirable theoretical property that our method has but a naive sampler lacks.  In the case of Algorithm 1, the restoration operator suffers from compounding error; if you perform the sampling operator multiple times with $t \approx 0$, the image keeps changing even though it should remain constant.  Meanwhile, the TACoS sampler has a fixed-point property; applying the sampler with $t=0$ does not change the image, and applying it with $t \approx 0$ updates the image in the correct direction.  This fixed point property is a necessary, but not sufficient, requirement for good noiseless reconstruction.  We do not claim to have a complete theory that predicts the good behavior of diffusion (nor does any paper on this topic), but we think this analysis helps to explain one reason why this update rule is superior to something more simple.

Interestingly, the “small t” regime seems to be where Alg 1 breaks down and Alg 2 performs better.  This behavior is seen qualitatively in Figure 2, where we compare the two sampling methods. Notice how in the small $t$ regime (final three frames) a colorful artifact begins to appear when Algorithm 1 is used. We have modified section 3.3 to clarify this point and emphasize that we are reasoning about the asymptotic, and not global, convergence properties of the two algorithms.
 
 
>Finally, I still suggest a FID comparison with Gaussian diffusion model on CIFAR10. The author can compare with the basic one [1], which does not involve much engineering. This can further verify the argument that "randomness is not necessary", if the gap between this work and Gaussian diffusion is small.
[1] Denoising diffusion probabilistic models
>
To broaden our comparison, we have now conducted an experiment using the same architecture and loss function where we use noise as the degradation for CIFAR10. We degrade the clean image using noise to the similar levels as for blur in terms of FID score. Thus the degraded images for CIFAR10 have an FID score of 363.53, the direct reconstruction has 43.07 and the iterative sampling using [1] gives 11.26. 
This FID score is better than the FID score we achieve using blur, but we show 400 images in figure 21 that the image quality of generated images are nevertheless suprisingly good. The images in figure 21, further surprises us because it challenges the common belief that Gaussian noise in any form, either during training or sampling is **necessary** for diffusion models to even produce realistic images.

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: Author Response to wT7w
comment: Thank you for your detailed reply. We'll provide further clarification and additional experimental results below:

>Q3: I have understood the relationship between DDPM/DDIM and Algorithm 1. It seems there is still a difference. The noising operator D in DDPM/DDIM depends to xs, and the noising operator D in this paper is independent to xs
>
Thanks a lot for your response, and we would like to clarify further on this. The D in Algorithm 1 can depend on $x_s$ during sampling. 
Given the degradation operation $D(x_0, t) = \sqrt{\alpha_{t}} x_0 + \sqrt{1 - \alpha_{t}}\epsilon$, one can come up with different ways to choose $\epsilon$. If one chooses to sample a new $\epsilon$ from $\mathcal{N}(0, 1)$ at every operation of $D$, then yes the noising operator D is independent of the noisy image $x_t$. At the same time, if one chooses to estimate the $\epsilon$ based on the predicted clean image $\hat{x_0}$ and the noisy image $x_t$ as $\hat{\epsilon}(x_t, t) =  \frac{x_t - \sqrt{\alpha_t} \hat{x_0}}{\sqrt{1-\alpha_t}}$, then the operator $D$ depends directly on the noisy image $x_t$. This choice of D, is the simplified version of the sampling process described in DDIM given $\sigma_t=0$.
Hence the dependency of D on the noisy image $x_s$ is just the matter of how one choses the noise to degrade the image.

>Q5: How about CIFAR10 and MNIST?
>
Thanks for bringing this to our notice. For our experiments in section 4, we present our results on the standard held-out testing dataset which we mention in the first paragraph of section 4. Hence for both CIFAR10 and MNIST, we evaluate our results on 10,000 held-out testing data.

From: ICLR.cc/2023/Conference/Paper5035/Reviewer_BSRf
title: Response from reviewer
comment: I am satisfied with the authors' response and keeping my score, still leaning towards acceptance.

From: ICLR.cc/2023/Conference/Paper5035/Reviewer_wT7w
title: Further questions
comment: It seems that Q6 and Q7 are missing.

Finally, I still suggest a FID comparison with Gaussian diffusion model on CIFAR10. The author can compare with the basic one [1], which does not involve much engineering. This can further verify the argument that "randomness is not necessary", if the gap between this work and Gaussian diffusion is small.

[1] Denoising diffusion probabilistic models

From: ICLR.cc/2023/Conference/Paper5035/Reviewer_wT7w
title: Further questions
comment: Thanks for the detailed reply.

Q3: I have understood the relationship between DDPM/DDIM and Algorithm 1. It seems there is still a difference. The noising operator $D$ in DDPM/DDIM depends to $x_s$, and the noising operator $D$ in this paper is independent to $x_s$.

Q5: How about CIFAR10 and MNIST?

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: General Author Response 
comment: We thank all of the reviewers for their thoughtful comments. Based on all the suggestions, we have updated our draft and we would like to highlight main contributions of our work.

1. In this paper, we aim to challenge the common belief that Gaussian noise in any form, either during training or sampling is **necessary** for diffusion models to work. We believe this will not only be of theoretical interest, but it will move the community to investigate other components that could improve diffusion such as model architecture or training setup. 

2. We prove the above point by using different types of **deterministic** degradations. We believe works that use other noise-based degradations like gamma or salt-and-pepper noise, miss the more general message of our paper.

3. We noticed there was confusion about whether our Algorithm 1 was indeed equivalent to the standard sampling process for diffusion models explained in DDIM/DDPM. To clarify this, we have added a detailed explanation in Appendix A.8. 

Overall, we believe our finding that cold diffusions can perform high-quality generation is surprising and provides a key insight for the burgeoning study of diffusion models as a whole. Hence, we believe it would be valuable for our work to appear at ICLR.

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: Author Response to wT7w Part 2
comment: >In Appendix A.6, it author set, D(\hat{x_0}, t) = \sqrt{1- \alpha_t} \hat{z} +  \sqrt{\alpha_t} \hat{x_0}which is inconsistent with that in Section 5.1. It would be better to choose another notation for  in Appendix A.6.
>

We use $z$ to refer to a fixed noise used during the sampling We thank the reviewer for this suggestion, and we have updated the draft to clarify

>In Table 5, the author shows that using DDIM sampling method would improve the sample quality compared to TACoS. Would the sample quality be further improved if we use the traditional diffusion model training objective? i.e., using a random z instead of a fixed z during training.
>

We would like to stress that for our control experiments using Gaussian noise, we **only** use a randomly sampled $z$ during training. The mention of a fixed $z$ in Section 5.1 and Table 5 refers exclusively to the **sampling** process. Note that for sampling, one can either use the estimated noise $\hat{z}$ to degrade the predicted clean image $\hat{x_0}$ or one can use the initially sampled Gaussian noise $z$. If one decides to use the initial Gaussian noise $z$ throughout **all** the steps of the sampling process, we say that the degradation operator $D$ uses the fixed noise $z$.

>Besides, the idea of using different forward process is not new. For example, [1,2] have formally studied diffusion models with other forward processes.
>

We thank the reviewer for pointing out these references, and we have updated our related work section to include them. Though they do not use Gaussian noise, both of these papers still use noise-based forward processes. As mentioned above, the use of deterministic degradations for the forward process is a central component of our paper’s message. i.e. Randomness during training and sampling is a **sufficient** condition for diffusion to work, but it is not **necessary**.

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: Author Response to wT7w Part 1
comment: Thank you for your detailed review of our paper. We address each of your comments and hope to clarify the message of our work below:

>This work lacks a comparison to the diffusion model that adds Gaussian noise... is overclaimed.
>

We would like to stress, that the point of our work is to challenge the intuition from [score-matching, DDPM] that noise is **necessary** for learning the training data distribution and effective sampling. We do not claim our generation is better than the well-researched and highly engineered SOTA Gaussian diffusion models (see “hot diffusion” in Table 5 for a fair comparison to our models). However, we believe our experiments do effectively demonstrate that even entirely noise-free schemes (blur, inpainting, super-resolution) can still achieve high quality image generation. Ultimately, our goal in this paper is not to produce a SOTA model, but to refine our working understanding of why diffusion works in the hopes of facilitating future improvements.

>In Section 3.2, the author states "Rather, diffusion models (Song et al., 2021a; Ho et al., 2020) perform generation by iteratively applying the denoising operator and then adding noise back to the image, with the level of added noise decreasing over time. This is the standard update sequence in Algorithm 1." However, Algorithm 1 does not correspond to the denoising process of diffusion models (Song et al., 2021a; Ho et al., 2020). Indeed, these methods use a linear combination of x_0 and x_s to generate x_{s-1}, instead of only using x_0 as in Algorithm 1.
>

We thank the reviewer for bringing this confusion to our attention, and we have expanded further on why Algorithm 1 is equivalent to the sampling process of DDPM/DDIM  in Appendix A.8. 

To clarify the above point, consider Equation 12 in Song et al., 2021a which proposes the sampling method as follows: $x_{t-1} = \sqrt{\alpha_{t-1}} \cdot \text{``predicted $x_0$"} + \sqrt{1-\alpha_{t-1} - \sigma_t ^2} \epsilon_{\theta}(x_t) + \sigma_t\epsilon_t$ Here, $\epsilon_{\theta}(x_t)$ is the noise predicted by the diffusion model given $x_t$ and $t$. The term “predicted $x_0$” or $\hat{x_0}$ can be computed directly given $x_t$ and $\epsilon_{\theta}(x_t)$ as $\hat{x_0} = \frac{x_t - \sqrt{1 - \alpha_t} \epsilon_{\theta}(x_t)}{\sqrt{\alpha_t}}$. Using $\hat{z}$ instead of $\epsilon_{\theta}(x_t)$ and $\hat{x_0}$ to instead of “predicted $x_0$”, we have $x_{t-1} = \sqrt{\alpha_{t-1}} \cdot \hat{x_0} + \sqrt{1-\alpha_{t-1} - \sigma_t ^2} \hat{z} + \sigma_t\epsilon_t$

The sampling step can then be interpreted as follows: At each step $t$, we start with a noisy image $x_t$ and use the diffusion model to estimate the clean image $\hat{x_0}$ and the noise $\hat{z}$ that was added to $\hat{x_0}$ to arrive at $x_t$. In order to estimate $x_{t-1}$, one adds $t-1$ steps of noise to the predicted clean image $\hat{x_0}$ according to the schedule defined by the $\{\alpha_i\}$. Now one can add back noise in two ways:

1. Add back $\hat{z}$, the noise added to the clean image $\hat{x_0}$.

2. Sample a new, uncorrelated noise $\epsilon_t$.

One can also choose to use both using $\sigma_t$ as the coefficient that defines the weight for each type of noise. Equation 12 is structured such that for any choice of $\sigma_t$, the standard deviation of noise added back is always $\sqrt{1-\alpha_{t-1}}$. For the case of $\sigma_t=0$, this amounts to only adding back the estimated noise $\hat{z}$ and no uncorrelated noise $\epsilon_t$, which is exactly the DDIM sampling. For the case of $\sigma_t = \sqrt{(1-\alpha_{t-1})/(1-\alpha_{t})} \sqrt{1 - \alpha_{t}/\alpha_{t-1}}$, we get the sampling method described in DDPM (Ho et al., 2020). Nevertheless, for any choice of $\sigma_t$, the sampling method involves a denoising operation which we denote by $R(x_s,s)$ and a noising operation we denote as $D(\hat{x_0}, s-1) = x_{s-1}$ in Algorithm 1.
We thank the reviewer for bringing this confusion to our attention, and we have included this discussion in Appendix A.8.

>In Section 3.2, the author states "We find that the standard sampling approach in Algorithm 1 works well for noise-based diffusion". Is there any quantitative result?
>

As explained above, both DDPM and DDIM are equivalent to Algorithm 1. Hence, the existing literature indeed shows that the standard sampling approach works well for noise-based diffusion models.

>Missing experimental details. What is the number of generated samples to calculate FID? Which subset is used as the reference dataset to calculate FID?
>

For CelebA, we generate 6400 images to estimate the FID scores and they are measured with respect to the 27000 training images. For AFHQ, we again generate 6400 images to estimate the FID scores and they are measured with respect to the 14630 training images. We thank the reviewer for pointing this out, and we have updated the experimental details to include this information.

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: Author Response to BSRf Part 2
comment: >Figure 2 - could you give more details on this experiment? For example, is it a reconstruction (starting with a latent generated based on a real image) or generation task (starting from a latent sampled from a prior)? How many steps were taken between the shown images? Currently, the second image in the top row looks totally blank, did the predictor already produce a blank image?
>

We thank the reviewer for pointing out the confusion regarding this figure, and we have clarified the experimental details for this figure in Section 3.3. The context for Figure 2 is unconditional generation. The number of sampling steps at which these images are shown are [2, 4, 8, 16, 32, 64, 128, 192, 256] respectively. For Algorithm 1, we are likely seeing a totally blank square at first because the direct prediction is a very blurry image  (as seen in middle row of figure 17 in appendix), hence when we apply the degradation operator $D(R(x))$, we again loose all information. 


>"TACoS proposed in Algorithm 2 can be applied in this case by fixing the noise z used in the degradation operator D(x, s)" - I assume z is a constant used for generating all images, not resampled for every image?
>

We thank the reviewer for bringing this confusion to our attention, and we have updated Section 5.1 to clarify our explanations. We do **not** use a constant $z$ for training on all images, otherwise the learned model would overfit to this specific pattern of noise. Moreover, diffusion models are deterministic functions (i.e. for a fixed input, the output is always the same) so using a constant $z$ for **all** sampling would mean the input to the model is always the same, and thus the output would also never change.

When we say TACoS is applied by fixing the noise $z$, we mean that for a **single** generation we sample the gaussian noise $z$ once, and we use that same $z$ in the iterative degradation and restoration that takes place. This means at every step s, we have $D(R(x_s), s) = \sqrt{\alpha_t}R(x_s) + \sqrt{1-\alpha_t}z$ where z is fixed for all steps.

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: Author Response to BSRf Part 1
comment: Thank you for your detailed review of our paper. We address each of your comments and hope to clarify the message of our work below:

>Empirical results are not great in a sense that we can't generate better images with this method.
>

We would like to stress, that the point of our work is to challenge the intuition that noise is **necessary** for learning the training data distribution and effective sampling. We do not claim our generation is better than SOTA diffusion models right now. We believe our experiments effectively demonstrate that even entirely noise-free schemes work, and we believe this to be a surprising finding that is worthwhile to the community. 

>One downside of the method is that it leaves the question of learning the prior to ad-hoc approximations, or severely limits the number of generated images (as in case of animorphosis, when we can't generate more images with a deterministic procedure than the number of animal images in the training set).
>

We agree with the reviewer that for the cold diffusions we consider, the final distribution is estimated, which could present challenges for sampling. However, it is important to note that the original DDPM and score-based generative modeling papers have stronger claims about the necessity of noise than simply ease of sampling. As we mention above, the central goal of our work is to refine our understanding about the role of noise in diffusion models. Future work on   more favorable forward processes (with or without noise) could be engineered with simpler final distributions, but we believe this lies outside the scope of this paper. 

>“While this ansatz may seem rather restrictive, note that the Taylor expansion of any smooth degradation D(x,s) around x = x0,s = 0 has the form D(x,s) ≈ x + s · e + HOT" - the crucial detail though is that e = e(x) because it is the gradient of D and D depends on x. Would the following derivation still hold in that case? If not, you should state it clearly, because the way the current text is written suggests that it holds for all differentiable corruptions.
"x0 +(s−1)" - e is missing?
>

You are correct, our simple proposition only holds in the case of constant $e$.  We were aware of this when we wrote the paper, and did not mean for the comment in quotes to imply to the reader that it held for a broader class of models other than the simple ansatz.  We have modified the draft to indicate this more clearly. 

>worth citing https://arxiv.org/abs/2206.00364, they also investigate deterministic sampling
>

We thank the reviewer for pointing us towards this concurrent work. We have now added this paper as a reference in our Background section. 

>"denotes a norm, which we take to be l1 in our experiments" - does it work with l2, which is more commonly used?
>

Our implementation is built on the lucidrains repo which uses L1 as its default loss. The diffusion model literature uses both $\ell_1$ and $\ell_2$ reconstruction objectives (see the pytorch implementation of DDPM at https://github.com/lucidrains/denoising-diffusion-pytorch), although we now see a lot of repos using L2 and we agree with the reviewer that this may now be more common. We believe the choice of loss for degradations other than noise will be an interesting future study and could be an avenue to improve generation with cold diffusions, but we do not currently have models trained with the L2 loss. Currently we are running experiments using L2 loss as well.


[2] Zhao, Hang, et al. "Loss functions for image restoration with neural networks." IEEE Transactions on computational imaging 3.1 (2016): 47-57.

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: Author Response to 8WRE
comment: Thank you for your detailed review of our paper. We address each of your comments and hope to clarify the message of our work below:

>However, the quantitative results of existing diffusion models should be adequately reported in the table, and additional qualitative comparisons are necessary. For example, the authors can compare their models’ generation quality with DDIM [Song.etal2021] where the sampling step is also deterministic.
>

We thank the reviewer for bringing this confusion to our attention. Please refer to the Hot Diffusion columns of Table 5 to see FID scores for the comparable DDIM sampling method. We have now updated the text of the main body to clarify that we are indeed performing this comparison and where it can be found. Additionally, please see Appendix A.6, where we prove that TACoS is equivalent to DDIM when one uses estimated noise during sampling.

>Similarly, the comparisons with other baseline methods ,e.g., generative models , are not presented in this paper.
>

We would like to stress that the point of our work is to challenge the intuition that noise is **necessary** for **diffusion** models to learn the training data distribution and sample from it effectively. We do not claim our generation is better than SOTA diffusion models or any other generative models right now. Yet, we believe our experiments effectively demonstrate that even entirely noise-free schemes can still work. While we agree with the reviewer that comparisons to traditional diffusion models (e.g. DDIM) are valuable, we believe benchmarking with generative models as a whole is outside the scope of this work. 

>Only low-resolution image datasets are considered in this work (e.g., 32, 64,128). As exploring advances in large scale diffusion models for image generation is impactful, I wonder how the proposed cold diffusion compares to DDPM/DDIM on higher-resolution dataset, such as 256x256.
>

We agree with the reviewer that it would be interesting to expand our methods to diffusion models on higher resolution images and eventually text-to-image models as well. However, the experiments we ran with our limited computational resources were designed to demonstrate that the existing intuition about the role of noise in both sampling and training is incorrect.

>In addition to the insufficient comparisons to the baseline, a clear motivation to use different degradations rather than using AWGN seems to be missing in its current state.
>

We thank the reviewer for this suggestion, and we have now included a discussion on the motivation behind applying cold diffusions rather than using AWGN in the Conclusion of our paper:

It is certainly possible that AWGN has advantages that make it more successful at the task of image generation, but accurately assessing the role of noise will be essential for engineering better models in the future. By the same token, if cold diffusions were as thoroughly studied and optimized as hot diffusion models, they could push SOTA as well. e.g. Reviewer 1 brought up [1], which expands on our work and shows that a combination of blur and a small amount of noise can beat SOTA diffusion models on the CelebA dataset.

[1] Rissanen, Severi, Markus Heinonen, and Arno Solin. "Generative modelling with inverse heat dissipation." (2022).

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: Author Response to h4aq Part 2
comment: >By checking FID scores, there's a huge gap between the proposed deterministic processes and the normal Gaussian diffusion
>

We would like to stress, that the point of our work is to challenge the intuition that noise is **necessary** for learning the training data distribution and effective sampling. We do not claim our generation is better than SOTA diffusion models. We believe our experiments effectively demonstrate that even entirely noise-free schemes can still work, and we believe this to be a surprising finding that is worthwhile to the community.

>The results are also worse than other recent works on deterministic diffusion processes, e.g. [1]. Also [1] is a missing reference.
>

We thank the reviewer for noting the relevance of this interesting recent arXiv paper.  The goal of our work is to challenge the intuition that noise is a crucial component of the training and sampling processes for diffusion models, whereas [1] actually does use uncorrelated Gaussian noise in both the training and sampling processes.  Moreover, we show that a wide range of degradation processes, such as blurring, masking, animorphosis, image compression, and snowification, can all yield effective diffusion models, calling the commonly believed theoretical underpinnings of diffusion models into question.  Nonetheless, we agree that [1] is relevant and complementary to our work.  Namely, [1] shows how an alternative degradation process (blur) can be expanded upon to achieve performance comparable to SOTA.

We would also like to stress that in Section 5, we perform a completely unconditional generation unlike the work [1]. Because the final image $x_T$ is not blurred to the point of being a channel-wise mean (see Figure 22 in [1]), there could still be some non-trivial residual information about the ground truth present in $x_T$.

We have now updated our paper to include a discussion of [1].

From: ICLR.cc/2023/Conference/Paper5035/Authors
title: Author Response to h4aq Part 1
comment: Thank you for your detailed review of our paper. We address each of your comments and hope to clarify the message of our work below:

>The assumption that the degradation operator is linear only holds with an infinitesimal s ... final sample quality.
>

Our analysis for small $t$ is not meant to be a complete convergence theory – our goal is to highlight one desirable theoretical property that our method has but a naive sampler lacks.  In the case of Algorithm 1, the restoration operator suffers from compounding error; if you perform the sampling operator multiple times with $t \approx 0$, the image keeps changing even though it should remain constant.  Meanwhile, the TACoS sampler has a fixed-point property; applying the sampler with $t=0$ does not change the image, and applying it with $t \approx 0$ updates the image in the correct direction.  This fixed point property is a necessary, but not sufficient, requirement for good noiseless reconstruction.  We do not claim to have a complete theory that predicts the good behavior of diffusion (nor does any paper on this topic), but we think this analysis helps to explain why this update rule is superior to something more simple.

Interestingly, the “small t” regime seems to be where Alg 1 breaks down and Alg 2 performs better.  This behavior is seen qualitatively in Figure 2, where we compare the two sampling methods. Notice how in the small $t$ regime (final three frames) a colorful artifact begins to appear when Algorithm 1 is used. We have modified section 3.3 to clarify this point and emphasize that we are reasoning about the asymptotic, and not global, convergence properties of the two algorithms.

>In Appendix, it is shown that the proposed sampler is the same as DDIM with frozen $z$ in training and in inference. First, I feel it is kind of weird way of choosing the noise by using $\hat{z}$ in sampling.
>

We thank the reviewer for bringing this confusion to our attention. In Appendix A.6, we are showing our proposed sampler is the same as DDIM with **estimated** $z$ (i.e. $\hat{z}$), not frozen $z$. Moreover, choosing the noise by using $\hat{z}$ is exactly what is done for DDIM.  

To clarify the above points, consider Equation 12 in the paper on DDIM, which proposes the sampling method as follows:
$x_{t-1} = \sqrt{\alpha_{t-1}} \cdot$ “predicted $x_0$” + $\sqrt{1-\alpha_{t-1} - \sigma_t ^2} \epsilon_{\theta}(x_t) + \sigma_t\epsilon_t$

Here $\epsilon_{\theta}(x_t)$  is the noise predicted by the diffusion model given $x_t$ and $t$. The term “predicted $x_0$” can be computed directly given $x_t$ and $\epsilon_{\theta}(x_t)$. The term $\epsilon_t$ refers to the uncorrelated Gaussian noise added during sampling. The term $\sigma_t$ controls the relative magnitudes of predicted and uncorrelated noise used to compute $x_{t-1}$.
The DDIM paper tunes $\sigma_t$ as a hyperparameter and finds in their Table 1 that the best sampling performance is observed at $\sigma_t = 0$.

Hence, DDIM can be further simplified as
$x_{t-1} = \sqrt{\alpha_{t-1}} \cdot$ “predicted $x_0$” + $\sqrt{1-\alpha_{t-1}}\epsilon_{\theta}(x_t)$

Using our nomenclature, this equation becomes:
$x_{t-1} = \sqrt{\alpha_{t-1}}\hat{x_0} +\sqrt{1-\alpha_{t-1}}\hat{z}$, as used in Appendix A.6. The only difference from the original DDIM is the order for estimating $\hat{x_0}$ and $\hat{z}$. The original DDIM paper estimated $\hat{z}$ first and then used this to predict clean image $\hat{x_0}$, while we first predict the clean image $\hat{x_0}$ and then estimate the noise $\hat{z}$.

>Why using L1 norm for the training objective instead of L2
>

We chose the L1 norm because it was the default loss function used by the lucidrains repository that served as a base for our implementations.   We suspect this choice may have been motivated by observations that $\ell_1$ losses tended to perform better than $ell_2$ in previous studies of neural network based image restoration [2]. The diffusion model literature uses both $\ell_1$ and $\ell_2$ reconstruction objectives (see the pytorch implementation of DDPM at https://github.com/lucidrains/denoising-diffusion-pytorch which offers both choices). Currently we are running experiments using L2 loss as well.

[2] Zhao, Hang, et al. "Loss functions for image restoration with neural networks." IEEE Transactions on computational imaging 3.1 (2016): 47-57.

From: ICLR.cc/2023/Conference/Paper5035/Reviewer_h4aq
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper proposes a general framework for diffusion models with different types of deterministic degradation process. The framework minimizes a training objective similar to the original diffusion models, which is about minimizing l1 norm between the original clean sample and the predicted clean sample by a restoration network. A new sampler is proposed and some analysis conducted to show that this sampler is preferred than the naive noise-denoise sampler if the sampling process starts from an infinitesimal noise level. Experimental results show that the proposed framework can produce reasonable samples for generation and several image restoration tasks, with several deterministic degradation processes. 
strength_and_weaknesses: ### Strengths:
- It is interesting to see that by some simple tweaks of the training objective and the sampler can work for several very different degradation  processes. The results is suggestive that more principled stepwise reversal of arbitrary lossy transformations could work really well. 
- The paper is well-written and easy to follow. 
- The paper conducts exhaustive experiments to valid the proposed approach across multiple tasks and multiple degradation processes.

### Weaknesses:
- I'm confused by the analysis of TACoS: The assumption that the degradation operator is linear only holds with an infinitesimal $s$. Therefore, the conclusion that TSCoS is robust to error only holds if sampling starts from an infinitesimal $t$. However, the empirical comparisons of the two samplers are shown with very large $s$ which cannot be explained by the analysis. In practice, the sampling around high-nois levels also likely contributes a lot to the final sample quality.
- In Appendix, it is shown that the proposed sampler is the same as DDIM with frozen $z$ in training and $\hat{z}$ in inference. First, I feel it is kind of weird way of choosing the noise by using $\hat{z}$ in sampling. Second, the FID is much worse than DDIM from a normal Gaussian diffusion. Does it indicate that switching from stochastic to deterministic degradation process can be a very important factor for degradation in performance? 
- Why using $\ell1$ norm for the training objective instead of $\ell2$?
- The results are not very convincing: By checking FID scores, there's a huge gap between the proposed deterministic processes and the normal Gaussian diffusion. The results are also worse than other recent works on deterministic diffusion processes, e.g. [1]. Also [1] is a missing reference. 
- Compared with Direct restoration baseline, TACoS seems to introduce more artifacts in some scenarios (e.g., Fig. 6 for celebA). 

1. Rissanen, Severi, Markus Heinonen, and Arno Solin. "Generative modelling with inverse heat dissipation." (2022).


clarity,_quality,_novelty_and_reproducibility: - Clarity: the paper is clear in written.
- Quality: the theoretical analysis and experimental results can be further improved.
- Novelty: the framework is general and prosed sampler is kind of novel. 
- Reproducibility: they provide enough details to reproduce the main results. 
summary_of_the_review: Overall speaking, I think this paper serves as a good proof of concept that it is indeed possible to generalize normal Gaussian diffusion to highly non-trivial degradation processes by designing the reversal process carefully. However, the results unfortunately are not quite convincing. The theoretical analysis also doesn't well support the claim that the new sampler is better. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5035/Reviewer_8WRE
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a new variant of two related classes of diffusion models, e.g., denoising diffusion and denoising score-matching. Unlike traditional diffusion models that generate data through denoising reverse processes, this work uses deterministic degradations, such as blurring, and trains the corresponding restoration neural network for the restoration reverse processes. Numerical validations are conducted on several image degradation models, such as blurring, inpainting, and super-resolution.
strength_and_weaknesses: Strengths:

1, The idea of using treating the reverse process of a diffusion model as a series of deterministic image restoration rather than denoising is very interesting. 

2, The analysis and experiments are extensive and can support the claims.

3,The paper is well written and easy to follow.

Weakness: 

1, Numerical comparison with existing denoising-based diffusion models is missing.  The fact that this paper edges out the existing result is okay, as this paper provides a totally different scheme for diffusion models. However, the quantitative results of existing diffusion models should be adequately reported in the table, and additional qualitative comparisons are necessary. For example, the authors can compare their models’ generation quality with DDIM[Song.etal2021] where  the sampling step is also deterministic.

2, Similarly, the comparisons with other baseline methods ,e.g., generative models , are not presented in this paper.
 
3, Only low-resolution image datasets are considered in this work (e.g., 32, 64,128).  As exploring advances in large scale diffusion models for image generation is impactful, I wonder how the proposed cold diffusion compares to DDPM/DDIM on higher-resolution dataset, such as 256x256.

4, In addition to the insufficient comparisons to the baseline, a clear motivation to use different degradations rather than using AWGN seems to be missing in its current state.  Although this works aims to proof of concept, it is unclear the benefits to do so. Will this change result in better performance, or a more efficient reverse process ? The authors don’t show clear cut about this.

5, There is no unified routines to design the latent space. Unlike traditional diffusion models sampled from a simple distribution, the approach loses its appeal when considering sophisticated design of the latent space. For example, the number of steps should depend on the image dimensions when using SR as degradation.

clarity,_quality,_novelty_and_reproducibility: Clarity/Quality : The description of the method is clearly stated, and the paper is easy to follow.

Novelty: The studied problem is interesting and worth devoting to. To this best of the reviewer’s knowledge, this is the first work to show that multiple degradations can be used to design diffusion models, but there are still several remaining issues as above.

Reproducibility: Code is provided in the supplementary, so I think the reproducibility is not the concern here.
summary_of_the_review: Consequently, given the pros and cons on balance, I feel this is a borderline paper, and I vote for borderline accept tentatively.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5035/Reviewer_BSRf
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a new method for deterministic generation of images called TACoS. It allows to invert arbitrary image transformations, with a few investigated experimentally: deterministic gaussian noise, blurs, animorphosis, masking, pixellating and snowing. The paper first evaluates it on the task of reconstruction based on a latent produced from a real image, and then on the task of unconditional generation by heuristically suggesting a prior for generating a latent.
strength_and_weaknesses: \+ The topic is certainly interesting and advances our understanding of generative models.

\+ Paper is clearly written and easy to read.

\- Empirical results are not great in a sense that we can't generate better images with this method.

\- One downside of the method is that it leaves the question of learning the prior to ad-hoc approximations, or severely limits the number of generated images (as in case of animorphosis, when we can't generate more images with a deterministic procedure than the number of animal images in the training set).

\- There are some questions about the motivation for the specific functional form of TACoS which need clarification (see below).

Questions/minor comments:
* worth citing https://arxiv.org/abs/2206.00364, they also investigate deterministic sampling
* "Diffusion has been understood as a random walk around the image density function using Langevin dynamics (Sohl- Dickstein et al., 2015" - not sure Sohl-Dickstein is relevant, it's variational
* "denotes a norm, which we take to be l1 in our experiments" - does it work with l2, which is more commonly used?
* [motivation] "While this ansatz may seem rather restrictive, note that the Taylor expansion of any smooth degradation D(x,s) around x = x0,s = 0 has the form D(x,s) ≈ x + s · e + HOT" - the crucial detail though is that e = e(x) because it is the gradient of D and D depends on x. Would the following derivation still hold in that case? If not, you should state it clearly, because the way the current text is written suggests that it holds for all differentiable corruptions.
* "x0 +(s−1)" - e is missing?
* Figure 2 - could you give more details on this experiment? For example, is it a reconstruction (starting with a latent generated based on a real image) or generation task (starting from a latent sampled from a prior)? How many steps were taken between the shown images? Currently, the second image in the top row looks totally blank, did the predictor already produce a blank image?
* "TACoS proposed in Algorithm 2 can be applied in this case by fixing the noise z used in the degradation operator D(x, s)" - I assume z is a constant used for generating all images, not resampled for every image?
clarity,_quality,_novelty_and_reproducibility: * Quality is high - the experiments are conducted properly.
* Clarity is also high, except the motivation for the functional form.
* Originality is moderate - prior work did investigate deterministic sampling, and I can't say the quality of samples in this work stands out in any way.
* Reproducibility is high - the authors provided source code in a supplementary material.
summary_of_the_review: This is an interesting empirical paper contributing to our understanding that per-step sampling might not be such a crucial component of diffusion generative models that was previously thought. I am learning towards acceptance.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5035/Reviewer_wT7w
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This work proposes to use forward processes other than the Gaussian noise. This work learns using the x0 prediction formulation. This author proposes TACoS to build the reverse process. Experiments provide various types of forward processes.
strength_and_weaknesses: Strength:

This paper demonstrates that it is possible to use forward process other than the Gaussian noise.



Weakness:

1. This work lacks a comparison to the diffusion model that adds Gaussian noise. This work provides some generated samples, e.g., Figure 7 and Figure 21, and their sample quality looks worse than the diffusion model that adds Gaussian noise. For example, the diffusion model that adds Gaussian noise can generate sharper images than this work.

2. Since the sample quality is worse than the diffusion model that adds Gaussian noise, the argument " The success of these fully deterministic models calls into question the community’s understanding of diffusion models.." is overclaimed. 

3. In Section 3.2, the author states "Rather, diffusion models (Song et al., 2021a; Ho et al., 2020) perform generation by iteratively applying the denoising operator and then adding noise back to the image, with the level of added noise decreasing over time. This is the standard update sequence in Algorithm 1." However, Algorithm 1 does not correspond to the denoising process of diffusion models (Song et al., 2021a; Ho et al., 2020). Indeed, these methods use a linear combination of $x_0$ and $x_s$ to generate $x_{s-1}$, instead of only using $x_0$ as in Algorithm 1.

4. In Section 3.2, the author states "We find that the standard sampling approach in Algorithm 1 works well for noise-based diffusion". Is their any quantative result?

5. Missing experimental details. What is the number of generated samples to calculate FID? Which subset is used as the reference dataset to calculate FID?

6. It seems that TACoS sometimes performs worse than direct reconstruction. For example, in Figure 6, the samples of TACoS have more artifacts than the direct reconstruction. In Table 4, the TACoS quantative result is missing.

7. In Section 3.3, the author uses the first order Taylar approximation for arbitary $s$. However, this approximation only holds for small $s$.

8. In Appendix A.6,  it author set $D(\hat{x}_0, t) = \sqrt{\alpha_t} \hat{x}_0 + \sqrt{1-\alpha_t} \hat{z}$, which is inconsistent with that in Section 5.1. It would be better to choose another notation for $D$ in Appendix A.6.

9. In Table 5, the author shows that using DDIM sampling method would improve the sample quality compared to TACoS. Would the sample quality be further improved if we use the traditional diffusion model training objective? i.e., using a random $z$ instead of a fixed $z$ during training.


Summary:

Overall, the sample quality of the proposed method is not competitive to diffusion model with Gaussian noise. This is perhaps because TACoS is not able to exactly reverse the forward process. This contrasts to the reverse SDE or the ODE formulation in Gaussian diffusion model, which are able to exactly reverse the forward process. Besides, the idea of using different forward process is not new. For example, [1,2] have formally studied diffusion models with other forward processes. So in my opinion this work does not provide a sufficient contribution for a ICLR publication.

[1] Denoising Diffusion Gamma Models

[2] Structured Denoising Diffusion Models in Discrete State-Spaces

clarity,_quality,_novelty_and_reproducibility: This work is written generally clearly, although there is some notation inconsistency. This work is relatively less novel, since using other forward processes is not new.
summary_of_the_review: See "Strength And Weaknesses"
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold