Paper: DiracDiffusion: Denoising and Incremental Reconstruction with Assured Data-Consistency (ICLR-2024)

Reviewer comment (Novelty):
Comparison and discussion to other relevant published work is not complete (see questions below).

Author response:
Thank you for the further feedback. We agree with the reviewer that the comparison with Delbracio and Milanfar (2023) could have been better highlighted and that more discussion on prior work with respect to the perception-distortion trade-off is justified. To address your points we updated the manuscript with the following changes:

- We included the work of Heitz et al (2023) in the introduction along with Delbracio and Milanfar (2023).

- We added an extra discussion on the connection between our framework and blending, namely blending can be thought of as a deterministic degradation process conditioned on the degraded image. This can be found in Section 3.2. We also added a pointer to the extra discussion in the Appendix as the reviewer suggested.

- We added a more thorough overview of prior work observing the perception-distortion trade-off in the context of iterative restoration/diffusion. Furthermore, we restructured the introduction to better highlight related diffusion-based image restoration methods.

As we mentioned earlier, given an extra page in the camera-ready version we would be open to move the comparison experiment with blending to the main paper. We highlighted the new additions in response to the reviewer’s feedback in blue in the updated manuscript. We hope that the above changes have sufficiently addressed your concerns. Thank you again for your constructive feedback and please let us know if you have further concerns.

Reviewer comment (Experiments):
Experimental results compare the proposed method to general frameworks that do not require retraining for specific degradations. In that sense it should be more fair to compare to other methods that train a dedicated model for a given degradation process.

Author response:
**Re Question 2:** Thank you for this suggestion. As we mentioned above, we provided a comparison on FFHQ with InDI which is a trained diffusion model-based solver. However, at the time of writing this paper the source code and checkpoints have not been publicly released and thus we had to rely on our own reproduction of the method with computationally heavy training. We are unfortunately unaware of diffusion solvers with fast fine-tuning for adaptation to various inverse problems, but we are happy to compare if the reviewer has some suggestions. Moreover, we added a strong supervised image restoration baseline (SwinIR) to our main result in the newer version of the manuscript that the reviewer might find useful. Based on the feedback, we would be open to moving the discussion on InDI to the main paper in the final version of the manuscript.

Reviewer comment (Novelty):
The reviewer would like the authors to comment on how the presented work compares to Daras et al. (2022). It seems that Daras et al. (2022) can be used to solve inverse problems, in a non-blind formulation, similar to what this paper is proposing. The reviewer asks whether the same strategy/sampling can be used in this context.

Author response:
**Re Question 1**: Thank you for bringing up these relevant works. 

- Re Daras et al. (2022): We build upon the work in Daras et al. (2022) and propose a novel framework specifically tailored for inverse problem solving and extending the formulation to general corruptions (they consider linear corruptions only). In particular, Soft Diffusion is defined as a solely generative framework, and thus it does not cover sampling from a posterior distribution given noisy measurements. Therefore, the considerations around data consistency, which is the main focus of our work, does not arise in their setting.

Reviewer comment (Novelty):
The reviewer would like the authors to comment on how the present work compares to Delbracio and Milanfar (2023), since this work also discusses the advantage of their bridge formulation in terms of Perception-Distortion trade off, is similar in the sense that it requires training a dedicated model for a specific degradation, and the convex combination formulation leads to a series of intermediate degraded images.

Author response:
**Re Question 1**: Thank you for bringing up these relevant works. 

- Re Delbracio and Milanfar (2023): In fact, we did compare our method with InDI (see Appendix G for a discussion and experimental results ). Overall, we found that InDI and DiracDiffusion provide similar perceptual reconstruction quality (we achieve slightly better FID, InDI has slightly better LPIPS). However, we have found that DiracDiffusion is more robust with respect to noise level perturbations in test time. We hypothesize that this is due to the fact that our model is trained to denoise on a range of noise variances due to the similarity of our objective to denoising score-matching. However, InDI reconstructs samples from convex combinations between noisy and clean images, and thus the model has not seen any other noise levels but the one used for training. We expect similar observations for Bridge methods as well. We would also like to highlight that these techniques work under a different framework from ours, where the forward model is unknown and thus their focus is not on data consistency (in fact it cannot be evaluated without knowing the forward process).

Reviewer comment (Novelty):
The reviewer notes that the above comparison also applies to Bridge methods that are applied to solving inverse problems, such as the one in Liu, G.H., et al., *I²SB: Image-to-Image Schrödinger Bridge*, ICML 2023.

Author response:
**Re Question 1**: Thank you for bringing up these relevant works. 

- Re Delbracio and Milanfar (2023): In fact, we did compare our method with InDI (see Appendix G for a discussion and experimental results ). Overall, we found that InDI and DiracDiffusion provide similar perceptual reconstruction quality (we achieve slightly better FID, InDI has slightly better LPIPS). However, we have found that DiracDiffusion is more robust with respect to noise level perturbations in test time. We hypothesize that this is due to the fact that our model is trained to denoise on a range of noise variances due to the similarity of our objective to denoising score-matching. However, InDI reconstructs samples from convex combinations between noisy and clean images, and thus the model has not seen any other noise levels but the one used for training. We expect similar observations for Bridge methods as well. We would also like to highlight that these techniques work under a different framework from ours, where the forward model is unknown and thus their focus is not on data consistency (in fact it cannot be evaluated without knowing the forward process).

Reviewer comment (Experiments):
The experimental results should also include methods that require training or finetuning a diffusion prior for solving a specific inverse problem, as proposed by the authors, and the reviewer would like the authors to comment on this point.

Author response:
**Re Question 2:** Thank you for this suggestion. As we mentioned above, we provided a comparison on FFHQ with InDI which is a trained diffusion model-based solver. However, at the time of writing this paper the source code and checkpoints have not been publicly released and thus we had to rely on our own reproduction of the method with computationally heavy training. We are unfortunately unaware of diffusion solvers with fast fine-tuning for adaptation to various inverse problems, but we are happy to compare if the reviewer has some suggestions. Moreover, we added a strong supervised image restoration baseline (SwinIR) to our main result in the newer version of the manuscript that the reviewer might find useful. Based on the feedback, we would be open to moving the discussion on InDI to the main paper in the final version of the manuscript.

Reviewer comment (Experiments):
My main concern is that, while the proposed method is sound, it relies on an assumption that seems unrealistic. Specifically, it assumes the degradation model $A$ being a priori known, which is unrealistic in practical applications, even in the tasks of deblurring and inpainting considered in this paper. The provided experimental results are based on synthetic data, in which the degradation model $A$ can be easily specified. However, when considering practical tasks, I believe the proposed method would not perform satisfactorily, as it is hard or even impossible to accurately specify the degradation model $A$. The authors have not provided evaluation on practical tasks, e.g., practical deblurring and inpainting tasks, in comparison with representative methods.

Author response:
As the reviewer points out, DiracDiffusion is a solver for inverse problems with *known* forward model. This is a clear distinction from general image restoration techniques that rely only on clean-noisy paired data for training without knowing the corruption process. However, in this general scenario there is no notion of *data consistency*: given a corrupted input and the reconstructed image we are unable to evaluate how well the reconstruction explains the input (without a ground truth). In stark contrast, our work focuses on inverse problems with known forward models, a common case in real-world problems where the physical process producing the observations is known (known camera blur, microscopy with known probe function, medical imaging problems such as CT or MRI etc). In this setting, a crucial requirement is data consistency: the reconstruction has to be consistent with the noisy measurement as dictated by the forward model. In fact, ensuring this consistency while producing exceptional image fidelity is the main goal of our paper. We would like to also highlight that other diffusion-based solvers overwhelmingly assume a known forward model as well ([1-3]). To address the reviewer’s concern we updated the main text to better emphasize this assumption.

Reviewer comment (Novelty):
The proposed perception-distortion tradeoff method by early-stopping the reverse process is quite straightforward and heuristic.

Author response:
Our early stopping method is indeed straightforward and intuitive, yet very effective. Furthermore, the early stopping technique is an interesting consequence of our proposed Stochastic Diffusion Process framework, and not the sole contribution of our work.

Reviewer comment (Theory):
In equation (10), the network learns to reconstruct $A_t(x0)$ rather than directly reconstructing $x_0$ or noise, compared with DDPM. However, when the degradation function $A_t(·)$ is linear, it would be equivalent to reconstructing $x_0$. In some tasks, the degradation model $A(·)$ would be linear, for example, the blur function $y=k*x + n$ in the deblurring task, where $k$ is a blur kernel. How can the deblurring experiment show the superiority of Dirac over DDPM when the degradation model $A(·)$ is linear?

Author response:
We point out that when the degradation process is linear ($\mathcal{A}_t(\cdot)$ is a linear mapping), reconstructing $\mathcal{A}_t(x_0)$ is not equivalent to reconstructing $x_0$ as the linear mapping is typically not invertible (as is the case in deblurring or inpainting). In fact, as we highlight in the paper, predicting $\mathcal{A}_t(x_0)$ from the noisy and degraded input (denoising only) is an easier problem than completely reconstructing $x_0$ in one step (denoising and reconstruction/inversion).

Reviewer comment (Writing):
There should be a space in “projectionSong”, line 11, the second paragraph of Introduction.

Author response:
Thank you for catching the typo, we fixed it in the new version. We are happy to discuss further if there are additional concerns and would like to ask the reviewer to consider raising their score in case we addressed all major issues.