Paper: Zero-shot Image Restoration via Diffusion Inversion (ICLR-2024)

Reviewer comment (Novelty):
Unclear motivation: Similar to DPS [1], the proposed method (SHRED) also solves IR tasks in an optimization manner. The main difference is, DPS [1] optimizes the intermediate result $\mathbf{x}\_{t}$, but SHRED optimizes the initial noise $\mathbf{x}\_{T}$. I wonder what are the benefits of optimizing $\mathbf{x}\_{T}$? This article does not clearly point out SHRED's advantage, and it does not compare with DPS [1]. Besides, the authors claim that "This design choice does not alter the intermediate diffusion latents and thus provides more guarantees that the generated images lies in the in-distribution manifold", which lacks experimental or theoretical support. For example, the visualization of the optimized $\mathbf{x}\_{T}$ is necessary to judge whether it lies in the in-distribution manifold. In summary, the necessity and superiority of this method are questionable.

Author response:
**Unclear motivation/ Lack of a theoretical support**:

our guarantee that SHRED generates samples along the data manifold is **by construction**. SHRED works by sampling from the posterior distribution $P(x|y, \mu, \Sigma)$, where $\Omega_{real} = \\{x is real; \| H(x) - y\| = 0 \\}$  is its support. SHRED ensures realness (and so that the generated samples are in the data manifold) by directly using the diffusion reverse process as is (without modification).

To give an idea, our method is equivalent to a brute force approach where we sample infinity initial noises $x_T$, apply the diffusion to get $x_0$ and at the end return the $x_T$, that minimizes $\| H(x_0) - y \|$. It is clear here that the generated $x_0$ are by construction in the data manifold. Instead of a computationally "impossible: brute force approach, we use gradient descent. So, the guarantee that the generated image by SHRED is within the data manifold is  **Equal** to the guarantee that when sampling a random noise and feeding through DDIM will generate a sample within the data manifold. Why? again, simply because the sampling path (between $x_T$ and $x_0$) of SHRED and  DDIM is exactly the same.

Reviewer comment (Experiments):
Insufficient experiments: Lack of comparison with optimization-based methods, e.g., DPS [1], GDP [2].

Author response:
**Insufficient experiments**: we will consider adding comparisons with the more recent works such as DPS[1] and GDP[2] in our next version of the paper.

Reviewer comment (Evaluation):
Objectivity: Lack of discussion of limitations.

Author response:
**Lack of discussion of the limitations**:  we thank the reviewer for pointing this out. We will take this into account in the next revision.

Reviewer comment (Writing):
Typos: There are many typos in this article. The authors need to check carefully. For example, "the generated images lies in the in-distribution manifold" should use "lie" rather than "lies"; Errors in labeling the best and second best methods in Table 1 and Table 2.

Author response:
**Typos**: we will improve the writing of the paper in its next version.

Reviewer comment (Experiments):
There is no comparison with DPS which is a well-known diffusion-based solver that has better performance than the included competing techniques and source code is made public. For instance, for 4x SR *with noise* DPS reports far better LPIPS than any techniques in this paper.

Author response:
**Doubts about the quantitative results:** we will consider adding more quantitative comparisons with recent works such as DPS using the same experimental setting (1000 samples). The reported results in our paper can't be directly compared with the ones published in the competing method's papers  as we don't use the same set/number of images.

Reviewer comment (Evaluation):
Distortion metrics such as PSNR, NMSE and SSIM are completely missing, which are crucial in evaluating inverse problem solvers.

Author response:
**No distortion metrics are used**:  The restoration tasks that we consider in our paper involve severe degradation (inpainting, x8, x16 super resolution), so the range of the plausible solutions is wide. The PSNR between the ground truth $x$ and reconstructed image ($PSNR(x, reconstructed)$) considers only a unique solution and advanges an over-smoothed result. For this, we followed [1] and [2] and report only perceptual metrics such as FID and LPIPS. However, we will consider adding a metric capturing the fidelity such as $PSNR(y, H(reconstructed))$ where $y$ is the degraded input image and H(reconstructed) is the degraded reconstructed image.

Reviewer comment (Evaluation):
A very small (100) number of samples is used for evaluation. In other competing methods it is standard to use a 1000-sample validation split. Thus the results are not necessarily reliable and it is very difficult to compare to existing published results (DDRM, DDNM, DPS). Furthermore, since FID heavily depends on the number of samples used in the generated distribution, the reported FIDs are not compatible with the ones reported in competing method's original papers.

Author response:
**Doubts about the quantitative results:** we will consider adding more quantitative comparisons with recent works such as DPS using the same experimental setting (1000 samples). The reported results in our paper can't be directly compared with the ones published in the competing method's papers  as we don't use the same set/number of images.

Reviewer comment (Evaluation):
I have doubts about the reported timing results in Table 3. SHRED is reported as approx. 5x slower than DDRM. According to the DDRM paper, they use 20 NFEs. How is the reported timing possible, when SHRED uses 100 outer loop iterations with 10 NFEs in each outer loop (total 1000 NFEs) plus the additional cost of 100 backpropagation?

Author response:
**Doubts about the reported timing**: In our reported results for both DDNM and DDRM, we use number of denoising steps = 100 for the superresolution task. For SHRED, we use 50 optimization iterations and for every optimization iteration, we use 10 denoising steps ($\delta t = 100$). So at the end, the computational time for SHRED is approximately $\times 5$ times the one of DDNM and DDRM as shown in Table 3.

Reviewer comment (Evaluation):
The robustness experiments could be more rigorous. Instead of showing some good looking samples, it would be more meaningful to quantify the variation of image quality metrics for the validation dataset over 5-10 samples.

Author response:
**Quantitative metrics for the robustness experiments**: we will consider this point in our next revision.

Reviewer comment (Experiments):
The framework is developed for noisy inverse problems, however there are no experiments for the noisy case. Reconstruction performance under measurement noise is crucial in evaluating the utility of the algorithm.

Author response:
**Noisy inverse problems**: We report the result of SHRED for the case real-world image enhancement in Figure 7, where the images are noisy. Concerning the quantitative results, we will consider adding this case in the next version.

Reviewer comment (Experiments):
How does the method compare with DPS and other methods already presented in the paper (DDRM, DDNM) in terms of both perceptual and distortion metrics and on the standard 1000 samples from ImageNet and CelebA?

Author response:
**Doubts about the quantitative results:** we will consider adding more quantitative comparisons with recent works such as DPS using the same experimental setting (1000 samples). The reported results in our paper can't be directly compared with the ones published in the competing method's papers  as we don't use the same set/number of images.

Reviewer comment (Evaluation):
How is the discrepancy in point 5) under Weaknesses explained with respect to timing?

Author response:
**Doubts about the reported timing**: In our reported results for both DDNM and DDRM, we use number of denoising steps = 100 for the superresolution task. For SHRED, we use 50 optimization iterations and for every optimization iteration, we use 10 denoising steps ($\delta t = 100$). So at the end, the computational time for SHRED is approximately $\times 5$ times the one of DDNM and DDRM as shown in Table 3.

Reviewer comment (Experiments):
How does the method perform on noisy inverse problems?

Author response:
**Noisy inverse problems**: We report the result of SHRED for the case real-world image enhancement in Figure 7, where the images are noisy. Concerning the quantitative results, we will consider adding this case in the next version.

Reviewer comment (Theory):
Why is the technique framed as a linear inverse problem solver? The linearity of the operator is not exploited.

Author response:
**SHRED is only a linear inverse problem solver?**:  one of the advantage of SHRED compared to the other existing diffusion such as DDNM and DDRM is that can equally handle linear and non-linear inverse problems. In Figure 8 of our appendix, we show the result of (non-linear) non-uniform image deblurring. We will more point out this aspect in our next version of the paper.

Reviewer comment (Theory):
**Guarantee of Data Manifold Integrity:** The paper positions SHRED as a method that maintains the integrity of the data manifold during image restoration, which is a central claim for its novelty. However, the paper lacks a rigorous demonstration or proof that the samples generated by SHRED indeed lie on the data manifold. This is a significant gap, as the main criticism of prior methods is their potential deviation from the manifold. To strengthen this claim, the authors could provide empirical evidence or a theoretical guarantee, possibly through visualizations of the manifold or quantitative measures that can assess this aspect.

Author response:
**Theoretical justification for the claim that SHRED generates samples along the data manifold**: our guarantee that SHRED generates samples along the data manifold is **by construction**. SHRED works by sampling from the posterior distribution $P(x|y, \mu, \Sigma)$, where $\Omega_{real} = \\{x is real; \| H(x) - y\| = 0 \\}$  is its support. SHRED ensures realness (and so that the generated samples are in the data manifold) by directly using the diffusion reverse process as is (without modification).

To give an idea, our method is equivalent to a brute force approach where we sample infinity initial noises $x_T$, apply the diffusion to get $x_0$ and at the end return the $x_T$, that minimizes $\| H(x_0) - y \|$. It is clear here that the generated $x_0$ are by construction in the data manifold. Instead of a computationally "impossible" brute force approach, we use gradient descent. So, the guarantee that the generated image by SHRED is within the data manifold is  **Equal** to the guarantee that when sampling a random noise and feeding through DDIM will generate a sample within the data manifold. Why? again, simply because the sampling path (between $x_T$ and $x_0$) of SHRED and  DDIM is exactly the same.

Reviewer comment (Evaluation):
**Computational Efficiency:** In Table 3, SHRED is slower than DDRM and DDPM, which could limit its practicality for real-world applications where computational resources or time are constrained. The authors could explore ways to improve the efficiency of SHRED, perhaps by optimizing the algorithm or by proposing a more computationally efficient variant that maintains most of the method's benefits.

Author response:
**Computational efficiency boost**: we control the efficiency of SHRED by the hyperparameter $\delta t$ which defines the jump length in DDIM sampling. We will consider thinking of other potential ways to further improve the efficiency.

Reviewer comment (Novelty):
**Novelty and Originality in Mathematical Derivation:** The mathematical derivation of SHRED's methodology does not appear to be a novel contribution, which may lead to questions about the paper's originality. The authors could strengthen this aspect by clearly delineating the novel components of their mathematical approach, contrasting it with existing methods, and discussing how these novel aspects contribute to the method's performance.

Author response:
**Novelty in the mathematical derivation of SHRED**:
we argue that SHRED is an original inverse problem solver. Our approach is orthogonal to all the existing diffusion based methods as being the first method based on diffusion inversion. Our method takes advantage of the bijection property between $x_T$ and $x_0$ in DDIM and  proposes an efficient optimization scheme (with respect only to the initial noise $x_T$) for DDIM inversion for solving inverse problems through playing on its hyper-parameters such as $\delta t$. although being simple, our inversion scheme is original as it is the first optimization-based one. The existing method for inverting DDIM is done by solving the reverse ODE. In our case, reversing the ODE will not work as the degraded image is out of distribution and so we propose an optimization based approach. A big part of  SHRED derivation is based on the exact DDIM derivation as we are inverting DDIM at the end.

Reviewer comment (Writing):
The writing of the paper requires substantial improvement. The method described is not complex—it essentially pertains to an optimization-based inversion method supplemented with some implementation tricks—yet the paper is very difficult to understand. It requires multiple readings to identify the important optimization objectives and to guess the optimization methods. Figure 1 is also difficult to comprehend, and Section 3.1 seems unnecessary. The authors need to carefully revise the structure of the paper to improve the efficiency of information delivery. The current version is not suitable for publication.

Author response:
**Better writing/presentation**: we thank the reviewer for pointing this out. We will improve the paper presentation in our next revision.

Reviewer comment (Novelty):
There is a large body of literature related to GAN inversion that has not been discussed. Many existing works are actually very relevant to the methods of this paper, but they have not been carefully considered.

Author response:
**GAN inversion discussion**: we discussed some of the representative works of the GAN inversion based methods (both the optimized-based and learning based ) in "Methods based on GAN inversion" section of the Related work. Due to space limitation and that those methods are no longer sota, we didn't give more space to this discussion. However, We will consider enlarging it in our next revision.

Reviewer comment (Experiments):
The experiments presented in the paper are somewhat insufficient. There is no quantitative data supporting the discussion on the blind problems. Moreover, the paper's inversion method seems to not address the prompt issues of the diffusion model.

Author response:
**More experiments for the blind case**: One of the restoration tasks that we consider is blind deconvolution. The corresponding results are shown in Table 1 and Table 2. We will consider showing the results of more blind restoration tasks in the next revision. 

However, we don't understand what the reviewer means by  "the paper's inversion method seems to not address the prompt issues of the diffusion model"

Reviewer comment (Writing):
In relation to the method and Figure 1, $x_T$ is randomly sampled from a Gaussian distribution. It is unclear why $x_{0|T}$ can be such an image with a similar face to $y$ when there is no $y$ involved in this process. This appears nonsensical from the paper's description; the reviewer can guess what is actually done but considers this a writing problem.

Author response:
**Code**: we included our code in the supplementary material. At the first iteration of the optimization $x_T$ is a random noise and $x_0$ is not similar to $y$, throughout the gradient descent steps using the loss $\| H(x_0) - y \|$, $x_0$ becomes more and more similar to $y$.

Reviewer comment (Reproducibility):
The reviewer requests that the authors provide additional supplementary materials to show their actual method, such as code.

Author response:
**Code**: we included our code in the supplementary material. At the first iteration of the optimization $x_T$ is a random noise and $x_0$ is not similar to $y$, throughout the gradient descent steps using the loss $\| H(x_0) - y \|$, $x_0$ becomes more and more similar to $y$.