Paper: A Restoration Network as an Implicit Prior (ICLR-2024)

Reviewer comment (Novelty):
The motivation of this work is not convincing enough. As I can understand, the aim of DRP is to better or more generally characterize the prior of images. However, it is doubtful whether this generalization is necessary since the deep denoising prior can be flexible enough by virtue of advanced neural network architectures.

Author response:
Our work is the first to generalize the notion of the priors beyond denoising networks. Despite their popularity, nothing in the existing theory says that denoisers are better priors in general. While denoisers are a special case of our formulation (H = I), our work opens the door to other restoration networks that were not considered before (H ≠ I). In fact, our work provides concrete evidence that the SR networks can be better than denoising networks on some tasks. 

Our motivation goes beyond “advanced network architectures”. One can train the same architecture—such as SwinIR—for denoising or the SR task. However, denoiser and SR models will give different priors and different results, despite corresponding to the same architecture. Prompted by your comment, Section B.7 shows the performance of DPIR using SwinIR trained as a denoiser. Our DRP uses the same SwinIR architecture, but trained to perform SR. Note how DRP does better than DPIR, despite both using priors with **identical network architectures**.

Reviewer comment (Experiments):
The experimental comparison seems unfair. In specific, the authors used SwinIR for their DRP, while using DRUNet for DPIR, though DRUNet was adopted in the original DPIR work. It is necessary to re-implement DPIR with SwinIR, or re-implement DRP with DRUNet, such that the influences of network structure can be controlled.

Author response:
Prompted by your comment, we re-implemented DPIR using SwinIR trained as a denoiser. These results are presented in Section B.7. DRP uses the same SwinIR architecture as the new DPIR (SwinIR), but the SwinIR in DRP is trained to perform SR. Note how DRP still does better than DPIR, despite both using priors with identical network architectures.

Reviewer comment (Evaluation):
As for evaluation metric, more commonly-used metrics should be employed to have a comprehensive comparison, such as SSIM, LPIPS.

Author response:
This is an excellent suggestion. Prompted by your comment, we report SSIM and LPIPS as additional evaluation metrics in Section B.5 of the supplement. Note the excellent performance of DRP for all metrics – PSNR, SSIM, and LPIPS – on all tasks.

Reviewer comment (Experiments):
More comparative methods should be considered, including the DIP prior-based methods DIPFKP (CVPR 2021) and BSRDM (CVPR 2022) and some directly trained based method, such as BSRNet (ICCV 2021), RealESRNet (ICCV 2021 workshop).

Author response:
Prompted by your comment, the revised manuscript considers two additional methods in Section B.6, including the DIP-based BSRDM and the directly-trained method BSRGAN. Note how in Table 8, DRP achieves better performance compared to both of these methods.

Reviewer comment (Experiments):
This is a non-blind method. The experiments only verify its simple case with known degradation. I wonder that is it able to handle the real-world case?

Author response:
As you correctly observed, DRP is designed to address non-blind inverse problems. Such problems are common in many applications, including microscopy and medical imaging. Extension of DRP to blind inverse problems would be an exciting research direction for the future. For example, one can imagine an extension that allows the update of the degradation operator during the inference of the DRP algorithm. We state in the “Limitations and Future Work” section of the revised manuscript our interest in the extension to blind inverse problems.

Reviewer comment (Experiments):
As shown in the appendix, its performance is obviously inferior to DRUNet. Additionally, it relies on the SwinIR as a prior. It is necessary to conduct a comparison with SwinIR.

Author response:
Section B.1 of the supplement does not imply that our method is in general inferior to DRUNet. It shows that SwinIR trained for bicubic SR can be used as a prior for denoising, but **as expected** it is worse than DRUNet specifically trained for denoising. This is better clarified in the revised Section B.1 .

In Section B4, we presented the SISR performance with and without DRP. Notably, SwinIR itself struggles with the SISR task in the absence of DRP. However, with the integration of DRP, we achieve state-of-the-art performance.

What’s more, In Section B.7, we reimplement DPIR using SwinIR denoiser and add it as a new ablation study for all experiment settings. Table 9 and Table 10 in Section B.7 of the supplement show that DRP with SwinIR SR prior can consistently outperforms DPIR with SwinIR denoiser prior.

Reviewer comment (Novelty):
According to my understanding, the motivation and idea of this work is very similar to IRCNN. The main difference to IRCNN is that it uses a more powerful restoration network SwinIR instead of DnCNN. The sprox operator in Eq. (5) corresponds to the sub-optimization problem Eq. (6a) of the paper of IRCNN. The core step 3 in Algorithm 1 corresponds to Eq. (6b) of IRCNN.

Author response:
There has been a lot of interest in using priors given as networks trained to perform image denoising. IRCNN, PnP-ADMM, RED, GS-PnP, and DPIR are all similar in the sense that they are based on this same principle.

Our work is the first to generalize this idea to priors given as networks trained to perform other restoration tasks. Our method is similar to IRCNN only when H=I; however, unlike IRCNN, our method allows for H ≠ I, making it compatible with many more priors.

The main difference is not that we use a “more powerful network” SwinIR, but that we use a network trained for a different task. Prompted by your comment, Section B.7 shows the performance of DPIR—a more recent variant of IRCNN—using SwinIR trained as a denoiser. Our DRP method uses the same SwinIR architecture, but trained to perform SR. Note how DRP still does better than DPIR, despite both using priors with ***identical network architectures***.

Reviewer comment (Writing):
Following the last question, step 3 introduce the restoration prior in Algorithm 1. I'm interested in that how to induce the updated procedure of step 3. In other word, a more intuitive explanation on step 3 should be provided.

Author response:
We added a more detailed explanation in Section 5.2 of the paper. As a  bicubic SR model is adopted as prior in the setting of Section 5.2, the R(Hx) in Step 3 of Algorithm 1 performs a bicubic downsampling of the intermediate image and inputs it into the bicubic SR SwinIR.

Reviewer comment (Evaluation):
The paper lacks some discussion/analysis of why restoration networks for problems other than denoising can lead to better results than denoising regularizers. Do they learn better image priors? This is somewhat surprising, in that these other networks are expected to have learned (in addition to a "prior") also specific aspects of the particular problem in which they were trained. This raises the question: what problems yield the best restoration networks? Is this choice related to the main problem to be solved?

Author response:
This is an excellent observation. As you correctly noted, by suggesting to consider restoration networks beyond image denoisers, our work opens an intriguing question of what is the **best** restoration network? This question was not considered before since the research in the area was primarily focused on using denoisers as priors. While we cannot yet answer the question of optimality, our paper provides concrete evidence that for some tasks other restoration networks are better priors than denoisers. We added a short discussion on this in the “Limitations and Future Work” section.

Reviewer comment (Experiments):
Although the authors present the method as working for general restoration networks, they end up only using restoration networks trained for super-resolution. Super-resolution is a very particular type of inverse problem, and arguably the one closest to pure denoising among inverse problems. The authors should comment on this.

Author response:
Indeed, it would be interesting to explore the performance of DRP for more priors, beyond the SR network. Prompted by your comment, we ran new experiments by including another prior corresponding to a deep unfolding network for solving image deblurring. The results are presented in Section B.9 of the supplement. As can be seen in Table 13, the deblurring network can serve as an excellent prior for regularizing SISR within our DRP method. We are certainly interested in exploring the performance of DRP under even more priors, but will leave this to future work due to limited rebuttal time.

Reviewer comment (Theory):
In Section 4, there is some confusion regarding necessary and sufficient conditions. The authors write "Our analysis will require several assumptions that act as sufficient conditions for our theoretical results." This is self-contradicting: if the assumptions are *required* they are *necessary*. Furthermore, further down, the authors contradict that sentence by writing "This mild assumption is necessary ...".

Author response:
We have reworded our statements for clarity.

Reviewer comment (Theory):
The first inequality in Theorem 2 is trivial and does not need to be mentioned.

Author response:
We have removed the first inequality from Theorem 2.

Reviewer comment (Writing):
Minor style issue: it is not good style to write "...discussion in Kamilov et al. (2023) on ..." or "... in Chapter 3 in Beck (2017) ...". Kamilov and Beck are people, not papers or books. A nicer way to write these sentences is "...discussion in the work of Kamilov et al. (2023) on ..." or "...discussion in the paper by Kamilov et al. (2023) on ..." and "... in Chapter 3 of the book by Beck (2017) ...".

Author response:
Prompted by your comment, we updated our style to always use (Author, Year) rather than Author (Year). We also revised corresponding sentences to make sense.

Reviewer comment (Theory):
A probability density function is degenerate not only if it is supported on a subspace, but on any zero-measure set in the ambient space, for example, a d-manifold with d < n.

Author response:
Indeed, we revised the statement to clarify that “subspace” is only one example.

Reviewer comment (Theory):
The form of Equation (9) suggests that using the regularizer trained to solve (4) is in some sense equivalent, or at least related, to analysis regularization/priors, where \( p_s \) is a learned prior and \( \mathbf{H} \) the analysis operator. See https://doi.org/10.1117/12.826663. The authors are asked what they think of this connection.

Author response:
While we didn't initially see the connection to analysis/synthesis priors, your comment provides a new perspective to our work. One can indeed view Eq. (9) in our formulation as an analysis regularizer with a transform H. This might open new directions for designing data-driven priors. We mention this new connection in the revised paper and cite the paper you shared there.

Reviewer comment (Reproducibility):
Question regarding the computational approach: How much is lost by solving (10) and (11) by conjugate gradient (CG), and how do the authors know that 3 iterations is enough? Would it be worth (at least for (10)) giving some careful thought to the form of the inversions needed; perhaps it is possible to exploit the fact that both \( \mathbf{H} \) and \( \mathbf{K} \) are convolutions to obtain closed-form inversions using FFTs?

Author response:
Indeed, one can design dedicated closed form implementations of the scaled proximal operator for some linear transforms. The reason we used CG was due to its generality, in the sense that the same implementation can be used for all linear transforms. We didn’t observe any improvements from a higher number of CG sub-iterations, which suggests that 3 is sufficient for our experiments. Note that it is common to use 2-5 iterations of CG within proximal algorithms by using outer iterations to initialize CG (see, for example, the discussion on page 88 of (Kamilov et al., 2023)).

Reviewer comment (Experiments):
The method utilizes a pre-trained SwinIR [1]. However, according to the Table 2 from SwinIR, the performance of SwinIR for the task super-resolution is ~6db higher in terms of PSNR on set5, which is confusing. The authors should include the SwinIR for all of the test datasets as a baseline and explain the reason there is a performance drop after adapting their method.

Author response:
We are addressing a different problem than the original SwinIR paper. The original SwinIR paper shows that SwinIR trained for bicubic SR performs well on bicubic SR. Our work shows that SwinIR trained on the bicubic SR can be a good prior to solve **other restoration tasks**. Table 2 in our paper uses SwinIR trained for bicubic SR as a prior to solve what we call the SISR task (which is SR under Gaussian blur kernels and additive white Gaussian noise). Figure 9 in Section B.4 of the supplement shows that SwinIR trained for bicubic SR cannot directly handle SISR, but provides an excellent prior for our DRP method to perform SISR.

Reviewer comment (Evaluation):
There is another main weakness in the experimental results. There seems to be no discussion on the sources of randomness for the proposed method and the compared works. I would assume for the proposed method, there could be randomness involved in all three main stages: the pre-training, the refinement process, and the Algorithm 1 itself. The same issue could also be true for the considered compared methods. The experimental study should consider multiple runs for each of the mentioned stages and provide individual or combined analysis on the distribution of results rather than just a single run. The reviewers need to be sure whether the provided performance numbers are the worst single runs of compared methods versus the best single run of the proposed method, or are average cases for all considered methods. Without providing such analysis using multiple runs, it is hard to assess the significance of the results for the proposed method.

Author response:
All the experiments presented in the paper use the same random seed to ensure consistency for all methods. We also always use the official models provided on GitHub and follow the recommended settings for each method. Prompted by your comment, we ran new simulations using additional random seeds. Table 14 in Section B.10 of the supplement reports these results. Note how DRP consistently outperforms DPIR for all Seed IDs, which indicates the robustness of our results to multiple random runs.

Reviewer comment (Evaluation):
The authors should include other evaluation metrics, such as SSIM and LPIPS, for the results. That would help to make sure the performance is not biased towards a single evaluation metric.

Author response:
This is an excellent suggestion. Prompted by your comment, we report SSIM and LPIPS as additional evaluation metrics in Section B.5 of the supplement. Note the excellent performance of DRP for all metrics – PSNR, SSIM, and LPIPS – on all tasks.

Reviewer comment (Experiments):
The authors mention one of the limitations for the proposed method is “the assumption that the restoration prior used for inference performs MMSE estimation.” However, it is not clear if for the compared methods, they have included any of recent methods that are trained based on l_1 loss and SSIM to clarify how much this limitation affects the final performance in different applications.

Author response:
The MMSE assumption is **only** necessary for our theoretical analysis of DRP. DRP can be run using models trained on L1 or SSIM losses. Prompted by your comment, we retrained SwinIR using the same settings introduced in Section 5.1, but using the L1 loss instead of L2. The results are presented in Table 15 in Section B.11 of the supplement. Note how there are no significant differences between DRP using L2 or L1 SwinIR priors, which suggests that numerically DRP is stable with or without the MMSE assumption.