Paper: Solving Diffusion ODEs with Optimal Boundary Conditions for Better Image Super-Resolution (ICLR-2024)

Reviewer comment (Experiments):
The methods compared in Table 2 are all outdated. It is necessary to compare them with some state-of-the-art real-world super-resolution tasks [1,2].

Author response:
**W1. The methods compared in Table 2 are all outdated. It is necessary to compare them with some state-of-the-art real-world super-resolution tasks [1,2].**

Thanks for the reminder. We have included the comparisons in our rebuttal with more clarifications.

First, the proposed method is a novel sampling method that aims at getting better SR results from existing diffusion-based SR models. Thus, the effectiveness of our method is more revealed by the performance gap of the model with/without our sampling technique, instead of the overall performance comparison, which is demonstrated in existing comparisons (i.e., the bottom 3 rows in Tab. 2).

Second, our comparisons to DASR (published in ECCV 2022) have shown the effectiveness of our method in both LPIPS and NIQE.

Third, we give the results of Real-ESRGAN [1] and KDSR-GAN [2] in the following table.

||DIV2k-test|||RealSR|||
|-|-|-|-|-|-|-|
||NIQE$\downarrow$|LPIPS$\downarrow$|PSNR$\uparrow$|NIQE$\downarrow$|LPIPS$\downarrow$|PSNR$\uparrow$|
|Real-ESRGAN|4.805|0.3109|22.36|6.182|0.2511|25.12|
|KDSR-GAN|5.253|0.2840|22.92|6.673|0.2425|26.09|
|StableSR + DDPM-200 (official)|4.741|0.3189|19.42|6.185|0.3065|21.37|
|StableSR + DDIM-50 + $\tilde{\mathbf{x}}_T$ (Ours)|4.309|0.3169|19.55|5.252|0.2999|22.13|

Although our method does not outperform them in LPIPS due to the performance gaps between different baselines, it can still generate SR results with significantly better NIQE.

Reviewer comment (Experiments):
You need to compare with the state-of-the-art diffusion SR method [3], which also appears to have performed bicubic-SR and real SR tasks.

Author response:
**W2. You need to compare with the state-of-the-art diffusion SR method [3], which also appears to have performed bicubic-SR and real SR tasks.**

Thanks for the reminder. We will make efforts to add the related results in our updated version.

First, the proposed method is a novel sampling method that aims at getting better SR results from existing diffusion-based SR models, instead of a new diffusion-based SR model. Thus, the vital comparison is between our sampling method and previous sampling methods on the same diffusion-based SR model (i.e., the bottom 3 rows in Tab. 2), as we mentioned in the answer to your question 1. 

Second, we leveraged StableSR (https://arxiv.org/abs/2305.07015) as our baseline, which was pre-printed in May 2023. DiffIR [3] (https://arxiv.org/abs/2303.09472) was pre-printed in March 2023. Thus, StableSR is also up-to-date and competitive compared with DiffIR.

Last, we are glad to include the related comparisons. However, in the original paper of DiffIR [3], the real-SR task presented in our paper is not involved (only including image inpainting, bicubic-SR, and motion deblurring). We need more time to train the DiffIR on the task of real-SR to adapt it to our setting and include the related results in the future.

Reviewer comment (Evaluation):
Tables 1 and 2 show that the PSNR is not particularly high, indicating that the network's fidelity is not good. Super-resolution tasks not only seek visual improvement but also place great importance on fidelity. Compared methods have better fidelity. Therefore, I suggest the authors work on improving both PSNR (fidelity) and LPIPS, as this would provide stronger evidence of the effectiveness of your method.

Author response:
**W5. Tables 1 and 2 show that the PSNR is not particularly high, indicating that the network's fidelity is not good. Super-resolution tasks not only seek visual improvement but also place great importance on fidelity. Compared methods have better fidelity. Therefore, I suggest the authors work on improving both PSNR (fidelity) and LPIPS, as this would provide stronger evidence of the effectiveness of your method.**

The authors understand that PSNR is a well-recognized measure for SR. However, in recent years, its drawbacks have been revealed due to its failure to consider sampling randomness in measurement and not robust to simple geometric transformation. Therefore, recent generative model-based SR methods (including GAN-based, flow-based, diffusion-based, and so on) [4, 5, 6, 7] focus on the perceptual quality of SR images, mainly measured by LPIPS. This protocol is also adopted in a recently published paper [8], confirming that worse PSNR but better LPIPS leads to better perceptual quality.

Furthermore, thank you for your valuable suggestion of improving both distortion and perception simultaneously. However, it is extremely challenging to achieve such a goal due to the widely known "Perception-Distortion Tradeoff [9]" which has been studied in depth. We will take your advice into consideration seriously in our future work.

We really appreciate your excellent review of our paper. If our responses do not address all of your concerns, you are welcome to ask us to give more detailed discussions. We are glad to answer your further questions.

[1] Wang, Xintao, et al. "Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data", ICCVW, 2021.

[2] Xia, Bin, et al. "Knowledge Distillation based Degradation Estimation for Blind Super-Resolution", ICLR, 2022.

[3] Xia, Bin, et al. "DiffIR: Efficient Diffusion Model for Image Restoration", ICCV, 2023.

[4] Ledig, C., et al. "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network", CVPR, 2017.

[5] Wang, Xintao, et al. "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks", ECCVW, 2018.

[6] Lugmayr, Andreas, et al. "SRFlow: Learning the Super-Resolution Space with Normalizing Flow", ECCV, 2020.

[7] Saharia, Chitwan, et al. "Image Super-Resolution via Iterative Refinement", IEEE TPAMI, 2022.

[8] Gao, Sicheng, et al. "Implicit Diffusion Models for Continuous Super-Resolution", CVPR, 2023.

[9] Blau, Yochai, and Tomer Michaeli. "The Perception-Distortion Tradeoff", CVPR, 2018.

Reviewer comment (Evaluation):
The proposed method leverages LPIPS as the implementation of distance measurement function M(·,·). Can we leverage pixel-level metrics like negative PSNR as M? The authors should give more discussions.

Author response:
We can also use MSE (which is equivalent to negative PSNR) as the implementation and give results on bicubic-SR in the following table. 
||DIV2k-test||Urban100||BSD100||
|-|-|-|-|-|-|-|
||LPIPS|PSNR|LPIPS|PSNR|LPIPS|PSNR|
|DDIM-50 + LPIPS-oriented $\tilde{\mathbf{x}}_T$|0.1053|28.65|0.1164|24.26|0.1552|23.99|
|DDIM-50 + MSE-oriented $\tilde{\mathbf{x}}_T$|0.1060|28.56|0.1164|24.24|0.1557|24.01|

The results are slightly worse than the LPIPS-oriented $\tilde{\mathbf{x}}_T$ which is used in our paper. This is because MSE is not a less optimal metric to evaluate such distance. Lower MSE may not mean better image quality, which is a consensus in the domain of image SR [1, 2].

Reviewer comment (Theory):
The paper assumes that the model is well-trained. However, even a “well-trained” model cannot fit the real data distribution precisely. Thus, does such an assumption cause potential inaccuracy of the conclusion?

Author response:
Indeed, the gap between the parameterized distribution of diffusion models and real data distribution will affect the accuracy of our analysis, and this is the reason that our $\tilde{\mathbf{x}}_T$ is an "approximately" optimal boundary condition. We have pointed it out in our paper (in the text description before the Eqn. 16). In order to mitigate the inaccuracy, we leverage a reference set of LR-HR image pairs to find the $\tilde{\mathbf{x}}_T$, instead of one single LR-HR pair.

Reviewer comment (Evaluation):
It seems that the process of calculating the approximately optimal BC is time-consuming. How long does it take?

Author response:
The time of calculating the approximately optimal BC is about 20 hours on a V100 GPU. As we have mentioned in our paper, we take $R=300$ and $K=1000$ when calculating the approximately optimal BC. In practice, we did not simply sample $R \cdot K=300,000$ SR images. We first test all the $K=1000$ $\mathbf{x}_T$ on a small reference set containing 10 LR-HR pairs and only retain the top 300 $\mathbf{x}_T$ as a pre-screening process. Then we leverage a larger reference set and repeat the filtering process. At last, the final reference set contains all the 300 reference pairs. 

Such a time cost is a one-off. We just need to calculate the approximately optimal BC once for one diffusion-based SR model. This time cost is much lower than the time cost of training the model itself. Thus, such a process is not time-consuming.

Reviewer comment (Theory):
The paper only discusses the context of SR (and other low-level tasks in the Sec. 5). But it seems that the theory does not limit the relationship between guidance and generated results. Do the authors think that the method can be leveraged in more general generation tasks such like text-to-image generation?

Author response:
Your suggestion of leveraging the method in other tasks is so valuable. As we have mentioned in our paper, the proposed method can also be used in other low-level vision tasks. However, maybe it is not suitable enough for high-level generation tasks. Because these tasks do not have unique ground truths, it is difficult to define the concept of "optimal sample" which was defined by the ground truth (to be more specific, HR images in our paper). For example, in the task of text-to-image generation, as you mentioned, there can be several images that are semantically aligned to the input text. In other words, there is no exact "ground truth" to an input text.

Thank you so much for your suggestion. We will consider your advice seriously in our future works of leveraging our method in more low-level vision tasks.

Reviewer comment (Evaluation):
It seems that the improvement on StableSR (in the task of real-SR) is less than the improvement on SR3 (in the task of bicubic-SR). Could the authors discuss the reasons for this phenomenon?

Author response:
We think that the reason to the performance gap on StableSR and SR3 is that StableSR builds diffusion model in the latent space, while SR3 builds diffusion model in the image space. However, we actually propose the concept of "optimal sample" in the image space (Eqn. 9). Thus, the diffusion process in the latent space affects the accuracy of the approximation of our method (Eqn. 15, 16, 18) slightly, causing less improvement of StableSR. However, our method still helps StableSR to sample better results than its official sampling method (DDPM-100) in shorter time.

Thank you again for your approval of our paper and your valuable review. You are welcome to ask us to give further responses to address your concerns.