Paper: DIFFNAT: IMPROVING DIFFUSION IMAGE QUALITY USING NATURAL IMAGE STATISTICS (ICLR-2024)

Reviewer comment (Evaluation):
The visual results in Fig 6 do not show evident superiority of using KC loss. It is recommended to provide visual results with their corresponding KC statistic maps.

Author response:
Response: Thanks for the suggestion!  More visual results have been provided in Fig. 18, 19, 20, 21, 22, 23, 24, 25 (Appendix).

Reviewer comment (Evaluation):
The computational complexity of KC loss is not discussed. It is recommended to report the additional time consumption caused by KC loss.

Author response:
Response: The computation complexity of the proposed loss is presented in Sec. F (Computation complexity). The runtime analysis has been provided in Sec. G (Training time analysis) and Tab. 6 (Appendix).

Reviewer comment (Theory):
The authors argue that they provided theoretical insights into the advantage of KC loss, however, the theorems only analyzed the correctness of KC loss but did not analyze why the KC loss is beneficial for diffusion model training. I think the authors over claimed their contribution.

Author response:
**Q1. [the theorems only analyzed the correctness of KC loss but did not analyze why the KC loss is beneficial for diffusion model training]**

Response: In the theoretical analysis, we motivate justifying how the KC loss improves denoising. Note that we are not proposing a new training methodology for diffusion models, rather the proposed loss is a plug-and-play module which can be integrated into any standard diffusion model pipeline. (as shown empirically in Sec. 4.1, 4.2, 4.3 of the main paper).

Reviewer comment (Experiments):
In the literature of natural image prior modeling, a large variaty of models have been suggested for modeling natural image prior. Besides KC loss, are there any other priors which is beneficial for training diffusion models.

Author response:
**Q2. [ Besides KC loss, are there any other priors which are beneficial for training diffusion models?]**

Response: The most well-known prior for natural images is scale invariance, and (Zhang & Lyu) show that kurtosis concentration property can be motivated by the well-known prior for natural images, characterized by scale invariance. There might be some other natural image priors, but we haven’t considered those in this paper (beyond the scope of this paper). Alternatively, we have also compared our method with LPIPS loss (i.e., a perceptual loss) , which we have added as a baseline as shown in Tab. 1, Tab. 2, Tab. 3 in the revised version.

Reviewer comment (Evaluation):
The authors only analyzed the final generation quality with widely used image quality metrics, it will be the best if the authors could further present other properties of the proposed model, for example, can KC loss improve the convergence speed, will the additional KC loss leads to longer training time?

Author response:
**Q3. [it will be the best if the authors could further present other properties of the proposed model, for example, can KC loss improve the convergence speed, will the additional KC loss leads to longer training time?]**

Response: Thanks for the suggestion! We have provided the complexity analysis of the proposed KC loss in Sec. F (Computation complexity). Additional experiments for timing analysis have been provided in Table. 6 (training time analysis). 

The main idea of the diffusion model is to train a UNet, which learns to denoise from a random noise to a specific image distribution. More denoised steps typically ensure a better denoised version of the image, as seen in DDPM, LDM. In proposition 1, we show that minimizing projection kurtosis further denoise input signals. Therefore, KC loss helps in the denoising process and improves the convergence speed of the UNet. This has been discussed in the paragraph right after proposition 1 in the main paper. Fig. 3 shows using the same number of steps (400), a model trained with KC loss generates better quality images (measured in terms of SNR) than a standard guided diffusion model (both models were trained for the same number of epochs). This has been addressed in Sec. I (convergence analysis) in the Appendix of the updated version of the paper. The Loss convergence is shown in Fig. 14 in the Appendix of the updated version of the paper.

Reviewer comment (Experiments):
The experiments are not extensive. More comparisons on the conditional image-to-image translation should be provided.

Author response:
Response: We have updated the main paper with additional experiments using more baselines in Tab. 1, Tab. 2, Tab. 3. We also include additional qualitative results in Fig. 18, 19, 20, 21, 22, 23, 24, 25 in the Appendix.

Reviewer comment (Evaluation):
Despite improvement in quantitative results, the visual differences in using the loss or not are minor.

Author response:
Response: We have provided more qualitative results (zoomed version) in Fig. 18, 19, 20, 21, 22, 23, 24, 25 (Appendix).

Reviewer comment (Experiments):
This technical route, i.e. using IQA for improving generation quality, seems to be less effective than other alternative routes, e.g. [1].  
[1] Susung Hong, Gyuseong Lee, Wooseok Jang, Seungryong Kim, "Improving Sample Quality of Diffusion Models Using Self-Attention Guidance," ICCV, 2023.

Author response:
Response: Our proposed loss function is used during training diffusion models. On the other hand, [1] is a training free approach and they propose to use self-attention guidance during inference. Therefore, these approaches are not comparable, but rather complementary to each other. Moreover, [1] uses an additional forward pass to generate the attention map, which increases the computation complexity 2x compared to guidance free method, whereas our method introduces very less overhead during training (See Table. 6, Appendix).

Reviewer comment (Presentation):
The presented visual results are in small resolutions. Higher resolution results are suggested for putting into the paper.

Author response:
Response: We have presented the results in Fig. 11, 12, 18, 19, 20, 21, 22, 23, 24, 25 (Appendix) in high resolution.

Reviewer comment (Evaluation):
Why are the FID values in the paper so high compared to those in other papers?

Author response:
Response: For a fair comparison, we have reproduced the baselines with the same training setting and images, and the results are reported. In a similar setting, using KC loss produces images with lesser FID and higher MUSIQ scores across tasks and datasets, which validate the efficacy of the proposed KC loss.

Reviewer comment (Evaluation):
My main concern is about effectiveness of this KC loss. In the experiment part, the authors gives FID and MUSIQ in Table 1, 2, which I think shows little improvements. And we all know that these metrics can not accurately measure image quality. Moreover, in Fig 6, Fig 8, Fig 9 and Fig 10, it is not obvious whether KC loss yields better results. I think the authors need to at lease hightlight the region or illustrate using zoom-out regions to show the difference.

Author response:
Response: The effectiveness of KC loss has been shown quantitatively in Tables 1, 2, 3 (main paper). We have shown some qualitative examples in the paper, and based on the suggestion we have added more results highlighting the improved image quality. We have also verified the quality improvement using user study (Sec. human evaluation, page 7 main paper) by Amazon Mechanical Turks, and show that the generated images are indeed high quality. The comprehensive analysis is added in Sec J (Qualitative analysis, Appendix) ( Fig. 18, 19, 20, 21, 22, 23, 24, 25 in Appendix).

Reviewer comment (Theory):
Although I also appreciate very much those early works about natural image priors, I think most of them are based on simplistic assumptions which may not be needed in today's large-data-driven methods. More specically, I doubt that why kurtosis-based losses suit for the mentioned artifacts problem? Actually, losses need to be designed based on properties that you want to distinguish. I think at least statistics of a large amount of artifact images are needed to prove kurtosis is a proper metric.

Author response:
Response: We use the naturalness prior to design a plug-and-play loss function to improve the image quality of the generated images. The loss can be added to any diffusion pipeline and further improve the image quality which is experimentally validated in the paper (Tab. 1, Tab. 2, Tab. 3 in the main paper). Hence, we contend that this loss does not serve as a substitute for large-scale data-driven methods; instead, it complements and enhances existing pipelines.

Reviewer comment (Evaluation):
In Fig 1, the authors show severe unnatural artifacts. However, I wonder is that really a common phenomenon? It is more like a bug in training process, and similar artifacts are not present in the other illustrations in the paper. Then what exactly are the types of artifacts that the authors want to solve?

Author response:
Response: Our primary goal is to improve the image quality of generated images produced by diffusion models for different tasks. We are not specifically targeting any unnatural artifact reduction during the generation process. One of the tasks is personalized few-shot finetuning (e.g., Dreambooth), featured in the teaser figure (Fig. 1). In Dreambooth, having unnatural artifacts is a known phenomenon (Ruiz et al.) since overfitting/underfitting might happen due to training a large model from a few examples, which might lead to poor-quality images, having higher FID scores. Therefore, the unnatural artifact issue is just an anecdotal instance and is not the primary motive of the paper. For example, this is not an issue in unconditional image generation, image super-resolution tasks. We primarily focus on improving the overall image quality in this paper.