Paper: From Posterior Sampling to Meaningful Diversity in Image Restoration (ICLR-2024)

Reviewer comment (Writing):
A suggestion for the authors: The discussion on future work could be expanded, e.g., what specific desired variations can be for potential research in the field? This would provide more clarity and guidance for researchers interested in pursuing further work in this area.

Author response:
Thank you for your positive feedback. We believe there exist several interesting directions for future research. In particular, in this paper we investigate general meaningful diversity, which focuses on exploring different kinds of diversity at once. For example, in the context of restoration of face images, we aim for our representative set to cover diverse face structures, glasses, makeup, etc.  
However, for certain applications it can be desirable to reflect the diversity for a specific property, e.g. covering multiple types of facial hair and accessories while keeping the identity fixed, or covering multiple identities while keeping the facial expression fixed.  
The ability to achieve diversity in only specific attributes can potentially be important in e.g. the medical domain, for example to allow a radiologist to view a range of plausible pathological interpretations for a specific tumor in a CT scan, or to present a forensic investigator with a representative subset of headwear that are consistent with a low quality surveillance camera footage. \  
We included this in the revised manuscript.

Reviewer comment (Novelty):
The observation that random sampling leads to limited diversity for long tailed distributions was also noted in [a] in the context of image generation. [a] also proposes to modify the sampling process in diffusion models to generate samples from low density regions of a long tailed distribution. Though the focus in [a] is on generation, and not restoration, it is still a relevant work, and could be cited by the authors.

Author response:
Thank you for the constructive review. 

1. **Relevant citations:**\
Thanks for pointing us to these relevant works. In the updated manuscript, we added the papers by Sehwag et al. and by Montanaro et al. into a related work paragraph dedicated to works that enhance perceptual coverage.

Reviewer comment (Novelty):
The authors could also cite [b] which attempts to generate diverse solutions to linear inverse problems using pretrained GANs.

Author response:
Thank you for the constructive review. 

1. **Relevant citations:**\
Thanks for pointing us to these relevant works. In the updated manuscript, we added the papers by Sehwag et al. and by Montanaro et al. into a related work paragraph dedicated to works that enhance perceptual coverage.

Reviewer comment (Experiments):
It is not clarified what kind of down-sampling is used to obtain degraded images for super-resolution.

Author response:
3. **Additional questions/issues:**\
Thanks for pointing out these issues. We address them below and have revised the manuscript to clarify them:

   a. *Downsampling kernel*: In all super-resolution experiments, we use a bicubic downsampling kernel to obtain the degraded LR image, and add white Gaussian noise to the low-resolution image (with the mentioned noise level) for the noisy SR experiments. We agree that this was not clearly stated. We added this information to Sec. 4.1 and also to App. C.

Reviewer comment (Presentation):
Fig. 14, degraded input for super-resolution looks weird, the results of DPS look like down-sampled images.

Author response:
3. **Additional questions/issues:**\
Thanks for pointing out these issues. We address them below and have revised the manuscript to clarify them:

   b. *Fig. 14 in the supplementary*: We regret the confusion. We have now revised the figure to show, for each example, the degraded image on the left, and its corresponding model output on the right. We updated the caption accordingly.

Reviewer comment (Presentation):
Figs 28, 29 — super-resolution or inpainting: degraded image looks like a low resolution image, but the caption says inpainting.

Author response:
3. **Additional questions/issues:**\
Thanks for pointing out these issues. We address them below and have revised the manuscript to clarify them:

   c. *Figures 28 & 29 (now 34,35)*: Thanks. We fixed the captions to reflect the correct figure contents.

Reviewer comment (Experiments):
Experimental results are interesting but limited. For example, image colorization is one of the main problems where diversity reconstruction could have a significant impact.

Author response:
Thanks. Following your suggestion, we applied our diversity guided process (using DDNM) to image colorization for the CelebA-HQ and Imagenet datasets. We added example results to App. I.1 in the updated manuscript, where we indeed see a significant diversity gain with our method compared to the vanilla reconstruction.

Reviewer comment (Experiments):
The method seems to do the job, but the reviewer wonders if the authors could provide more insight on their proposed solution. For example, a few alternative designs could be: instead of increasing distance with respect to NN, just maximize the sample variance (or average pairwise distance as in the reported LPIPS metric).

Author response:
a. *Alternatives to increasing distance w.r.t. Nearest Neighbor*: Thanks for this good point. We actually experimented with several alternatives for increasing diversity, but found them all to lead to similar results. This is the reason we eventually decided to go with the simple nearest-neighbor (NN) solution. In the updated manuscript, we added to App. F a few comparisons between NN guidance and guidance that pushes the samples away from their average. As can be seen, both lead to good results, and the differences between them are small.

Reviewer comment (Evaluation):
The reviewer asks what happens with the sample average. In the traditional formulation, the average of the generated samples should be close to the posterior mean (which is also another possible estimator that minimizes distortion). Have the authors tried to compute and compare the sample means with and without the diversified guidance? The reviewer guesses that the mean wouldn't be as good as in the traditional case, but maybe a more robust mean estimator could lead to a very similar result (an estimator that minimizes distortion).

Author response:
b. *The effect on the sample average*: Following your question, we compared the RMSE of the average of our diversity-guided samples to the RMSE of the average of the vanilla samples. We found that the former is only 0%-2% larger than the latter (e.g. an increase of 0.63% in RMSE for SR with DDRM on the CelebA-HQ dataset). This relates to the previous point you raised, as it implies that although our samples are more spread than the vanilla samples, they are still spread roughly around the same point as the vanilla sample  – the posterior mean.

Reviewer comment (Novelty):
The reviewer suggests exploring connections to other work on GANs and mode collapse, which seems related, citing:
- Mao, Q., Lee, H.Y., Tseng, H.Y., Ma, S. and Yang, M.H., 2019. Mode seeking generative adversarial networks for diverse image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1429-1437).
- Yu, N., Li, K., Zhou, P., Malik, J., Davis, L. and Fritz, M., 2020. Inclusive gan: Improving data and minority coverage in generative models. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16 (pp. 377-393). Springer International Publishing.

Author response:
These GAN papers are indeed relevant. We added a discussion in the related work section about these and other related works, under a paragraph titled “Enhancing perceptual coverage”.