Paper: Universal Guidance for Diffusion Models (ICLR-2024)

Reviewer comment (Experiments):
As any kind of guidance function can be used in the proposed method, it would be interesting to see how its performance varies according to the design of the guidance function. For example, in the case of segmentation map guidance, we may have a lot of publicly-available segmentation models that can be utilized for the guidance. In this case, can we simply choose the best performing model? This point is not clear in the manuscript, because only a single model is examined for each conditional generation task.

Author response:
In principle, our algorithm is not restrictive to the guidance function used. So we believe it is totally possible to experiment with different segmentation networks and simply pick the best one for empirical use.

Reviewer comment (Experiments):
It lacks ablation: as claimed in Sec 3.1, "directly replacing fcl and lce with any off-the-shelf guidance and loss functions does not work in practice". I think it is needed to include an experiment to support this claim. In addition, only "object location guidance" in Sec 4.2 provided the ablation of forward and backward guidance, the results for at least more than one tasks are helpful.

Author response:
As suggested by reviewers, we also present 8 more combinations of text and bounding boxes. We summarize the quantitative results in the table below, and present the qualitative results in Figure 1 of the [anonymized rebutall figures collection](https://drive.google.com/file/d/1DVmdqZ8QBcopO0npQLXGWdDDWQyZp3Qa/view?usp=sharing). The value is obtained by averaging over 10 random generations. We note that the "text" in the following table refers to a combination of object locations and text phrases, and please see the figure for details of each combination. Both quantitative and qualitative results demonstrate that using our proposed forward guidance with Eq. 6 leads to a much better match with the external guidance compared to Eq. 5 for off-the-shelf guidance functions.

|         | text 1 | text 2 | text 3 | text 4 | text 5 | text 6 | text 7 | text 8|
|---      | ---    | ---    | ---    | ---    | ---    | ---    | ---    | ---   |
|Eq. 6 (Ours)|0.51 | 0.69   | 0.69   | 0.92   | 0.44   | 0.61   | 0.55| 0.80|
|Eq. 5       | 0.13| 0.01   | 0.01   | 0.08   | 0.02   | 0.01   | 0.02| 0.06|

Reviewer comment (Experiments):
I appreciate the experiments on conditional stable diffusion in Sec4.1 and unconditional imagenet diffusion in Sec 4.2. However, I am curious why not also conducting segmentation map guidance in 4.2. I think including same task in both 4.1 and 4.2 can be helpful for readers to understand the differences between the two diffusion models.

Author response:
As suggested by the reviewer, we also present the generation results on ImageNet diffusion model using segmentation guidance in Figure 2 and Figure 3 of the [anonymized rebuttal figures collection](https://drive.google.com/file/d/1DVmdqZ8QBcopO0npQLXGWdDDWQyZp3Qa/view?usp=sharing). In particular, Figure 2 shows images generated with various segmentation maps, demonstrating that our algorithm is effective regardless of the underlying diffusion model. Figure 3 includes comparison between generations using ImageNet diffusion model and Stable Diffusion with the same segmentation mask.

Reviewer comment (Experiments):
My primary concern is: all presented visual results are not diverse - dogs are everywhere. I also checked all results in the supplemental, it is similar. How about other cases?

Author response:
In Figure 1 of the [anonymized rebuttal figures collection](https://drive.google.com/file/d/1DVmdqZ8QBcopO0npQLXGWdDDWQyZp3Qa/view?usp=sharing), we show generations with various combinations of dogs and cats in different positions. The results indicate that our algorithm can generate images beyond the "dog category", and we hope this alleviates the reviewer's concern about the diversity of generated images.

Reviewer comment (Experiments):
The method uses the predicted clean image as the input of the conditioning model. Would the inaccurate predicted clean image affect the performance?

Author response:
Indeed, as the reviewer mentioned, we use predicted clean images to compute the guidance signal and perform guided image generation. We argue that inaccurate clean image prediction has limited impact due to the step-by-step nature of image sampling for diffusion model. Specifically, impacts of guidance using gradient signal is averaged out across different sampling step. In the earlier sampling steps, the guidance signal provides coarse adjustment to the image based on comparatively inaccurate clean image predictions. As the sampling step progresses, the clean image prediction becomes more accurate and the corresponding guidance signal further refines the generated image. The success of our algorithm also validates the reasoning.

Reviewer comment (Novelty):
Some previous work [1, 2, 3] on the conditional diffusion model should be discussed.

Author response:
We thank the reviewer for providing pointers to these valuable references. Collectively, these reference papers belong to a family of algorithms that freezes the unconditional diffusion backbone and train auxiliary models for controlled image generation. This family of algorithms has a different goal in mind compared to our proposed Universal Guidance, where we aim for an algorithm that does not require re-training at all for different conditional generation tasks. Nevertheless, we believe this family of algorithms is an interesting middle ground between our algorithm and full-blown conditional training,  and presents exciting directions for further research. We will include the related discussion upon further revision of the paper.

Reviewer comment (Experiments):
The method uses the predicted clean image as the input of the conditioning model. Would the inaccurate predicted clean image affect the performance? The authors need to study this issue.

Author response:
Indeed, as the reviewer mentioned, we use predicted clean images to compute the guidance signal and perform guided image generation. We argue that inaccurate clean image prediction has limited impact due to the step-by-step nature of image sampling for diffusion model. Specifically, impacts of guidance using gradient signal is averaged out across different sampling step. In the earlier sampling steps, the guidance signal provides coarse adjustment to the image based on comparatively inaccurate clean image predictions. As the sampling step progresses, the clean image prediction becomes more accurate and the corresponding guidance signal further refines the generated image. The success of our algorithm also validates the reasoning.