Paper: DPO-Diff: On Discrete Prompt Optimization of Text-to-Image Diffusion Models (ICLR-2024)

Reviewer comment (Presentation):
The figure 1 should be professionally re-shaped by highlighting the main contributions, instead of listing each block.

Author response:
**Suggestion 1: show changes in Figure 1**

*“The figure 1 should be professionally re-shaped by highlighting the main contributions, instead of listing each block.”*

Thank you for your suggestion! We’ve modified Figure 1 to show the inference process of obtaining text gradient with and without shortcut gradient.

[1] Betker et al. Improving Image Generation with Better Captions. 2023

[2] Hao et al. Optimizing Prompts for Text-to-Image Generation. 2022

[3] Avrahami et al. Blended Diffusion for Text-driven Editing of Natural Images. CVPR 2022

Reviewer comment (Evaluation):
Only one metric of CLIP loss is used for evaluation, which makes the experimental results somewhat unconvincing. As pointed in [A], clip score does not correlate well with human choices. It is necessary to report the results under more metrics (e.g., Aesthetic Score, Human Preference Score [A], or Human Preference Score v2 [B]).

Author response:
**Concern on evaluation metrics:**

*“Only one metric of CLIP loss is used for evaluation, which makes the experimental results somewhat unconvincing. As pointed in [A], clip score does not correlate well with human choices. It is necessary to report the results under more metrics (e.g., Aesthetic Score, Human Preference Score [A], or Human Preference Score v2 [B]).* It is not safe to evaluate the prompt improvement task with clip loss in Table 1 (b), where the state-of-the-art method (Promptist) even shows worse performance than user input. I checked the Promptist paper, where extensive human study is performed to validate the effectiveness compared to user input. Thus I have to say the comparison in Table 1 (b) is somewhat not fair, and this work does not give enough credit to existing work of Promptist. Moreover, following Promptist paper, it is also necessary to perform human study to evaluate the proposal.”

**[Response]**

Thank you for your suggestion. We agree that the lack of a powerful automatic evaluation method is a limitation faced by the text-to-image generation field as a whole, and there is currently no perfect metric, including CLIP score. This is also why our original paper included human evaluation results to compensate it.

- Automated evaluation
1. **CLIP Score: While clip is not perfect, it is still the most widely-adopted automated metric to judge text and image similarity**, and hence for evaluating prompt following ability of diffusion models. For instance, it has been adopted in many existing works as the de facto metrics in this aspect, such as DALLE-3 [1], Promptist [2], BlendedDiffusion [3], etc. So we also followed the pioneers to adopt it as the automated metrics.
2. Aesthetic score: Aesthetic is an important aspect for evaluating image generation for artistic applications. However, **the main focus of our experiments is on the prompt following ability, and we do not optimize for aesthetics** like Promptist or MagicPrompt. In our paper, we respectfully acknowledged the effectiveness of Promptist in generating aesthetically pleasing images (Section 2 para 2). On this matter, we follow DALLE-3 to view prompt following and aesthetics as distinct aspects of image generation that can be evaluated and reported separately [1].
3. Human Preference Score v2: We included results on HPSv2 score in the Appendix F; Table 3 suggests that the performance of DPO-Diff is consistent on HPSv2. Note that HPSv2 entangles prompt-following ability and aesthetic ability into a single score; **Although we only optimize for prompt following ability, it is interesting to see that the proposed method also performs well when evaluated with HPSv2.**
- Human evaluation

**The human study on prompt optimization is already included in the original submission**, and discussed in Section 5.1, 5.3 and Appendix D.2. The practice of reporting quantitative results using an automatic metric and and human study is borrowed from previous works such as Promptist and DALLE-3. Moreover, following the suggestion of R3, **we also conducted the human evaluation on adversarial attack.** The result is summarized in the Appendix F, where DPO-Diff obtains a success rate of 44%.

We hope that the above discussion could address your concern on the evaluation metrics.

Reviewer comment (Experiments):
The prompt dataset in this work contains only 300 prompts for each task. It is better to evaluate the 3200 prompts on HPS benchmark [B].

Author response:
**Concern on the evaluation dataset size**

*“Moreover, the prompt dataset in this work contains only 300 prompts for each task. It is better to evaluate the 3200 prompts on HPS benchmark [B].”*

**[Response]**

Thank you for your suggestion.

1. We would like to clarify that, the 600 prompts used in our experiments are “hard prompts” filtered from **a pool of roughly 6000 prompts** (from coco, diffusiondb and chatgpt), as mentioned in Section 5.1. Concretely, for prompt improvement (adversarial attack) task, we focus on prompts that are hard (easy) for the diffusion model to follow. The filtering is conducted automatically by thresholding CLIP loss on the original Stable Diffusion model, without favoring a certain method.
2. Moreover, we found that **the statistics tend to stabilize after 100 prompts** (providing they are sampled from diverse sources). We conjecture that this is probably one reason why recent works typically only use a few hundred prompts, such as 170 for DALLE-3 and 400 for LLM-grounded Diffusion when evaluating the prompt following ability, as this is more resource and labor friendly (human evaluators).

Reviewer comment (Experiments):
Authors select an earlier version of Stable Diffusion 1-4 as the base model. I am curious to see the results when applying the proposal into superior version of Stable Diffusion (SDXL).

Author response:
**Concern on Stable Diffusion version**

*“Authors select an earlier version of Stable Diffusion 1-4 as the base model. I am curious to see the results when applying the proposal into superior version of Stable Diffusion (SDXL).”*

**[Response]**

Thank you for your suggestion. We originally picked v1-4 since this is most commonly used in academia due to resource efficiency. We also experimented with v2 and v1-5, but the results are very similar to v1. **We are currently repeating the prompt improvement experiments in SDXL.** As “hard prompts” for SDXL are different than v1-4, we need to rerun the data collection as well. Since SDXL is nearly 10x slower than v1 and can only be run on the very limited number of 48G GPUs we have, the final results will be delayed. We will try to update them in our revision if they can be finished before the end of the rebuttal. But based on the current progress, we observe the following preliminary results:

1. SDXL demonstrates improved prompt following ability than SD v1-4 or v2, as there are fewer hard prompts.
2. **The improvement of DPO-Diff over baselines seems consistent on SDXL.**

Reviewer comment (Experiments):
Following Promptist paper, it is also necessary to perform human study to evaluate the proposal.

Author response:
**Concern on evaluation metrics:**

*“Only one metric of CLIP loss is used for evaluation, which makes the experimental results somewhat unconvincing. As pointed in [A], clip score does not correlate well with human choices. It is necessary to report the results under more metrics (e.g., Aesthetic Score, Human Preference Score [A], or Human Preference Score v2 [B]).* It is not safe to evaluate the prompt improvement task with clip loss in Table 1 (b), where the state-of-the-art method (Promptist) even shows worse performance than user input. I checked the Promptist paper, where extensive human study is performed to validate the effectiveness compared to user input. Thus I have to say the comparison in Table 1 (b) is somewhat not fair, and this work does not give enough credit to existing work of Promptist. Moreover, following Promptist paper, it is also necessary to perform human study to evaluate the proposal.”

**[Response]**

Thank you for your suggestion. We agree that the lack of a powerful automatic evaluation method is a limitation faced by the text-to-image generation field as a whole, and there is currently no perfect metric, including CLIP score. This is also why our original paper included human evaluation results to compensate it.

- Automated evaluation
1. **CLIP Score: While clip is not perfect, it is still the most widely-adopted automated metric to judge text and image similarity**, and hence for evaluating prompt following ability of diffusion models. For instance, it has been adopted in many existing works as the de facto metrics in this aspect, such as DALLE-3 [1], Promptist [2], BlendedDiffusion [3], etc. So we also followed the pioneers to adopt it as the automated metrics.
2. Aesthetic score: Aesthetic is an important aspect for evaluating image generation for artistic applications. However, **the main focus of our experiments is on the prompt following ability, and we do not optimize for aesthetics** like Promptist or MagicPrompt. In our paper, we respectfully acknowledged the effectiveness of Promptist in generating aesthetically pleasing images (Section 2 para 2). On this matter, we follow DALLE-3 to view prompt following and aesthetics as distinct aspects of image generation that can be evaluated and reported separately [1].
3. Human Preference Score v2: We included results on HPSv2 score in the Appendix F; Table 3 suggests that the performance of DPO-Diff is consistent on HPSv2. Note that HPSv2 entangles prompt-following ability and aesthetic ability into a single score; **Although we only optimize for prompt following ability, it is interesting to see that the proposed method also performs well when evaluated with HPSv2.**
- Human evaluation

**The human study on prompt optimization is already included in the original submission**, and discussed in Section 5.1, 5.3 and Appendix D.2. The practice of reporting quantitative results using an automatic metric and and human study is borrowed from previous works such as Promptist and DALLE-3. Moreover, following the suggestion of R3, **we also conducted the human evaluation on adversarial attack.** The result is summarized in the Appendix F, where DPO-Diff obtains a success rate of 44%.

We hope that the above discussion could address your concern on the evaluation metrics.

Reviewer comment (Reproducibility):
Clarification question: how are the compact domain space contacted and shortcut gradient related? Do we dynamically select words to search for synonyms/antonyms based on the gradients?

Author response:
Thank you for the question. We feel that briefly summarizing the pipeline might answer your questions. The DPO-Diff framework is formulated as an optimization problem (eq.4), which includes a domain (a.k.a. search space) that includes potential candidates, and an objective that need to be solved over the domain space. The entire pipeline can be summarized in chronological order as:

- Domain Space: We define the domain space as the compact Synonym/Antonym Space generated for each user prompt. So these compact spaces are dynamic only in the sense that they differs for each user prompt. Once the domain space is generated for a given user prompt, it will be kept fixed through the optimization process.
- Given the domain space generated above, we use the proposed gradient-based algorithm (GPO) to search for a good prompt from the domain space. The gradient used in this algorithm is obtained via “Shortcut Gradient”.

To further improve the clarity of the paper, we’ve added the above summarization pipeline to beginning of Section 4.

Reviewer comment (Writing):
Clarity: 1) The asterisk symbol in eq 6 seems not usually used in this case; 2) it seems that the main issue with backproping through the entire inference chain is the prohibitive memory cost, so abbreviating it as "text gradient" is a little misleading and inaccurate.

Author response:
1. Asterisk

Thank you for pointing it out. Yes, since this is just a multiplication sign, it can indeed be omitted here. We’ve modified the equation.

1. Using the term “Text Gradient”

You are right that the main issue with backpropagating through the inference process is the memory and runtime cost. We use the term “text gradient” because we only deal with the gradient over discrete text prompts to solve the optimization problem. We then use the term “shortcut gradient” to refer to our method to efficiently obtain the “text gradient”. In terms of “shortcut gradient”, **it is indeed a much more general method than just computing “text gradient”, as it can be used to compute gradients w.r.t. any model components (weights, embedding, any conditional signal) through the inference steps.** We hope that could address your concern, but please let us know in case we misinterpreted your point.

Reviewer comment (Experiments):
Shortcut gradient: 1) is it necessary to use estimated x_0 from t-K? is there any ablation to support the claim? 2) would strategies like randomizing the truncation lengths [1] be helpful in this case?

Author response:
Thank you for raising this important point. Yes, it is necessary to estimate $x_0$ from $x_t$. The justification is as follows. Suppose we are currently located at a randomly sampled step $t$ between $T \sim 0$:

1. If we do not estimate $x_0$ from $x_t$, we will have to run the UNET of the diffusion model $t$ times to obtain the final $x_0$ and then propagate the loss on $x_0$ all the way back from step $0$ to step $t$. This is exactly what Randomlized Truncated Backpropagation (RTB) did, and the memory and runtime complexity becomes $\mathcal{O}(t)$.
2. If $t$ is uniformly sampled from $T \sim 0$, the complexity becomes $\mathcal{O}(t) = \mathcal{O}(T/2) = \mathcal{O}(T)$, which is still the same as brute-forcely backpropagate.
3. For RTB, the only way to reduce its theoretically linear complexity in practice is to pick a constant $t$ near the final image $x_0$. This means to truncate most but the last few inference steps. On GPUs with 48G memory, we found that the maximum $t$ it can support is only $t=3$ (out of 50 steps) for Stable Diffusion v1-4 (let alone larger ones such as SDXL). However, the gradient signal on text is not informative for such a small $t$, since the image is almost finalized at this point. To show this, we added an ablation on the gradient norm across timesteps in Figure 6 in the Appendix, where it shows that gradient norm near $t=0$ is almost negligible compared with the middle timesteps.

Hence, we conclude that estimating $x_0$ from $x_t$, as done in Shortcut Gradient, is critical in obtaining an informative text gradient efficiently. It is the only current solution to this problem, and for the first time makes gradient-based optimization on the inference steps of large-scale diffsion models practical.

Furthermore, we would like to make the following remarks to the above response:

1). **RTB and Shortcut gradient solve different problems.** RTB is proposed to solve the over-optimization problem [cite their paper], and it still incurs linear runtime and memory complexity, while Shortcut Gradient is proposed for efficiency.

2). **Our closed-form estimation of $x_0$ from $x_t$ is a principled solution grounded in the derivation of DDPM** (Remark 1 in the paper, with derivation added to the Appendix).

Reviewer comment (Writing):
The paper is not well written and easy to follow. Some core sections are not fully explained.

Author response:
- Reproducibility

We're planning to release all our code, since we view our method only as a starting point for future advancement. **For the rebuttal, we attached a draft version of the codebase to the supplementary material.** We are actively working on refactoring the codebase and are also looking into adding a demo on HuggingFace.

- Clarity

Thank you for your detailed suggestions! We’ve improved the presentation accordingly. The roadmap of the current manuscript is as follows: we first explain the framework in Section 4.1. This is followed by the search space design in Section 4.2 and the search algorithm for navigating that search space in Section 4.3. Within Section 4.3, we roughly follow the order of execution to lay out the complete algorithm, that is, we discuss how to learn a distribution over text space (GPO) first, then how to sample from it (evolutionary search).

**To improve the clarity of the paper, we made the following changes based on your suggestion:**

1. \+ The above roadmap to the beginning of Section 4.
2. \+ Discussions on Definition 4.1 in the Appendix
3. \+ Derivation for Remarks 1 and 2 in the Appendix.
4. \+ Extended discussion on the ablation study results in the Appendix.

We also welcome any further suggestions you may have, as we are also eager to make this rather dense work more reader friendly.

Reviewer comment (Presentation):
The paper structure in the method section can be better organized to clearly show the components step by step.

Author response:
- Reproducibility

We're planning to release all our code, since we view our method only as a starting point for future advancement. **For the rebuttal, we attached a draft version of the codebase to the supplementary material.** We are actively working on refactoring the codebase and are also looking into adding a demo on HuggingFace.

- Clarity

Thank you for your detailed suggestions! We’ve improved the presentation accordingly. The roadmap of the current manuscript is as follows: we first explain the framework in Section 4.1. This is followed by the search space design in Section 4.2 and the search algorithm for navigating that search space in Section 4.3. Within Section 4.3, we roughly follow the order of execution to lay out the complete algorithm, that is, we discuss how to learn a distribution over text space (GPO) first, then how to sample from it (evolutionary search).

**To improve the clarity of the paper, we made the following changes based on your suggestion:**

1. \+ The above roadmap to the beginning of Section 4.
2. \+ Discussions on Definition 4.1 in the Appendix
3. \+ Derivation for Remarks 1 and 2 in the Appendix.
4. \+ Extended discussion on the ablation study results in the Appendix.

We also welcome any further suggestions you may have, as we are also eager to make this rather dense work more reader friendly.

Reviewer comment (Experiments):
The experiments lack important baselines for comparison, such as instructzero [1].

Author response:
Your concern actually points out a critical unbalance of this field - there is much more effort in developing prompt optimization for LLMs compared with diffusion models. To the best of our knowledge, Promptist is the first and the only comparable method developed for and tested on the text-to-image diffusion model. This is also the reason why in our opinion, efforts such as Promptist and ours for the diffusion models might be valuable to the community.

Prompt optimization methods like APE [1] and InstructZero [2] are designed for LLM, and their common formulation makes them not directly extendable to diffusion models. Concretely, they mainly study discriminative tasks and assumes a labeled training dataset (demos) is provided [1, 2]. This is one of the reasons that these methods do not report experiments on diffusion models.


[1] Zhou et al. Large Language Models Are Human-Level Prompt Engineers. ICLR 2023

[2] Chen et al. InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models. 2023

Reviewer comment (Experiments):
The ablation study can be further improved with more discussions.

Author response:
- Reproducibility

We're planning to release all our code, since we view our method only as a starting point for future advancement. **For the rebuttal, we attached a draft version of the codebase to the supplementary material.** We are actively working on refactoring the codebase and are also looking into adding a demo on HuggingFace.

- Clarity

Thank you for your detailed suggestions! We’ve improved the presentation accordingly. The roadmap of the current manuscript is as follows: we first explain the framework in Section 4.1. This is followed by the search space design in Section 4.2 and the search algorithm for navigating that search space in Section 4.3. Within Section 4.3, we roughly follow the order of execution to lay out the complete algorithm, that is, we discuss how to learn a distribution over text space (GPO) first, then how to sample from it (evolutionary search).

**To improve the clarity of the paper, we made the following changes based on your suggestion:**

1. \+ The above roadmap to the beginning of Section 4.
2. \+ Discussions on Definition 4.1 in the Appendix
3. \+ Derivation for Remarks 1 and 2 in the Appendix.
4. \+ Extended discussion on the ablation study results in the Appendix.

We also welcome any further suggestions you may have, as we are also eager to make this rather dense work more reader friendly.