Paper: Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models (ICLR-2024)

Reviewer comment (Experiments):
In the process of prompt synthesis using language model, the authors only ask the LLM to generate variations based on the object category and number of objects. Other important aspects such as spatial relationships and object attributes like material, color, size is not included. This greatly limit the scope of the experiment in this work. I also only saw sampled result images related to object category and object number in this paper. It would be great if more analysis on those other aspects can be studied as well.

Author response:
**[W1] Additional aspects of text-image alignment in prompt synthesis.**

We instruct the LLM to generate variations with an emphasis on the categories and quantities of objects, as they are common alignment issues with text-to-image models. However, we also utilize different sets of few-shot examples tailored to the type of input prompt to guide the LLM towards generating relevant variations. As an example, for the prompt “a red box on top of a blue box” in Table 14, the set of synthesized prompts includes variations such as “two blue balls on top of a pink box” and “a black triangle on top of a green box”, exploring alternative color attributes for the objects. We have revised the draft to add these details in Section 4.1.

Reviewer comment (Evaluation):
In figure 5, the difference of TextNorm and existing methods are not significant. BLIP2 has very similar results as the TextNorm on the correctness of the object number and categories.

Author response:
**[W2] Qualitatively comparable results.**

Note that Figures 1 and 4 contain examples that further highlight the shortcomings of the baseline models, including BLIP-2. We also added Figure 8 with an additional example that better showcases the limitations of BLIP-2 in the revised draft. Given the variability in performance across prompts, we remark that it is important to supplement the qualitative results with a human evaluation, as we have done in the work.

Reviewer comment (Experiments):
In the human evaluation, the authors are comparing with “the best-performing baseline reward model” but did not mention which one it is. Please provide the details on which reward function you are comparing with.

Author response:
**[Q1] Details on the baseline models in human eval.**

Thank you for pointing this out. To clarify, PickScore is used as the baseline for the composition and counting prompts, while ImageReward is the baseline for the comprehensive prompts. We have also revised our draft to include the details.

Reviewer comment (Evaluation):
One major contribution is the proposed text-image alignment assessment benchmark. However, the role/completeness of the benchmark are not fully explored. For example: Are the comprehensive/counting/composition enough to compare those reward models?

Author response:
**[W1] Are the prompts in the benchmark sufficient for comparing the reward models?**

We believe our benchmark offers a sufficiently diverse collection of prompts for evaluating reward models across various aspects of text-image alignment. We introduced separate composition and counting sets to specifically assess the ability of the reward models to evaluate compositional generation, a known challenge for text-to-image models, particularly when involving multiple objects [1]. To increase prompt diversity, we added the comprehensive set, which contains 100 prompts spanning nine fine-grained semantic categories: creative, spatial, text rendering, style, counting, color, composition, location, and open-ended. As a future direction, we aim to expand the scope of our benchmark by incorporating additional prompts, images, and human labels.

Reviewer comment (Evaluation):
In figure 3 (a), we can see ImageReward > PickScore > CLIP > BLIP-2 in terms of "comprehensive". Are these reward model rankings aligned with human evaluation (Do humans also think ImageReward > PickScore > CLIP > BLIP-2)?

Author response:
**[W2] Do the rankings in Figure 3 align with human assessments?**

Figure 3 indeed summarizes the overall alignment between the reward models and human assessments, as measured by the expected calibration error (ECE) computed with the human labels from the benchmark. Therefore, it is reasonable to conclude that a reward model for which more of the prompts fall within lower ECE ranges is overall more aligned with human preferences.

Reviewer comment (Evaluation):
Any failure cases for the proposed text norm method? (Failure cases for both comprehensive/counting/composition are helpful)

Author response:
**[Q2] Failure cases of TextNorm**

Our method employs a set of semantically contrastive prompts to enhance calibration by normalizing reward scores over these prompts. In case the set contains semantically equivalent or irrelevant prompts, normalization can result in worse calibration. Below are the results from an ablation experiment where we compare semantically contrastive prompts to random prompts in terms of calibration. This experiment underscores the importance of prompt set construction, highlighting that a poorly chosen set could yield worse results.

| Reward |  | Comprehensive |  |  | Counting |  |  | Composition |  |
|---|---|:---:|---|---|:---:|---|---|:---:|---|
|  | ECE | ACE | MCE | ECE | ACE | MCE | ECE | ACE | MCE |
| Random | 0.2777 | 0.2094 | 0.4986 | 0.3071 | 0.2323 | 0.5565 | 0.1758 | 0.1364 | 0.3883 |
| LLM | 0.2516 | 0.1809 | 0.4852 | 0.2213 | 0.1609 | 0.4635 | 0.1604 | 0.1146 | 0.3666 |

Reviewer comment (Evaluation):
The paper talks about overoptimization as the motivation, however doesn't directly evaluate for overoptimization. It instead evaluates for caliberation and human evaluation. To my understanding, current works get away with overoptimization issue with early stopping such as AlignProp (https://arxiv.org/abs/2310.03739) however this is not discussed in the paper. I would expect the right way to evaluate would be to not do early stopping and see the tradeoffs with the proposed solutions. Also talk about the benefits of not doing early stopping.

Author response:
**[W1] Evaluating overoptimization**

In our study, we consider human assessment of text-image alignment as the true objective and base our investigation on human annotations and evaluations. For our fine-tuning experiments, we evaluate the reward models against this true objective by conducting a human evaluation (the true objective) and report the results in Figure 6. The calibration analysis, which is also based on the human annotations from our benchmark, is for understanding the limitations of existing rewards and the associated risk of overoptimization.

Reviewer comment (Experiments):
Further AlignProp says that they only need to do early stopping for the Aesthetics reward function, however not for HPS v2. I think it would be worth comparing against the HPS v2 reward function and also discuss methods such as AlignProp or DDPO (https://arxiv.org/abs/2305.13301) and the tricks they use to prevent overoptimization. And why using those tricks might not be a good idea.

Author response:
**[W2] Connection to early stopping**

Thank you for your pointer. While early stopping is another viable approach (orthogonal to ours) to addressing overoptimization, it has several limitations. One challenge is determining the optimal stopping point, especially when evaluating the true objective is expensive. As you pointed out, the authors of AlignProp use early stopping in their experiments to guard against “loss of image-text alignment if optimized above a specific score” [1]. However, it is difficult to justify whether the 10th epoch is the optimal point at which to stop training, without conducting many human evaluations on multiple checkpoints. The authors of DDPO [2] recognize that overoptimization is an issue in text-to-image generation but do not discuss any specific methods for addressing it.

We believe that our method of enhancing reward model calibration through alignment with human assessment can be also integrated into early stopping to more accurately determine when to halt optimizing against proxy rewards. We have revised our draft to include a discussion comparing early stopping and our proposed method.

Reviewer comment (Theory):
Is there a fundamental reason why Softmax CLIP objective is better at caliberation than Sigmoid CLIP objective (https://arxiv.org/abs/2303.15343)?

Author response:
**[Q1] Softmax CLIP objective vs. sigmoid CLIP objective.**

While our proposed TextNorm (Eq. 1) has a notational similarity to the softmax-based CLIP objective, it is important to note that we are not claiming that the softmax-based objective is superior to the SigLIP objective [1] in terms of calibration. Our main focus is to evaluate the effectiveness of normalizing reward scores across contrastive prompts on improving calibration. The proposed method does not depend on the training objective of reward models. Nevertheless, we think evaluating SigLIP-based models is an interesting future question to explore.

Reviewer comment (Theory):
"Hence, we use the insight that the terms that contribute significantly to the denominator of the softmax score are the ones for which rφ(x, y) is sufficiently large." - It's unclear to me how using semantic-contrastive semantic prompts results in finding the terms that have high reward value.

Author response:
**[Q2] Use of semantically contrastive prompts.**

The underlying rationale behind our prompt set design, which employs semantically contrastive prompts that are syntactically similar, is based on the following hypothesis: prompts $x$ that differ from the input prompt $x_0$ in both syntax and semantics are unlikely to generate high reward values $r_{\phi}(x, y)$. Random prompts serve as an example of such $x$. As we show in our response to [Q3] in this letter, the prompt set using our method outperforms a random set of prompts, supporting the validity of the hypothesis. We have clarified this point in the revised draft.

Reviewer comment (Experiments):
Can the authors instead of finding contrastive prompts, take random prompts and show an ablation?

Author response:
**[Q3] An ablation using random prompts.**

Thank you for your suggestion. We conducted an additional experiment to compare semantically contrastive prompts to random prompts in terms of calibration. We used 20 randomly selected prompts from the comprehensive set to evaluate the counting and composition sets. To evaluate the comprehensive set itself, we excluded the input prompt and randomly sampled from the rest.

| Reward |  | Comprehensive |  |  | Counting |  |  | Composition |  |
|---|---|:---:|---|---|:---:|---|---|:---:|---|
|  | ECE | ACE | MCE | ECE | ACE | MCE | ECE | ACE | MCE |
| Random | 0.2777 | 0.2094 | 0.4986 | 0.3071 | 0.2323 | 0.5565 | 0.1758 | 0.1364 | 0.3883 |
| LLM | 0.2516 | 0.1809 | 0.4852 | 0.2213 | 0.1609 | 0.4635 | 0.1604 | 0.1146 | 0.3666 |

The experiment emphasizes the importance of using a relevant set of contrastive prompts and demonstrates that a random set of prompts can lead to inferior calibration. The results also underscore the effectiveness of the LLM-based approach to generating contrastive prompts. We have also revised the draft to add the results.