Paper: FABRIC: Personalizing Diffusion Models with Iterative Feedback (ICLR-2024)

Reviewer comment (Evaluation):
Limited Expansion of Distribution: The method struggles to widen the distribution beyond the initial text-conditioned one provided by the model.

Author response:
It is true that FABRIC does not necessarily expand the conditional distribution of generated images, especially when the feedback images are sampled from the same model, but this is generally the case for interventions that improve quality. Indeed, in order to improve quality it is necessary to constrain the distribution by emphasizing desirable and cutting out undesirable parts. Still, it is possible to reintroduce some diversity. For example, by adding feedback images from an external corpus in addition to the ones from the current generation, one can add more diverse images, guiding the generative process in new directions. This can even be automated by retrieving images from the external corpus that are similar to the prompt or existing feedback images (e.g. using CLIP similarity). Additionally, by adding random alphanumeric characters to the prompt it is possible to artificially increase the diversity of the generations [1]. We have added clarifying comments about this and the following two points to Section 6 of the manuscript.

Reviewer comment (Theory):
Feedback Loop Limitation: Since the feedback originates from the model's output, it creates a cyclical limitation where the model might only reinforce its existing biases.

Author response:
Similar to the previous point, this is more a limitation of our experimental setup rather than a fundamental limitation of FABRIC. Indeed, it is possible to move the balance of the exploration-exploitation trade-off in the direction of the former with very simple extensions to FABRIC such as the addition of a retrieval corpus. More sophisticated methods might also take the prompt into consideration, by rephrasing the prompt for increased diversity or by iterating on the prompt and taking prior feedback into account, but we leave exploration of this issue to future work.

Reviewer comment (Evaluation):
Diversity Collapse: As the strength of the feedback and the number of feedback images increase, the diversity of the generated images tends to diminish. The images tend to converge towards a single mode that closely resembles the feedback images.

Author response:
We agree with the reviewer that in our experimental setup FABRIC clearly suffers from diversity collapse. However, we would like to point out that this is not necessarily the case in a general setup. In order to automate the process of evaluation, we were only giving images from the ones generated during previous rounds as feedback images. There, naturally (and hopefully), the process has to converge to a result very similar to the target image. In practice, the user can select feedback based on a variety of criteria, hence producing more diverse feedback images and (anecdotally) more diverse results. Additionally, we would like to emphasize that of all the methods we investigated, FABRIC was able to elicit the highest increase in quality metrics for the lowest decrease in diversity (see Section 4.3 on the quality-diversity trade-off), comparing favorably to both finetuning and LoRA training.

Reviewer comment (Evaluation):
Lack of Detailed Feedback: Users cannot specify which particular aspects of an image they appreciate or dislike. This restricts the model's ability to fine-tune its output based on detailed user preferences.

Author response:
Even though we did not evaluate it experimentally, it is possible to provide a textual description in order to steer the feature extraction process. Namely, we use the null prompt for extracting attention features from the feedback images in our experiments. However, one may use an arbitrary prompt for each feedback image, describing for instance the specific aspects they like/dislike. This would significantly complicate the experiment design, as one would have to find a way to automatically generate textual descriptions of the feedback. In addition, it again goes against the aim of reducing the amount of prompt engineering required to achieve desired results by just shifting the prompting to a different stage of the process. We also note that attention masking is another simple extension that could be used to constrain the feedback to certain parts of the image (similar to inpainting with Stable Diffusion). We have added a clarifying comment about this to Section 6.

Reviewer comment (Writing):
The weakness of the paper mainly lies in writing.

Author response:
We thank the reviewer for the excellent summary of our contributions and for the concise, constructive feedback. We agree that certain sections of the writing could be improved and have uploaded a revised version of the paper. In particular, the methods section has been majorly overhauled, hopefully addressing your concerns. We refer to the general comment for a more detailed description of the changes that have been made.

Reviewer comment (Presentation):
It is better to incorporate more method descriptions, including model design and formulations in the main script instead of the appendix.

Author response:
We thank the reviewer for the excellent summary of our contributions and for the concise, constructive feedback. We agree that certain sections of the writing could be improved and have uploaded a revised version of the paper. In particular, the methods section has been majorly overhauled, hopefully addressing your concerns. We refer to the general comment for a more detailed description of the changes that have been made.

Reviewer comment (Writing):
I'd like to accept this paper if the writing problem is addressed.

Author response:
We thank the reviewer for the excellent summary of our contributions and for the concise, constructive feedback. We agree that certain sections of the writing could be improved and have uploaded a revised version of the paper. In particular, the methods section has been majorly overhauled, hopefully addressing your concerns. We refer to the general comment for a more detailed description of the changes that have been made.

Reviewer comment (Novelty):
**Limited technical novelty**: While the proposed method is effective in incorporating user feedback, the extension to enabling 'iterative feedback' is rather naive, and the feedback is constrained to binary labels (which the author(s) have acknowledged as a limitation). It would be more interesting to explore more advanced way of users' feedback across multiple rounds, and incorporating other modalities, such as text explanations beyond binary preferences.

Author response:
While we agree that the core conditioning mechanism isn’t novel and has been proposed by the authors of ControlNet [1], the combination with weighted attention and CFG makes for a very versatile and flexible algorithm even beyond the scenarios that were evaluated experimentally. For example, real-valued ratings can be incorporated by varying the feedback strength according to the magnitude of the rating and using it as positive/negative feedback depending on the sign. Textual descriptions as well as masking of certain parts of the image can also be supported by varying the prompt used for extracting attention features or by masking out certain keys and values.
  
However, both of these extensions go against the core motivation, which is to alleviate the need for prompt engineering by making the best possible use of sparse binary feedback. In addition, it would have further complicated experiment design, making the results harder to reproduce and more difficult for future work to compare against. This was somewhat unclear from the writing and we have added clarifying comments to Section 6.

Reviewer comment (Experiments):
**Lack of human rating in a paper focused on iterative human feedback**: While the author(s) have used reasonable proxy to evaluate the effectiveness of the model in following human preferences, it would strengthen the paper if the author(s) can include some form of user study, given this papers' focus is in incorporating human feedback in the image generation process.

Author response:
We agree with the reviewer that studying human interaction with the system would certainly be insightful. Doing a user study was considered, but we ultimately decided against it due to the challenges involved with the design and execution of such a study. To illustrate: A naive study design would simply let the user try out the system and subsequently have them rate their satisfaction and the perceived quality of generated images. This, however, leaves many variables uncontrolled. To name a few: How much of the perceived quality comes from the base model as opposed to the improvements made by FABRIC? How much does generation time (which is noticeably higher for FABRIC) impact user satisfaction and perceived quality? How much does the user interface influence the results? How big is the learning effect from using the system and how much skill is involved in achieving desirable results?
  
As a consequence, if not executed properly, the results of such a study would amount to little more than anecdotal evidence (which we now have added to Section 4, Experiments) and might prove difficult to reproduce due to the number of variables that can impact the final rating. Considering all of this, rather than drawing conclusions from potentially weak evidence, we believe that it is appropriate to leave the empirical analysis of human interaction with FABRIC for future work.

Reviewer comment (Novelty):
**Missing discussion to some prior work**: I believe the proposed method has some technical similarity to prompt-based image editing methods, such as instruct-pix2pix [1] and prompt2prompt. [2] While the proposed method is different in the types of feedback and preference investigated, it would be great if the author(s) can systematically compare and survey related techniques that use attention map for feedback and/or image editing. I also have some doubts about whether it is reasonable to claim that the method "outperformed" supervised-learning baselines (HPS), see question below.

Author response:
These two papers, prompt2prompt and instruct-pix2pix, are indeed related to our work, as they use similar techniques but have different goals. We thank the reviewer for pointing this out and have added a paragraph on image editing to Section 5 (Related Work).

Reviewer comment (Evaluation):
While the paper claims to outperform a supervised-learning baseline (HPS LoRA), it is unclear to me how does HPS relate to PickScore, as they both appear to measure human preference. Would the author(s) please clarify how might they relate to each other? As the models are evaluated on PickScore but LoRA-tuned on HPS.

Author response:
The reviewer is correct that HPS and PickScore are very similar and solve the same task, which is human preference estimation. In fact, in the early phases of the project, we were using HPS as the main evaluation metric but decided to replace it with PickScore when that was published since it demonstrated superior accuracy thanks to a larger training set (see the table below), which makes for a more accurate human proxy. We continue using the HPS LoRA as a baseline model explicitly trained to maximize human preference, which is ultimately also what PickScore measures. We note that FABRIC outperforms HPS LoRA whether measured by HPS or PickScore (early experiments used HPS).
| Method | #prompts | #choices | LoRA of SD available |
|---|---|---|---|
| HPS | 25k | 25k | yes |
| PickScore | 38k | 584k | no |

Reviewer comment (Novelty):
How does the method relate to/differ from prompt2prompt and instruct-pix2pix? As stated above, it would be helpful to systematically compare them (and other related prior work) in a table.

Author response:
We like the idea of adding a systematic comparison of methods which incorporate human feedback in the generation process of diffusion models, even beyond just the techniques using attention-injection, but unfortunately we couldn’t find space to include it in the paper. Instead, we’ll just attach it here and possibly add it to the Appendix if it is considered a valuable addition. The table is provided in the next comment due to the character limit.