Paper: A Data Perspective on Enhanced Identity Preservation for Diffusion Personalization (ICLR-2024)

Reviewer comment (Experiments):
The formatted prompt generation is limited to several categories. For example, according to the supp., for live objects, the prompts are all obtained via the subject of animal. How about the prompts for human? I do not think this prompt generation strategy is general enough.

Author response:
We use object/animal as categories, which covers most kinds of living and non-living objects. For a particular application domain, one could always generate more specific prompts with minor modifications. It would only take a few seconds by using our LLM strategy. We add an example on humans and show the results in Figure 15 (updated manuscript) along with the comparison with other methods. It demonstrates that the same prompt generation works for humans and that our approach preserves better identity than other methods while still aligning with the text prompts.

Reviewer comment (Experiments):
Moreover, I think the prompts should be generated according to different input images, employing multi-modality models.

Author response:
This is a promising direction that however bears its own challenges. We did an initial experiment on the “​​tortoise plushie” in Figure 14 (updated manuscript). We use the multi-modal model BLIP to generate the captions and then use ChatGPT to reformulate the sentence. Subsequently, we use an LLM to convert these into the correct format with the special <new> token inserted. We show the details in Appendix E (updated manuscript). This alleviates writing the prompt examples manually, i.e., it replaces the manual steps in Section 3. As shown in Figure 14, the results are as good as the ones with the manually created prompts and outperform other methods.

Reviewer comment (Experiments):
I think the new objects to be inserted into the generation in this paper’s experiments are few. More cases are needed to analyze the effects of the proposed method.

Author response:
We already have a range of qualitative results in Figure 9-12 (numbers referring to the updated manuscript). We added some more results for comparison in Figure 13-15. In addition, we evaluated on the DreamBench testsuite. We added this comparison to the supplementary materials. Due to the space limit (<100M), we only sample each prompt once for each subject, resulting in 750 comparison images.

Reviewer comment (Evaluation):
In Fig.4, why the performance of w/o format will lead to worse results as the increase of iteration number? Even without the use of formatted prompts, the generation results should be more fitted with the target object along with the training.

Author response:
It is a consistent finding as we also observed this before and analyzed in Sec 4.2 first paragraph “How important is the formatted prompt? …”. We additionally noticed that in the first 2,000 iterations, the simplistic regularization set with prompt “a [subject]” effectively prevents overfitting. However, it fails to do so with prolonged training and thus cannot preserve the fine details of the subjects. We believe it is due to the deviation between the simplistic prompt “a [subject]” and the complexity of the images. Such a simplistic prompt is not detailed enough to explain the image. Therefore, with longer training, the model degenerates. Interestingly, even without the regularization set, if the prompts of the training example are accurate instead of simplistic, the model overfits in terms of limited diversity but without degeneration, as shown in Figure 4 (w/o reg dataset).

Reviewer comment (Experiments):
I wonder the performance of the proposed method if there are fewer input examples, like 1-3 examples.

Author response:
We added the ablation tests, see common question #1.

Reviewer comment (Experiments):
Can the prompts be generated online with the training? It will save a lot of time.

Author response:
This is a promising direction that however bears its own challenges. We did an initial experiment on the “​​tortoise plushie” in Figure 14 (updated manuscript). We use the multi-modal model BLIP to generate the captions and then use ChatGPT to reformulate the sentence. Subsequently, we use an LLM to convert these into the correct format with the special <new> token inserted. We show the details in Appendix E (updated manuscript). This alleviates writing the prompt examples manually, i.e., it replaces the manual steps in Section 3. As shown in Figure 14, the results are as good as the ones with the manually created prompts and outperform other methods.

Reviewer comment (Evaluation):
Few examples can not reflect the true quality of the generation, is there any subjective evaluation?

Author response:
We added one, see common question #3.

Reviewer comment (Experiments):
On top of Custom Diffusion [3], it would be also interesting to see whether the data-driven approach can benefit multi-concept learning.

Author response:
It is indeed an interesting topic we would like to explore in the future. For this paper, we focus on the detailed subject alignment, i.e., identity preservation. We added this to our conclusion.

Reviewer comment (Experiments):
The method requires generating a relatively large regularization dataset (containing ~2000 images), which inevitably leads to much longer training time.

Author response:
We kindly refer you to the question #1 in the common response. A smaller regularization set (<500) also works.

Reviewer comment (Writing):
TYPO in section 5, the first bullet in the first paragraph should be “1)” instead of “2)”.

Author response:
Fixed, thx.

Reviewer comment (Presentation):
Poor presentation: I find this paper hard to follow. Several important aspects of the proposed method remain a mystery, e.g., how the 2000 "regularization images" are created, why the 2000 images are called regularization images (I am not able to see how they can regularize the model from "Each training batch contains one example from training set and one example from regularization set."). It seems to me that this paper tries to extend DreamBooth [Ruiz, 2023a] and the 2000 regularization images may be generated using Stable Diffusion and the 2000 samples may be used to compute the class-specific prior preservation loss to regularize the model. However, a reader needs to be very familiar with DreamBooth in order to make these guesses and they are just guesses.

Author response:
We sincerely thank you for pointing out the points where writing can be improved. To make it easier for readers, we created a separate subsection at the beginning of our method section (Section 3), to introduce DreamBooth in the updated manuscript.

Reviewer comment (Novelty):
To me, this is a trivial extension of DreamBooth. The authors proposed to generate more "regularization images" using prompts that are (1) generated by a large language model (LLM) or (2) generated following specific templates. It is hard for me to agree that this paper meets the bar for an ICLR paper.

Author response:
DreamBooth also generates a dataset, which is called a prior set. Their purpose is to preserve the ability to generate similar subjects so that the model does not overfit. While they can train 2000 iterations without overfitting, the model still overfits and gets degenerated with longer training, as shown in Figure 4. Our insight is that their simplistic prompt “a [subject]” and its generated images are not powerful enough to prevent overfitting. Furthermore, the simplistic prompt “a [subject]” may not be enough to explain the diverse generated images. The deviation between the prompts and the images may result in model degeneration with prolonged training, as shown in Figure 4 (w/o format). Interestingly, even without the regularization set, if the prompts of the training example are accurate instead of being simplistic, the model only overfits without breaking down entirely, as shown in Figure 4 (w/o reg dataset). Our paper focuses on the importance of accurate prompts and their similar prompts while DreamBooth focuses on the importance of images and their similar images. Our prompts prevent the model from degeneration. They work as hard examples, forcing the model to relate the details with the special token. Instead of being a trivial extension to DreamBooth, our approach provides a new perspective to the problem, pushing the direction taken by DreamBoth further and deviating from the current trend to hand-tune model architectures. With the significant improvement we show, we expect a range of future work to follow in this underexplored direction.

Reviewer comment (Evaluation):
Lack of human evaluation.

Author response:
We added one, see common question #3.

Reviewer comment (Reproducibility):
How are the 2000 regularization images created? Are they generated by a pre-trained diffusion model? If so, why do you use the word "created"?

Author response:
While our main approach is stated in Section 3, the details of how to generate the prompts and the corresponding images are in supplementary A. To make it easier to find, we now concatenated it with the original paper. As stated in Section 4, the 2000 regularization images are generated from Stable Diffusion XL with structured but diverse prompts. We use the word “create” because the regularization set contains not only the images but also the created prompts.

Reviewer comment (Writing):
Why the 2000 images are called "regularization images"? How can they help regularize the model? If you follow DreamBooth [Ruiz, 2023a], please specifically mention this.

Author response:
While our main approach is stated in Section 3, the details of how to generate the prompts and the corresponding images are in supplementary A. To make it easier to find, we now concatenated it with the original paper. As stated in Section 4, the 2000 regularization images are generated from Stable Diffusion XL with structured but diverse prompts. We use the word “create” because the regularization set contains not only the images but also the created prompts.

Reviewer comment (Experiments):
Would be great to see a comparison with DreamBooth + LoRA and Textual Inversion + LoRA.

Author response:
Investigating different model architectures and fine-tuning strategies is orthogonal to our goal of improving the data. To show that our method outperforms also the more recently developed models, we compare our method with the recent CustomDiffusion model. The results are illustrated in Figure 13-15 (updated manuscript). Our generated images have higher subject alignment than theirs while having better or equal text alignment. According to our user study in comparison to DreamBooth, our method is preferred in subject alignment by 64.9%. An indirect human comparison is possible to CustomDiffusion which, in an equivalent user study, scored only 56.62% compared to DeamBooth. Due to the time limit, we will add direct comparison with CustomDiffusion in the camera-ready version.

Reviewer comment (Experiments):
How does the performance of the proposed method change when more number of samples are available, e.g., 20 samples? How does the performance of the proposed method compare with other methods when more number of samples are available?

Author response:
We show in Figure 14-15, examples using theCustomConcept101(https://github.com/adobe-research/custom-diffusion/tree/main/customconcept101) dataset. With more than 10 examples, our method still outperforms other methods, especially in the subject alignment.

Reviewer comment (Experiments):
Continuing from Weakness 1, it is unclear to me how the authors choose the format described in Section 3 “Generating Against Training Prompts” and “Amplifying Diversity with Structured Prompts” since there is no related literature or ablation study to justify the effectiveness of each component in the format.

Author response:
## W2: Justification of the proposed prompt format? 
In Section 4.2, we already ablate the importance of the prompt format. Without our format, the model tends to overfit to the training examples. Our format “a [adjective] [subject] [in some background]” is a simple and straight extension of the commonly used prompt “a [subject]”. We believe this simple yet effective method could inspire future researchers to advance this direction further.

Reviewer comment (Experiments):
The additional time required for generating the regularization set is more substantial (2000 images for this setting v.s. < 1000 images for the original DreamBooth setting).

Author response:
## W3: Requires too many images for regularization?
We kindly refer you to the common question #1 in the common response. According to our ablation tests, it only needs < 500 images in practice for simple cases, such as dogs and backpacks.

Reviewer comment (Experiments):
Does the size of the regularization set affect the performance? (e.g. will a smaller regularization set also work? Can the authors provide more ablation studies on this?)

Author response:
## Q: Does the size of the regularization set affect the performance?
We kindly refer you to the question #1 in the common response. A smaller regularization set (<500) also works.