Paper: Subject-Diffusion: Open Domain Personalized Text-to-Image  Generation without Test-time Fine-tuning (ICLR-2024)

Reviewer comment (Reproducibility):
The reproducibility of the paper is questionable, given that the model training is relying on a large private dataset (at least the labels, masks, and grounding). While the dataset is being argued as one of this paper's main contribution, there is no discussion on quality of the data (not a lot examples in main paper or appendix), no discussion on how bias/privacy is protected, no plan or discussion on releasing the data.

Author response:
We have been releasing the data all along, but due to the strict open source process within the company, the period is long. However, ultimately, it is feasible for us to move towards open source. The entire dataset is almost 40T in size, so we are currently organizing our open source content. As the original images themselves from LAION are open source, we plan to organize the URLs of the original images and then open source the corresponding segmented images that we have processed along with structured information such as BLIP captions, masks, and boxes. 
In Appendix B.3 marked in red, we provide additional subjective quantitative statistics on data quality for reference.
Although we have not yet released the data, we have open-sourced our code. We are aware of one work, "High-fidelity Person-centric Subject-to-Image Synthesis," that has reproduced our paper and compared it with our method.

Reviewer comment (Presentation):
The image resolution provided in the paper is quite small (even counting the ones in the supplementary). Having high-resolution image generation is quite important to assess the quality of model in the field of image generation. Looking at figure 6 (when zooming in), I could find a lot of artifacts in the interpolated results, the face is losing details, the cat face is weird, the lion looks more like a horse to me. Similarly, figure 5 also seems to have a lot of artifacts from my perspective.

Author response:
The resolution of the generated images on paper is 512.  This is likely because the quality of the generated images is mainly determined by the parameters of the UNet model. For this part, we only trained the adapter and cross-attention K and V matrix parameters, so we speculate that this is probably an inherent issue with the base model SD2. 
Indeed, it is difficult to compare the base models without providing a metric such as FID or other measures to evaluate the generated images, as Subject-Diffusion does not support being used as a regular T2I model without additional reference image or other input conditions.We will continue to monitor this issue in the future.

Reviewer comment (Reproducibility):
During the inference on single-subject / multi-subject generation, do you provide those auxiliary information to the model?

Author response:
During inference, the mask plays two roles in both single-subject and multi-subject generations. 
The first role is to control the specific location of the subject to be generated in the final image. In our business application, this input can be randomly set, but with some degree of adherence to rules to control its freedom. For example, we may adjust the mask size and position based on the user's input reference image to better control the degree of freedom and meet the user's actual needs. 
The second role is to enhance the fidelity of the subject. Without the mask, the subject can still be generated, but the fidelity will be significantly decreased. According to our previous statistical results, the DINO metric reduces to a level similar to that of Textual Inversion.

Segmentation also plays the same role in both single-subject and multi-subject generations during inference. If the segmentation is not available, the generated results will be more heavily influenced by the background of the reference image. This is why we used SAM to remove the background effect during training. Both of these control conditions are consistent with the training process.

We are looking forward to your reply and happy to answer your further questions.

Best regards

Authors

Reviewer comment (Experiments):
If the mask / segmentations are available for the subject-diffusion only during the inference, what would be the performance if they are removed during the inference? Also how faithful is the model's generation to those control signal?

Author response:
During inference, the mask plays two roles in both single-subject and multi-subject generations. 
The first role is to control the specific location of the subject to be generated in the final image. In our business application, this input can be randomly set, but with some degree of adherence to rules to control its freedom. For example, we may adjust the mask size and position based on the user's input reference image to better control the degree of freedom and meet the user's actual needs. 
The second role is to enhance the fidelity of the subject. Without the mask, the subject can still be generated, but the fidelity will be significantly decreased. According to our previous statistical results, the DINO metric reduces to a level similar to that of Textual Inversion.

Segmentation also plays the same role in both single-subject and multi-subject generations during inference. If the segmentation is not available, the generated results will be more heavily influenced by the background of the reference image. This is why we used SAM to remove the background effect during training. Both of these control conditions are consistent with the training process.

We are looking forward to your reply and happy to answer your further questions.

Best regards

Authors

Reviewer comment (Experiments):
If the mask / segmentations are not available in inference, would the model's performance improve given those control signals?

Author response:
During inference, the mask plays two roles in both single-subject and multi-subject generations. 
The first role is to control the specific location of the subject to be generated in the final image. In our business application, this input can be randomly set, but with some degree of adherence to rules to control its freedom. For example, we may adjust the mask size and position based on the user's input reference image to better control the degree of freedom and meet the user's actual needs. 
The second role is to enhance the fidelity of the subject. Without the mask, the subject can still be generated, but the fidelity will be significantly decreased. According to our previous statistical results, the DINO metric reduces to a level similar to that of Textual Inversion.

Segmentation also plays the same role in both single-subject and multi-subject generations during inference. If the segmentation is not available, the generated results will be more heavily influenced by the background of the reference image. This is why we used SAM to remove the background effect during training. Both of these control conditions are consistent with the training process.

We are looking forward to your reply and happy to answer your further questions.

Best regards

Authors

Reviewer comment (Experiments):
The construction of the large-scale dataset might have biases or inconsistencies that could affect the model's performance.

Author response:
Thank you for your review. You are correct that the quality of the data can have a significant impact on the final results. We initially trained our models using the OpenImages dataset, we found that the performance metrics were lower  (but still outperforms ELITE and Blip-diffusion as shown in Table 1 and Table 3 ), especially for less common themes, such as certain artworks. This inspired us to construct large-scale structured data as thoroughly as possible to achieve better learning results. We used the largest open-source image-text matching dataset LAION, to ensure that as much learnable knowledge was included as possible.
Additionally, different screening criteria for the structured data, such as the shape, size, position, and number of entities in the bounding boxes and segmented images, can have a significant impact on the results. Our final screening principle was to maximize the quality of the data while ensuring that the data volume reached a certain scale. Of course, this required a significant amount of time and effort.
Finally, we are also currently working on organizing the data and making it available to the public in an open format to contribute to the development of personalized generation technology.

Reviewer comment (Evaluation):
The reviewer's primary concern is the actual experimental performance. In many results generated by Subject-Diffusion (e.g., Figure 1), the subjects lack diversity/identity variation. This suggests that Subject-Diffusion may have limited creative generation capability and generalizability. Some results are akin to image composition.

Author response:
It is true that our method lacks diversity/identity variation, and although we have made great efforts to balance diversity and fidelity in our strategy design, we have not yet achieved sufficient improvement in diversity. This is mainly due to the fact that during training, the individual samples and corresponding structured information are independent of each other. To improve diversity, we rely mainly on data augmentation techniques applied to the segmentation image, but this method is still weak. A more reasonable approach would be to retrieve and match segmented images from a database, then calculate the similarity between them to obtain condition inputs with certain attribute variations. We will actively try to implement such diversity-enhancing strategies in the future.

Furthermore, the use of a single-image input for inference without requiring finetuning has its limitations. Similar issues can be observed in models like FastComposer and SuTI that use multiple images without requiring finetuning. Balancing personalized image generation with diversity and fidelity still has a long way to go, and we will continue to work towards better solutions.

Reviewer comment (Experiments):
Apart from the quantitative results, presenting additional qualitative results of the ablation studies would strengthen this paper further.

Author response:
We provide additional qualitative results of the relationship studies, please refer to our Appendix F marked in red.

Reviewer comment (Presentation):
The presentation of this paper could be improved. Most images in the figures of the paper are too small, which requires careful zooming in to check details. Besides, the layout of some sections, especially the experiments, is a bit messy.

Author response:
Sorry, we overlooked this. We have corrected it. Thanks for the reminder.

Reviewer comment (Experiments):
About the proposed framework, what is the intuition of fixing the image encoder while training the entire text encoder?

Author response:
The input source of the image encoder for training the subject of the image is where the feature extractor comes into play, extracting relevant features. The motivation for training the entire text encoder is more direct, as its input introduces features of the image modality through certain rules of prompts. In order to better incorporate features of both modalities, it is necessary to shuffle the text encoder. However, we believe that this intuition requires more powerful experimental verification. In fact, we have verified that training only the last layer or two layers of blocks of the image encoder, or training only the first two layers of the text encoder, results in slightly worse actual experimental performance. However, because we did not think these ablation studies were the focus of the paper, we did not include them in the article.

Reviewer comment (Reproducibility):
Will the authors release the proposed SDD dataset? It is very useful if the dataset will be made publicly available. Also, the code should be made publicly available to ensure reproducibility.

Author response:
We have been implementing this plan all along, but due to the strict open source process within the company, the period is long. However, ultimately, it is feasible for us to move towards open source. The entire dataset is almost 40 T in size, so we are currently organizing our open source content. As the original images themselves are open source, we plan to organize the URLs of the original images and then open source the corresponding segmented images that we have processed along with structured information such as BLIP captions, masks, and boxes. Additionally, our code was publicly released four months ago and currently has 200+ stars.

Reviewer comment (Evaluation):
The limitation and failure case discussions are missing, which are highly recommended to be included.

Author response:
Apologies for this. We had initially included the limitations in our first version but later removed them to control the content of the main body. We will make sure to add the limitations back and make the necessary adjustments.
Regarding "failure case discussions," we will supplement them in Appendix G marked in red of the paper. Please refer to that section. Thank you for pointing it out.
For the shortcomings of "editing attributes" and "rendering harmonious images with two subjects," we provide an example where the attributes related to the failed images are highlighted in red prompts. As for generating images with two subjects, if the provided image itself is missing one or both subjects, it may result in disharmonious final images.

Reviewer comment (Presentation):
The last line of Page 1 has an Appendix reference error. This also applied to other places referring to the Appendix.

Author response:
Thank you for your reminder. The main text has been modified.

Reviewer comment (Experiments):
Regarding the quantitative results in Table 1, some results are borrowed from BLIP-Diffusion while others are tested. It would be useful if the authors could provide more details on the test settings to ensure a fair comparison.

Author response:
Thank you for your reminder. For inference, we utilize the PNDM scheduler for 50 denoising steps. We maintain a fixed text guidance scale of 3 and image guidance scale of 1.5 for all experiments conducted. These specifications will be highlighted in red in Appendix D marked in red of the paper.

Reviewer comment (Novelty):
While the authors have access to a substantial training dataset containing subject information (segmentation, text descriptions, and bounding boxes), it appears that the method may not introduce sufficiently novel or distinctive techniques compared to previous approaches. Instead, it seems to be a combination or integration of existing methods.

Author response:
Thank you for your review. We acknowledge that the paper focus more on engineering experiments and applications. Our primary motivation for this work is to alleviate the additional costs associated with fine-tuning that is typically required in current personalized generation methods. To achieve this goal, we have constructed a large-scale SSD dataset (which is currently being prepared for public release) and designed a unified framework that incorporates the strengths of many existing personalized generation methods. We would like to express our gratitude to the researchers who have worked on this topic before us. Moving forward, we will do our best to address these issues and aim to solve current personalized generation problems from a novel theoretical perspective.

Reviewer comment (Experiments):
The training mechanism leverages multiple concept information from a single image, which is indeed a notable feature of the proposed model. However, it doesn't provide a detailed explanation or evidence to support the claim that this single-reference image approach can consistently yield better results compared to fine-tuning methods with access to multiple subject images.

Author response:
In fact, compared to fine-tuning methods with access to multiple subject images, our method does have advantages in fidelity-related metrics for multiple subject image generation. However, the diversity of our generated outputs may be weaker, which is also one of the major weaknesses of our paper. This is due to the fact that our approach is designed to prioritize the fidelity of the subject, which inevitably sacrifices the variability in generated outputs, such as the ability to edit attributes, etc. Balancing these two abilities remains a challenging problem in personalized generation and is an area where we aim to improve. Of course, this situation is in line with single-subject generation.