Paper: Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach (ICLR-2024)

Reviewer comment (Novelty):
Encoder-based customized Text-to-Image generation is not new and missing comparisons with important previous works: [1][2][3][4]

Author response:
Q: Some related works are missing:

A: Actually, [3]  (E4T) is the most important baseline we have already discussed in our paper, our method obtained better results as presented in the paper. We will include [1] and [4], and add comparison. Although [2] is not an encoder-based method, we will add a discussion.

We will add more experiment results to illustrate the effectiveness of the Fusion Sampling in the Appendix soon.

There might be some misunderstanding, we would like to clarify more on the "encoder-based" method.
SuTI directly adopts the architecture from Re-Imagen [1], which is a retrieval augmented diffusion model. Different from E4T, ELITE or our method, it does not introduce an extra encoder network outside the diffusion model, but directly use the "encoder part" of the UNet. We did not call it encoder-based method, just like we would not call diffusion model itself as an encoder-based method.
Furthermore, SuTI focuses on apprenticeship learning so that the final model can imitates the behaviors of different experts, rather than focusing on design encoder mapping to map input image into embeddings like E4T do. We are happy to add SuTI as related work for discussion, but want to emphasize the difference between it and other works like E4T and ELITE.

[1]. Re-Imagen: Retrieval-Augmented Text-to-Image Generator. Wenhu Chen, Hexiang Hu, Chitwan Saharia, William W. Cohen.

Reviewer comment (Evaluation):
Figure 9 shows that before fine-tuning, fusion sampling even leads to worse identity-preserving performance than baseline sampling, implying that performance gain in identity similarity mainly comes from fine-tuning.

Author response:
Q: Question about identity preservation.

A: We want to emphasize that we never claimed the better fine-grained details can be obtained by a sampling method. What we claimed is that **details can be better preserved if the model is trained/fine-tuned without regularization**. However, **baseline sampling fails to generate desired images with respect to text** in this case. Figure 9 shows that Fusion Sampling can generate images aligned with text, under the setting without regularization, no matter before or after fine-tuning. Better details are certainly obtained by fine-tuning. However, without Fusion Sampling, we can not enjoy the setting without regularization which has the best detail preservation. 

That's why we propose to **fine-tune without regularization and perform Fusion Sampling after fine-tuning**. As we show in the paper, compared to related encoder-based method [3] which **fine-tunes the model with regularization and performs baseline sampling**, better results are obtained by our method.  

Furthermore, we are able to perform more flexible generation with Fusion Sampling: we can choose to emphasize either better creativity or better details, which is why the results of our method in Figure 5 and Figure 7 are represented by line instead of point like other methods. And our line is above all the points, indicating better results are obtained by our method.

Reviewer comment (Writing):
The authors aim at "detail preservation" but do not clearly explain what information these details include. What is the difference between detail preservation and identity preservation?

Author response:
Q: Question about identity preservation.

A: We want to emphasize that we never claimed the better fine-grained details can be obtained by a sampling method. What we claimed is that **details can be better preserved if the model is trained/fine-tuned without regularization**. However, **baseline sampling fails to generate desired images with respect to text** in this case. Figure 9 shows that Fusion Sampling can generate images aligned with text, under the setting without regularization, no matter before or after fine-tuning. Better details are certainly obtained by fine-tuning. However, without Fusion Sampling, we can not enjoy the setting without regularization which has the best detail preservation. 

That's why we propose to **fine-tune without regularization and perform Fusion Sampling after fine-tuning**. As we show in the paper, compared to related encoder-based method [3] which **fine-tunes the model with regularization and performs baseline sampling**, better results are obtained by our method.  

Furthermore, we are able to perform more flexible generation with Fusion Sampling: we can choose to emphasize either better creativity or better details, which is why the results of our method in Figure 5 and Figure 7 are represented by line instead of point like other methods. And our line is above all the points, indicating better results are obtained by our method.

Reviewer comment (Theory):
Can the authors elaborate more derivation details from Eq.10 to Eq.11?

Author response:
Equation 11 is derived from Equation 8 and Equation 10. In Equation 8, we obtain an inaccurate prediction for timestep $t-1$, which is denoted as $\widetilde{x_{t-1}}$. In equation 10, we inject the structure information in $\widetilde{x_{t-1}}$ into the current noised sample $x_t$ to obtain an updated sample $x_t$. 
As we discussed in the paper. Simply feeding the conditions independently into the model will lead to structure conflict. Thus we need to first obtain an “inaccurate” next-step sample, which is harmonized in terms of structure information . It is called “inaccurate” because which it is conditioned on $\gamma S^*$. Using this $\widetilde{x_{t-1}}$ as a prediction for $x_{t-1}$ leads to generation with some details lost, as shown in Figure 12. But this “inaccurate” sample  $\widetilde{x_{t-1}}$ does not have structure conflict anymore. Thus we propose to use its structure to infer a corresponding updated sample  $\widetilde{x_{t}}$ for current time step $t$, then perform sampling based on this  $\widetilde{x_{t}}$ instead of  $x_t$.

Reviewer comment (Experiments):
Comparisons with IP-Adapter[1]. IP-Adapter also projects the input image into the text embedding space, but requires no additional finetuning or well-designed sampling. The authors are encouraged to compare to this simple baseline.

Author response:
Thanks for the suggestion, we will add the discussion.

Reviewer comment (Theory):
Please clarify the intuition or motivation behind Line#9 in the fusion stage in the fusion sampling method.

Author response:
Q: About line#9 in the algorithm.

A: As we discussed in the paper. Simply feeding the conditions independently into the model will lead to structure conflict. Thus we need to first obtain an “inaccurate” next-step sample, which is harmonized in terms of structure information . It is called “inaccurate” because which it is conditioned on $\gamma S^*$. Using this $\widetilde{x_{t-1}}$  as a prediction for $x_{t-1}$ leads to generation with some details lost, as shown in Figure 12. But this “inaccurate” sample  $\widetilde{x_{t-1}}$  does not have structure conflict anymore. Thus in line #9 of the algorithm we propose to use its structure to infer a corresponding updated sample  $\widetilde{x_{t}}$  for current time step $t$, then perform sampling based on this  $\widetilde{x_t}$  instead of  $x_t$.

Reviewer comment (Experiments):
Please provide the corresponding ablation studies to support that using m=1 in practice works well for the fusion sampling method.

Author response:
Q: About m=1 in the algorithm.

A: In all the experiments reported in the paper, we use m=1, which can already lead to better results than other methods. We believe that it is enough to show that m=1 works well in practice. We add ablation study on m>1, in the revised Supplementary Material. From which we can see that increasing m does not lead to noticeable improvement. Considering the fact that increasing m will lead to longer sampling time, thus we believe using m=1 in practice is a better option.

Reviewer comment (Experiments):
Do the baseline methods all use the data augmentation method in the paper? If not, the comparison may be unfair.

Author response:
Q: Do all the baseline methods use data augmentation?

A: We didn’t use data augmentation in the pre-training stage, which is the same as other method. Data augmentation during fine-tuning is part of our proposed method. We will provide ablation study of comparing baseline sampling and Fusion Sampling, under different settings including with data augmentation, without data augmentation, and different level of regularization during fine-tuning.

Reviewer comment (Reproducibility):
It is better to report the exact training time, fine-tuning time, and the GPU devices used for reproduction consideration.

Author response:
Q: Implementation details.

A: We conduct all the experiments on Nvidia A100 GPUs. The pre-training time for encoder on CC3M dataset cost around 1 week on 8 A100 GPUs. The fine-tuning time on a single A100 GPU is around 25 seconds for 50 steps.

Reviewer comment (Experiments):
One may trade off the detail preservation for more creative generations or the other way around. Can the authors provide such trade-offs in the proposed framework?

Author response:
Q: Trade-off between creativity and detail preservation.

A: Good question, we provide some results in the revised Supplementary Material. We can easily control the generation, obtaining different level of creativity and detail preservation, by tuning a single hyper-parameter $\gamma$.

Reviewer comment (Experiments):
The motivation is that "regularization can result in detail lost". However, the validity of this claim depends on the degree of regularization applied, and the degree of detail loss suffered. In figure 3, what is the regularization weighting (i.e. lambda)? Have you tried an even smaller regularization weighting, and what is the degree of detail lost then?

Author response:
In experiments, we tried regularization $\lambda \Vert S \Vert^2$ with $\lambda$ selected from [1e-5, 1e-4, 1e-3, 1e-2, 1]. Figure 3 is presented to illustrate the influence of regularization on generation, small regularization like 1e-5 will result in failure in generating creative contents, while larger regularization will lead to identity loss. In customization, we are looking for a point where the model can preserve the details while having the capability to generate creative contents. However, simply tuning regularization hyper-parameter can not solve this problem.