Paper: Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR-2024)

Reviewer comment (Experiments):
I would advise the authors to provide clear insights through experiments and offer some specific suggestions.

Author response:
In response to your advice for clearer insights and specific suggestions through experiments, we have introduced a new table (Table 1) in the revised manuscript. This table is aimed at providing a straightforward guide for readers in choosing suitable algorithms for various fine-tuning scenarios. For instance, based on our comprehensive experiments, we suggest that LoHa may be more effective for fine-tuning multiple "easier" concepts, whereas LoKr with full dimension appears better suited for targeting single "harder" concepts. This insight is reflected in the revised conclusion of our paper to provide more specific guidance.

In the meantime, we want to emphasize that while our summarized findings and suggestions provide a useful overview, they should be considered with caution. The complexity of fine-tuning text-to-image models, as discussed in detail in appendix C, means that a simplified summary could hardly capture the full picture. In this regard, our detailed text discussion offers a more nuanced explanation of the various ways in which a hyperparameter can influence the results, and is indispensable for a more comprehensive understanding of the fine-tuning process.

Reviewer comment (Experiments):
This paper mainly focus on LoRA-based finetuing strategies, can it be expanded to other parameter-efficient finetuning methods such as [1] and [2]? It doesn't provide a clear explanation.

Author response:
While our paper is primarily focused on LoRA-type methods, it is completely possible to include other methods in our library and experiments.

In fact, IA3 that we mentioned in the paper provides a different strategy via fine-tuning a multiplicative term, which is later generalized by orthogonal fine-tuning (OFT) [1] that you mention. Although we only got aware of this work after submission of the paper, we have already included it in the latest version of LyCORIS. This is also indicated in the revised version of the paper.

On the other hand, as far as we understand, DiffFit [2] is tailored to class-conditional models and can otherwise be regarded as a more restrained case of IA3 with several additional components, notably the bias and normalization layers, being fine-tuned. We can thus readily configure the training to get something that is really close with earlier version of our library.

Finally, the choice of the focus on LoRA, LoHa, LoKr, and native fine-tuning in the paper are due to their wider use in the community, but we are also considering conducting the same experiments for other methods, and notably OFT.

Reviewer comment (Presentation):
The conclusion about the performance of different finetuning methods is not clearly presented in the experimental section. Maybe some tables can more straightforwardly represent your final conclusions.

Author response:
Following your suggestion, we have added a new table (Table 1) in the revised version of our paper to demonstrate the influence of algorithm choice in the fine-tuning process. This addition should enhance clarity and accessibility for readers, providing a more direct understanding of our findings. Thank you for this valuable suggestion!

In addition to the table, the SHAP beeswarm charts readily included in our original paper serve as another effective tool for illustrating the influence of hyperparameters. However, it is crucial to note that many subtleties and context-specific insights discussed in our text cannot be fully captured by these visualizations alone. The readers should thus consider both the table and the charts in conjunction with the detailed textual explanations to gain a comprehensive understanding of the influence various factors on the fine-tuning process.

Reviewer comment (Novelty):
HuggingFace has also released the PEFT library, which supports a wider range of pre-trained models and includes the methods mentioned in the paper. Therefore, what are the advantages of the LyCORIS library compared to PEFT?

Author response:
We appreciate your question about the advantages of the LyCORIS library compared to HuggingFace's PEFT. PEFT indeed provides a wide collection of parameter-efficient fine-tuning methods. However, there are two key distinctions that set LyCORIS apart.
- LyCORIS is specifically designed to cater to users with less expertise in AI and Stable Diffusion, making it more accessible to a wider audience. In contrast, PEFT is tailored more towards expert users.
- In terms of fine-tuning of text-to-image models, LyCORIS covers more methods than PEFT. In fact, while PEFT initially supported LoRA, our library has incorporated other methods, with some of them, notably LoHa and LoKr, later got implemented by PEFT.

Reviewer comment (Novelty):
The paper conducted a multitude of experiments and comparisons on existing methods and various hyperparameters, leading to certain conclusions. Based on these findings, could there be a more optimal algorithm or design compared to previous ones?

Author response:
We conducted extensive experiments and comparisons on existing methods and various hyperparameters. While these studies have yielded important insights, we agree that asserting the existence of a universally "optimal" algorithm is challenging. The effectiveness of any given approach largely depends on specific datasets and tasks. For example, as we indicate in the conclusion of the revised paper, our experiments seem to suggest that LoHa would fit better the case of fine-tuning for multiple easy concepts while LoKr could be more suitable for the case of fine-tuning for a single difficult concept.
Besides this, our findings offer guidance on hyperparameter choices, while acknowledging the dynamic and context-dependent nature of performance in this field.

Reviewer comment (Experiments):
The results of this framework combined with ControlNet can be presented in this paper.

Author response:
We appreciate your suggestion in combining our framework with ControlNet. This is an interesting idea. StabilityAi indeed implemented LoRA for ControlNet during the writing of this paper [1]. However, their dimension is relative large (256 and 128) due to the heavy training workload and task drift. The introduction of other methods implemented in LyCORIS for ControlNet may then be helpful in this regard. While these have not yet been included in our library, we acknowledge their potential and are committed to exploring this integration in future work.


[1] https://huggingface.co/stabilityai/control-lora

Reviewer comment (Evaluation):
Efficiency (time and GPU memory cost) of different approaches are not provided and analyzed.

Author response:
To address this point, we have included a summary of the time and GPU memory costs of different configurations in Table 5 of the revised version of our paper. It is worth noticing that both the actual VRAM usage and training efficiency can vary greatly depending on the implementation and the used hardware. For example, we have modified the implementation of native training in LyCORIS after the experiments, and this should lead to faster training compared to what we report in the table.

Reviewer comment (Experiments):
A minor question: It will be better if the authors provide the results on other versions of stable diffusion, such as SD2.0 and SDXL.

Author response:
Our library has added the support for other versions of Stable Diffusion, such as SD2.0 and SDXL. In the paper, we choose to focus on SD 1.5 due to its wider use and larger user base in the community. Moreover, due to page limitations, we are constrained in the number of experiments we can include. Nevertheless, we believe the experiment based on SD 1.5 already provides a substantial assessment of our framework's capabilities.