Paper: ViCo: Plug-and-play Visual Condition for Personalized Text-to-image Generation (ICLR-2024)

Reviewer comment (Evaluation):
ViCo may have lower performance compared to fine-tuned methods, implying an ease-of-use vs. performance trade-off.

Author response:
Our model offers a **plug-and-play module** capable of markedly enhancing the performance of Textual Inversion, achieving results comparable to DreamBooth. It is also *feasible* to combine ViCo with other methods, such as OFT [1], which could potentially lead to further performance improvements, as demonstrated in the added experiment presented **in Figure 22 in Appendix I.** While it is technically feasible to combine ViCo with DreamBooth, we do not recommend this approach as it may compromise the flexible deployment property inherent to ViCo.

Reviewer comment (Evaluation):
The use of Otsu thresholding for mask binarization may slightly increase time overhead during training and inference for each sampling step. However, this is offset by shorter training time and a negligible increase in inference time.

Author response:
We can also use other *non-iterative* binarization methods to produce the mask, which can decrease the overhead to nearly zero. We choose otsu because it is robust to all types of data and consistently produces good binary masks. As the reviewer recognizes, the overhead is offset by **shorter training time** and a *negligible* increase in inference time.

Reviewer comment (Reproducibility):
Can you explain in more detail how ViCo generates object masks and incorporates them into the denoising process?

Author response:
We would like to explain more details as follows:

- **Mask generation** takes place in the computation of each text cross-attention layer between the latent feature of the reference image and the text prompt. The cross-attention map corresponding to S*, which provides a reliable indicator of object localization, is **binarized** into an object mask, sharing the same shape as the latent shape in the current layer.

- **Mask incorporation** takes place in the processing of each image cross-attention between the latent feature of the source image and the reference image. When computing $Attn = QK^T/\sqrt{d_k}$, we **mask** those elements corresponding to the masked regions of the reference image in $Attn$. In this way, the attention activations related to non-object regions in the reference image can be **suppressed**.

Reviewer comment (Evaluation):
How does ViCo compare to other methods in terms of computational efficiency and parameter requirements?

Author response:
Regarding **computational efficiency**, we showed the comparison of the **time cost** in **Table 3 in Section 4.1** and report the **maximum GPU memory** used during training for all compared models:

|                              | Max. GPU Mem. |
|------------------------------|:--------:|
| DreamBooth (optimized)       |   14.9G  |
| Custom Diffusion |   18.3G  |
| Textual Inversion            |   21.6G  |
| Ours                         |   23.4G  |

Note that we employ the *optimized* implementation of DreamBooth using mixed precision and 8bit Adam, to accommodate our GPUs. Without these optimizations, the training process would require over 30GB of GPU memory. During inference, all methods use approximately 19GB of GPU memory.

Regarding **parameter requirements**, we reported the comparison of model parameters in **Table 1 in Section 2**.

Reviewer comment (Evaluation):
Can you clarify the limitations mentioned and how the trade-off between keeping the frozen diffusion model and not fine-tuning it affects performance?

Author response:
Finetuning the diffusion model may affect the performance in two aspects. *(1)* First, finetuning diffusion models allows more parameters to learn the object, better preserving the object details. *(2)* Second, solely training the S* token can render it dominate the text space and weaken the influence of text-related information. These effects can be demonstrated in the comparison of image similarity and text similarity metrics between Textual Inversion and DreamBooth. Based on textual inversion, our method uses a **lightweight module** to overcome the above defects but there may still be room for performance improvement.

Not finetuning the diffusion model offers the advantage of **flexible transferrable deployment**. This allows for only saving a set of *small* parameters for each object without the need to save the entire diffusion model. It also facilitates *easy* transfer to other diffusion models without the requirement of re-tuning the whole model. Additionally, this approach addresses the *language drift* issue associated with finetuning the entire model as described in DreamBooth, because all established language knowledge is retained in the frozen pretrained diffusion model.

Reviewer comment (Novelty):
How does ViCo's cross-attention method for capturing object-specific semantics differ from others, and what are its advantages?

Author response:
Compared to *Textual inversion*, ViCo uses the visual condition module to incorporate **explicit visual information** that captures more object semantic details from the reference image. Our advantage is that the visual condition module can provide more information on object details than using a single token.

Compared to *DreamBooth* and *Custom Diffusion*, they fine-tune the U-Net to directly learn object semantics back in the backbone model while we introduce a **plug-and-play module** to explicitly extract visual semantics and inject it into the frozen denoising process. Our advantage lies in **flexible deployment** and avoiding the *language drift* issue in DreamBooth, by keeping the diffusion model frozen.

Reviewer comment (Evaluation):
In Table 4, the improvement introduced by masking is not so significant.

Author response:
We would like to bring to the attention of the reviewers that the quantitative comparison might not clearly demonstrate the object refinement achieved by the masking strategy, as it refines more visual details beyond just the texture characteristics of the object. These additional refinements are challenging to distinguish significantly through evaluation by visual encoders.

However, it is important to note that our proposed image cross-attention already possesses object capture capabilities during training, exhibiting high quantitative performance. The masking strategy additionally provides an explicit method to filter out the background of the reference image seamlessly in our framework. This rectifies certain cases where the generation without the mask may be unsatisfactory. For specific examples, we encourage the reviewer to refer to **Figure 5 (b)**. In cases where a mask is absent, the generation may solely capture the shape and texture of the object, neglecting important shape constraints. Introducing an automatic mask overcomes this issue and further enhances the object's identity in the generated images.

Reviewer comment (Experiments):
The method is incapable of using multiple reference images during inference, for more robust generation.

Author response:
We thank the reviewer for bringing up the point of using multiple reference images during inference. We would like to clarify that our framework indeed **supports an arbitrary number of reference images (one or multiple)** without any change to the architecture.
This capability is achieved by **concatenating tokens of multiple images** and passing them through the existing image cross-attention. 
The flexibility of our framework lies in the fact that the number of input tokens in the image cross-attention can be *variable*, accommodating any number of tokens from the concatenated reference images. This attribute allows us to **flexibly use an arbitrary number of reference images**.

We have conducted a comparison between using two reference images and using a single reference image for generation, as shown in **Figure 12 in Appendix C**. Our findings demonstrate that the results obtained using two reference images are similar to those achieved when using a single reference image. This observation indicates the *robustness* of our visual condition module to variations in the reference image. We appreciate the reviewer's recognition of this important characteristic of our framework.

Reviewer comment (Experiments):
Apparently, the method only works with images that have a single reference primary object.

Author response:
This is a **common limitation** for the majority of existing works, e.g., Textual Inversion, DreamBooth, and Custom Diffusion. Current research on personalized text-to-image generation mainly focuses on a single object. Learning from an image that has multiple objects is still an open problem for the community. 
Currently, the only work exploring this challenging problem is [1] which heavily relies on predefined object masks. Our method can also be adapted to this problem by manually providing object masks as done in [1]. We are indeed studying a more generalized case, in which no object masks are given, in our current research.

Reviewer comment (Experiments):
Do you use the same reference image for different model variations in the evaluation (especially T4)?

Author response:
Yes. For all experiments in the paper, we use **the same reference image** for each dataset. 
We have further clarified this in the revised paper.