Paper: DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation (ICLR-2024)

Reviewer comment (Experiments):
Why ELITE is not included in quantitative comparison?

Author response:
1. **Reference and comparisons(W1/Q1)**:

    + We have added all the mentioned references in the related work in our revised version. Thanks for making our related work more complete.
    + We have also compared the mentioned baselines together with the baselines suggested by Reviewer UqfN in Appendix A.5 in our revised version. Both the qualitative(Table 7) and quantitative results(Figure 13) show the superiority compared to the baselines. For easier reading, we also present the quantitative results in the following table, which shows that our proposed DisenBooth has the best image fidelity and text fidelity. 
    + Note that E4T[1] is a pretrained method and its open-source version is only for human subjects, which faces severe out-of-domain problem and cannot achieve satisfying performance on DreamBench. This phenomenon also indicates that although there are some non-finetuning methods for fast customization, effective finetuning to adapt to new out-of-domain concepts is still important. Although we tried to compare with [2] [3] in the suggested references, unfortunately we found these two works are two pretrained non-finetuning methods by Google, and there are no open-source codes and checkpoints for the pretrained models. We cannot compare with them because we have no access to the pretrained models. 

        |        | Custom | SVDiff | Break-A-Scene |   E4T  |  ELITE | DisenBooth(ours) |
        |--------|:------:|:------:|:-------------:|:------:|:------:|:----------:|
        |  DINO  | **0.675**  | 0.668  |        0.639  | 0.241  | 0.606  |   **0.675**    |
        | CLIP-T | 0.314  | 0.318  |        0.313  | 0.262  | 0.291  |   **0.330**    |  
    + Following the reviewer's advice, we have compared with ELITE quantitatively in our revised version and also in the table above. In our first version, we just want to compare our method with the finetuning-based method for fair comparison consideration, while ELITE is a pretrained and non-finetuning one.

Reviewer comment (Experiments):
InstructPix2Pix uses instructions to guide image editing. I wonder how it is used for comparison in this paper.

Author response:
3. **How InstructPix2Pix is used for comparison(Q2)**:

    We use an example to show how we use InstructPix2Pix for subject-driven text-to-image generation as follows. Assuming that we want to customize the backpack and we have an input image of it, we want to generate an image of the backpack under the Eiffel Tower. What we do in the experiment is send the input image to InstructPix2Pix, and then use the instruction ``put the backpack under the Eiffel Tower'' to generate an image. In the DreamBench setting, the backpack has 6 input images, we will repeat the generation process for each input image and calculate the average DINO and CLIP-T score of all the generated images.

Reviewer comment (Presentation):
A small problem with the structure. Since the finetuning objective consists of several separate losses and two of them are novel ones proposed for disentanglement, the ablation of the disentangled objectives is what readers are most curious about. I think the authors should consider moving Appendix A.1 to the paper section 5.3 if possible.

Author response:
4. **Paper Structure(Q3)**:

    We have tried to move Appendix A.1 to section 5.3, but we find it will exceed the page limit of ICLR. To emphasize this important part, in our revised version, we give the detailed cross-reference at the last of section 5.3 to make the readers easily find this ablation. Thanks for your suggestions.

Reviewer comment (Writing):
Typo: Figure 5 caption line 1. "The identity-irrelevant images are generated using the text prompt Ps." should be "The identity-relevant images"

Author response:
5. **Typo(Q4)**:

    We have fixed this typo. Thanks for your careful reading and pointing it out.

Reviewer comment (Experiments):
The proposed method might rely on multiple images to learn the identity-relevant and irrelevant embeddings. When there is only one single image for training, it may be challenging to disentangle them. Although the paper includes experiments on a single image in Section A.3, there is a lack of visual results to verify the effectiveness (like Figure 5), and the number of testing samples is also limited (only 3).

Author response:
2. **More experiments for one single image finetuning(W2)**:

    Following the reviewer's advice, we have conducted experiments of tuning with only 1 image on more subjects (1/3 of the DreamBench dataset), and we give both quantitative results(Table 5) and qualitative visual results(Figure 11) in Appendix A.3 in our revised version. The results show that our proposed method can still disentangle under this 1-shot challenging scenario. For easier reading, we give the average quantitative results in the following table, which shows the superiority over existing methods. 

    |        | Pix2Pix |   TI   | DreamBooth | DisenBooth(ours) |
    |--------|:-------:|:------:|:----------:|:----------:|
    |  DINO  |  0.613  | 0.538  |   0.610    |   **0.617**    |
    | CLIP-T |  0.302  | 0.288  |   0.297    |   **0.321**    |

Thanks for your suggestions, and we believe the revised contents further support our work and improve the quality of our paper.

Reviewer comment (Experiments):
More recent methods should be compared, including Break-a-scene[1], Custom Diffusion[2], and SVDiff[3].

Author response:
1. **Comparison with more methods(W1/W3)**:

    Following the reviewer's advice, we compare DisenBooth to the mentioned baselines together with the baselines suggested by Reviewer QDoi in Appendix A.5 in our revised version, where the reviewer can see both quantitative (Table 7) and qualitative comparisons (Figure 13) on the DreamBench. Both the qualitative and quantitative results show the superiority compared to the baselines. 
    
    For easier reading, we also present the quantitative results in the following table, which shows that our proposed DisenBooth has the best image fidelity and text fidelity. 
    Note that E4T is a pretrained method and its open-source version is only for human subjects, which faces severe out-of-domain problem and cannot achieve satisfying performance on DreamBench. This phenomenon also indicates that although there are some non-finetuning methods for fast customization, effective finetuning to adapt to new out-of-domain concepts is still important.

    |        | Custom | SVDiff | Break-A-Scene |   E4T  |  ELITE | DisenBooth(ours) |
    |--------|:------:|:------:|:-------------:|:------:|:------:|:----------:|
    |  DINO  | **0.675**  | 0.668  |        0.639  | 0.241  | 0.606  |   **0.675**    |
    | CLIP-T | 0.314  | 0.318  |        0.313  | 0.262  | 0.291  |   **0.330**    |

Reviewer comment (Experiments):
The paper employs a LoRA optimization strategy for fine-tuning the diffusion model, but it doesn't address how this choice impacts the quality of their approach compared to fine-tuning the entire UNet. It raises questions about whether the proposed loss functions and encoding strategies would still be effective in such a scenario and what impact this might have on result fidelity.

Author response:
**Lora Finetuning v.s. Full Finetuning(W1/Q1)**:

    Following the reviewer's suggestions, we have conducted experiments on finetuning the U-Net on 10 subjects and compared it with the results of Lora finetuning. The experimental results are provided in the following Table. DisenBooth-full in the Table means we finetune the whole U-Net in DisenBooth.

    |        | DreamBooth | DisenBooth-full | DisenBooth(ours) | 
    |--------|:----------:|:---------------:|:----------:|
    |  DINO  |   **0.671**    |      0.664      |   0.666    |   
    | CLIP-T |   0.321    |      0.333      |   **0.336**    | 

    From the results, we can see that the performance of finetuning the whole U-Net is comparable to Lora finetuning, and additionally, both DisenBooth-full and DisenBooth can alleviate the impact of overfitting and achieve higher CLIP-T(textual fidelity). However, considering that finetuning the whole U-Net for each subject will averagely consume 122 min while finetuning lora only requires 38 min on our V100 32G GPU. Therefore, we recommend finetuning with lora for efficiency.

Reviewer comment (Experiments):
The paper primarily concentrates on editing and personalizing single, often centrally located objects. While this is a challenging task, it would be intriguing to see how the method performs when dealing with scenes containing multiple objects. It is particularly important to understand how the encoding and disentanglement processes behave in such complex scenarios.

Author response:
**Customization for multiple subjects in a scene(W2/Q2)**:

    The reviewer proposes a very interesting scenario to customize multiple subjects in a scene, and we conduct a preliminary experiment under this scenario in Appendix A.7 in our revised version, which the reviewer can refer to for qualitative generation results(Figure 14) and analysis.
     
    During customizing multiple subjects in a scene, we will use a prompt to describe the multiple subjects for the identity-preserving branch, and leave the identity-irrelevant branch to learn other features. The preliminary exploration results show that our proposed method can also be applied to this scenario and generate overall satisfying images, demonstrating the potential of DisenBooth for broader applications. 
    
    Also, there are some challenges. Although the generated subjects are overall quite similar to the given subjects, some visual texture details of the generated subjects are a little different from the input subjects. The detail differences may result from the following reason: since the identity-preserving branch needs to customize more subjects, and the pixels for each subject become fewer, it will be hard to notice every detail of each subject. Although beyond the scope of this work, we believe customizing multiple subjects in a scene is an interesting topic and there are more problems worth exploring in future works.

Reviewer comment (Experiments):
It remains uncertain whether the mask could be explicitly integrated into the images before the encoding steps. How this approach might affect performance and how does it compare to the current masking strategy employed in the adaptor?

Author response:
**Using masks on pixels or on features(W3/Q3)**:

    Following the reviewer's advice, we compare the results of using pixel-level masks (explicitly integrating masks into images before encoding) and feature-level masks (masking in the adaptor) on 10 subjects in the following Table.

    |        | DreamBooth | Pixel Mask | DisenBooth(ours) |
    |--------|:----------:|:-------------:|:----------:|
    |  DINO  |   **0.671**    |     0.666     |   0.666    |
    | CLIP-T |   0.321    |     0.327     |   **0.336**    |

    We can see that both applying pixel-level masks (Pixel Mask) and feature-level masks (DisenBooth) can improve the CLIP-T score compared to DreamBooth, alleviating overfitting the identity-irrelevant information problem. However, it shows that applying the feature-level mask brings better performance. The reasons are as follows:

    + Masks at the feature level are filtering information at the semantic level instead of pixel level, enabling it to capture more identity-irrelevant factors such as the pose or position of the subject. Therefore, the identity-preserving branch does not have to overfit these factors, thus achieving better CLIP-T score and better text fidelity. However, using the masks on the pixels can only disentangle the foreground and the background, ignoring other identity-irrelevant factors such as pose.

    + The mask at the feature level is learnable and jointly optimized in the finetuning process, while adding an explicit mask to the image is a two-stage process, whose performance will be a little worse.

    We add the quantitative results (Table 4) and also qualitative analysis (Figure 10) in Appendix A.2 in our revised version to better explain the reasons (colored in blue). Generally speaking, if the users do not want to change other identity-irrelevant factors except for the background, such as the pose of the subject during generation, both ways can be applied. Otherwise, applying the mask at the feature level as our method may bring better performance.

Reviewer comment (Experiments):
The method appears to primarily focus on objects placed at the center of an image. What remains unclear is how it performs when objects are located in the corners of the image or occupy fewer pixels. It's essential to investigate potential failure cases in this context and assess whether the image and text encoders can accurately capture identity-preserving information in such scenarios.

Author response:
**Performance with objects in corners and covering fewer pixels(W4/Q4)**:
    
    Following the reviewer's suggestions, we conduct some experiments when the objects only cover fewer pixels and are not located in the center. The experimental results are presented in our revised paper in Appendix A.8 and Figure 15. The results show that:
    + Our method can still effectively learn the identity-preserving and identity-irrelevant information. 
    + However, similar to the multi-subject scenario, when the subject covers very few pixels, although the generated subject is overall similar to the given subject, some visual details are different. Therefore, in this scenario, a possible solution to preserve the details is to first crop the original image and then use a hyper-resolution network to obtain a clear image of the subject. After that, we can conduct finetuning with DisenBooth.

Thanks for your suggestions, we believe the review and the revised content can show more potential of DisenBooth and further improve the quality of the paper.