Paper: An Image is Worth Multiple Words: Learning Object Level Concepts using Multi-Concepts Prompts Learning (ICLR-2024)

Reviewer comment (Experiments):
The aim of this paper is to learn and compose multiple concepts in the same scene. However, the demonstrations to prove the composing ability are insufficient. In almost all demonstrations the concepts are composed in the same string as training examples without any changes and only some editing demonstrations are available. Upon the interaction between the two concepts is changed, the effects seem to be worse.

Author response:
* We thank the reviewer for raising this important experiment. Both ‘composing capability’ and ‘learning more than two concepts' are covered in Figure 11-12 of Section A.2, which involves **learning three or more concepts from a single image and composing a subset in a new scene**. 
* **Both tasks involve composing with strings differ from training**, our method generally learns targeted concepts and showed decent composing capability, outperforming Textural Inversion and DreamBooth, and was closer to **Break-A-Scene**, which represents the **state-of-the-art and a performance upper-bound in masked-based multi-concept learning**. **Unlike BAS, our approach neither uses segmentation masks as input nor updates model parameters.**

Reviewer comment (Writing):
Some writing mistakes exist in the paper. In the top right of Figure 3, the labeling of “on” and “under” should be reversed. On page 6, “The” in “Therefore The contrastive loss” should be lowercase in line 8 of the paragraph before “Implementation details”.

Author response:
* We thank the reviewer for the thoughtful reviews and for pointing out those writing mistakes, we updated Figure 3 and corrected the writing mistake in the revised submission.

Reviewer comment (Experiments):
The experiment is a little confusing. In the section “Baselines and experiments” on page 7, the author presents four learning methods to compare their effectiveness. However, the author seems not to explain the meaning of the first setting called textural Inversion applied to unmasked multi-concept images and the subsequent experiments don’t contain the effects of the first two settings. The detail and comparison for a setting called “MCPL-diverse” are also unavailable.

Author response:
* We are thankful for the reviewer’s question and interest in “MCPL-diverse” — a powerful but under-explored variant of our method in the original submission. We perform new experiments and provide clarifications as follows.
* **MCPL-diverse v.s. MCPL-one with segmentation masks**: we add new qualitative results in Figures 13 and 14 in Section A.3 for the task of learning per-image different concepts. In both examples, as **MCPL-diverse is specially designed for such tasks, it consistently outperforms MCPL-one**.
* MCPL-diverse has also been found powerful in more competitive tasks of learning more than three concepts from a single image, following R2(RxXV)’s suggestion on comparing competitive baseline Break-A-Scene, which represents the **state-of-the-art and a performance upper-bound in masked-based multi-concept learning**. **Unlike BAS, MCPL neither uses segmentation masks as input nor updates model parameters.**
* While R2 noted it's somewhat **unfair to compare directly with BAS due to its segmentation inputs**, we add Figure 12 of Section A.2 for this ambitious comparison. In Figure 12, we evaluated both MCPL-one and MCPL-diverse, the latter employing random crops akin to BAS’s ’union sampling’ strategy. The random crops strategy leads to each image containing different multiple concepts. In such case, **we found MCPL-diverse is a perfect fit, outperforming MCPL-one and comparable to the ambitious BAS baseline.**  
* We thank the reviewer for identifying this clarification issue! The first setting called ‘textural Inversion applied to unmasked multi-concept images was a very initial baseline we explored. It was designed to assess the learning of combined concepts (i.e. encoding two concepts in one prompt embedding). This baseline was evaluated by the t-SNE visualisation and because it was **less comparable to other experiments** hence has been presented in Figures 26 and 27 in Appendix A.5. 
* We apologise for the above confusion and revised the entire experiments section to further clarify all experiment setups.

Reviewer comment (Experiments):
The paper constructs a new dataset to evaluate the proposed framework for multi-concept learning. However, I see all provided examples contain only two distinct concepts so I doubt the generalization of this method.

Author response:
* We thank the reviewer for raising this important experiment. Both ‘composing capability’ and ‘learning more than two concepts' are covered in Figure 11-12 of Section A.2, which involves **learning three or more concepts from a single image and composing a subset in a new scene**. 
* **Both tasks involve composing with strings differ from training**, our method generally learns targeted concepts and showed decent composing capability, outperforming Textural Inversion and DreamBooth, and was closer to **Break-A-Scene**, which represents the **state-of-the-art and a performance upper-bound in masked-based multi-concept learning**. **Unlike BAS, our approach neither uses segmentation masks as input nor updates model parameters.**

Reviewer comment (Experiments):
The main concern is the analysis between MCPL-diverse mentioned in the preliminary study and the full version of MCPL-one.

Author response:
We are also thankful for the reviewer’s question and interest in **“MCPL-diverse” — a powerful but under-explored variation of our method in the original submission**. We perform new experiments and provide clarifications as follows

 > _… analysis between the MCPL-diverse mentioned in the preliminary study and the full version of MCPL-one. The authors could provide visualization of generated natural concepts from MCPL-diverse as well as corresponding generated segmentation masks to support the observation._

* **MCPL-diverse v.s. MCPL-one with segmentation masks**: we add new qualitative results in Figures 13 and 14 in Section A.3 for the task of learning per-image different concepts. In both examples, as **MCPL-diverse is specially designed for such tasks, it consistently outperforms MCPL-one**.
* MCPL-diverse has also been found powerful in more competitive tasks of learning more than three concepts from a single image, following **R2(RxXV)**’s suggestion on comparing competitive baseline **Break-A-Scene**, which represents the **state-of-the-art and a performance upper-bound in masked-based multi-concept learning**. **Unlike BAS, our MCPL neither uses segmentation masks as input nor updates model parameters.**
* While R2 noted it's somewhat **unfair to compare directly with BAS due to its relying on segmentation inputs**, we add Figure 12 of Section A.2 for this ambitious comparison. In the right example of Figure 12, we evaluated both MCPL-one and MCPL-diverse, the latter employing random crops akin to BAS’s ’union sampling’ strategy. The random crops strategy leads to each image containing different multiple concepts. In such case, **we found MCPL-diverse is a perfect fit, outperforming MCPL-one and comparable to the ambitious BAS baseline.**

Reviewer comment (Presentation):
The authors could provide visualization of generated natural concepts from MCPL-diverse as well as corresponding generated segmentation masks to support the observation.

Author response:
We are also thankful for the reviewer’s question and interest in **“MCPL-diverse” — a powerful but under-explored variation of our method in the original submission**. We perform new experiments and provide clarifications as follows

 > _… analysis between the MCPL-diverse mentioned in the preliminary study and the full version of MCPL-one. The authors could provide visualization of generated natural concepts from MCPL-diverse as well as corresponding generated segmentation masks to support the observation._

* **MCPL-diverse v.s. MCPL-one with segmentation masks**: we add new qualitative results in Figures 13 and 14 in Section A.3 for the task of learning per-image different concepts. In both examples, as **MCPL-diverse is specially designed for such tasks, it consistently outperforms MCPL-one**.
* MCPL-diverse has also been found powerful in more competitive tasks of learning more than three concepts from a single image, following **R2(RxXV)**’s suggestion on comparing competitive baseline **Break-A-Scene**, which represents the **state-of-the-art and a performance upper-bound in masked-based multi-concept learning**. **Unlike BAS, our MCPL neither uses segmentation masks as input nor updates model parameters.**
* While R2 noted it's somewhat **unfair to compare directly with BAS due to its relying on segmentation inputs**, we add Figure 12 of Section A.2 for this ambitious comparison. In the right example of Figure 12, we evaluated both MCPL-one and MCPL-diverse, the latter employing random crops akin to BAS’s ’union sampling’ strategy. The random crops strategy leads to each image containing different multiple concepts. In such case, **we found MCPL-diverse is a perfect fit, outperforming MCPL-one and comparable to the ambitious BAS baseline.**

Reviewer comment (Novelty):
Some recent works (such as Break-A-Scene) on similar tasks could also be mentioned in the related work section. On the other hand, though it would be a bit unfair to directly compare with Break-A-Scene (due to its given segmentation inputs), it could still be interesting to treat its performance as an upper bound and comment on how a good segmentation mask would affect the learned concepts.

Author response:
* We wholeheartedly concur and are thankful for the reviewer highlighting this significant recent reference, which had previously been overlooked in our work. In response, we have **conducted extensive experiments to incorporate Break-A-Scene into all our quantitative analyses and visual comparisons**. This inclusion enriches our study and provides a more comprehensive understanding of our method in the context of current advancements
* Specifically, we have updated Figures 5, 7, 8, and 16-23 to compare against **Break-A-Scene as a performance upper bound** in our 16 concepts experiments. **This expansion includes an additional 160 single GPU runs, approximately 2 million pairwise similarities, and extensive human-in-the-loop pre- and post-processing for integration with Break-A-Scene**, as detailed in Section A.1. Moreover, Figures 11-12 in Section A.2 provide a **visual comparison with Break-A-Scene as well as other baselines**.
* It is worth noting **BAS requires segmentation masks as input and employs separate segmentation models to produce masked objects**, hence integration BAS involves extensive human-in-the-loop pre- and post-processing efforts, as detailed in Section A.1. In contrast, **our method is mask-free at learning and employs its own AttnMask to generate masked objects.**
* The following tables highlight new results in Figure 8, our fully regularised method (MCPL-one+CL+Mask) shows **competitive performance against BAS on natural images**. For the **out-of-domain (OOD) medical dataset, BAS leads in the DINOv1 space, but we're comparable in others**. This is **due to our less precise object masks compared to BAS's human-assisted MedSAM segmentation**, shown in Figures 5 and 6. 

**Object-level fidelity learned concepts referencing to "ground truth" (natural images)**

| exp_names           | bert          | clip          | dinov1        | dinov2        |
|---------------------|---------------|---------------|---------------|---------------|
| MCPL-one+CL+Mask    | 0.271 ± 0.078 | 0.821 ± 0.050 | 0.534 ± 0.112 | 0.991 ± 0.006 |
| BAS                 | 0.276 ± 0.078 | 0.823 ± 0.050 | 0.552 ± 0.097 | 0.995 ± 0.030 |

**Object-level fidelity learned concepts referencing to "ground truth" (medical images)**

| exp_names           | bert          | clip          | dinov1        | dinov2        |
|---------------------|---------------|---------------|---------------|---------------|
| MCPL-one+CL+Mask    | 0.262 ± 0.072 | 0.792 ± 0.037 | 0.407 ± 0.081 | 0.983 ± 0.009 |
| BAS                 | 0.267 ± 0.079 | 0.823 ± 0.043 | 0.560 ± 0.080 | 0.987 ± 0.021 |

* **The new t-SNE results in** Figure 7 demonstrate that the learned embeddings from both the mask-based ’ground truth’ and BAS show less disentanglement compared to ours, attributable to their lack of a specific disentanglement objective, such as the PromptCL loss in MCPL.
* We've reflected on all the mentioned points in our revision, including ensuring proper citation of Break-A-Scene throughout the paper.

Reviewer comment (Presentation):
It would be better to discuss the limitations of current work in the last section and point out some future improvement directions.

Author response:
* We appreciate the reviewer's valuable suggestion! We've included a discussion on limitations and future work in the '**Limitation and Conclusion**' section, a copy of which is provided below for your convenience in reviewing.
* We found our method to suffer from the following limitations: 
    * (1) **Imperfect Masking**: Our reliance on image-level text descriptions, instead of segmentation masks, grants flexibility in exploring unknown concepts but results in less precise object boundary optimization. Future research could use our AttnMask as an input prompt to segmentation models for mask refinement. 
    * (2) **Composition Capability**: MCPL’s composition strength is weaker than BAS, as MCPL doesn’t update model parameters, unlike BAS. Integrating MCPL with weight optimization methods like BAS or DreamBooth may enhance performance, albeit at higher computational costs, which is a potential direction for future work. 
    * (3) **Evaluation Metrics**: Current quantification methods in this field (e.g. TI, DB, CD, BAS, and P2P), predominantly rely on prompt/embedding similarity due to the absence of more effective quantification mechanisms without known ground truth. This indicates a need for developing better evaluation metrics in future research. 
    * (4) Our method **relies on adjectives** serving as textual descriptors (e.g., colour) to differentiate between multiple concepts. While human-machine interaction using purely linguistic descriptions is generally preferred, challenges arise when two concepts are very similar and lack distinct visual cues in the image. In such cases, our method may struggle, and Break-A-Scene currently offers the best solution.

Reviewer comment (Experiments):
It seems that all learned concepts (nouns) are associated with a pre-defined adjective description in the prompt (such as “a green *”). Are these adjectives playing roles in disentangling the concepts? What if the initial prompt is provided in the form “A * and a @”?

Author response:
> _Q.1 - It seems that all learned concepts (nouns) are associated with a pre-defined adjective description in the prompt (such as “a green *”). Are these adjectives playing roles in disentangling the concepts? What if the initial prompt is provided in the form “A * and a @”?_

* We add Figure 15 in section A.4 for an ablation study on the impact of adjective words. **Adjective words are crucial in linking each prompt to the correct region**; without them, the model may struggle for regional guidance, **akin to the role of masks in Break-A-Scene**.  
* Figure 38 highlights how each regularisation technique enhances object-level prompt-concept correlation in MCPL. The left example shows **reduced accuracy in attention and masks without Bind Adjectives**.
* The aforementioned Figure 12-right results reinforce the significance of adjectives in maintaining model performance.

Reviewer comment (Experiments):
I wonder if there are any examples of cases when the number of objects in the image is mismatched with the number of learnable concepts.

Author response:
> _Q.2 - I wonder if there are any examples of cases when the number of objects in the image is mismatched with the number of learnable concepts_

* We showcase examples in Figure 11-12 of Section A.2, which involves: 1) learning three concepts from a single image and composing a subset in a new scene. 2) more challenging, entails learning six concepts from one image and composing 1 to 3 concepts in a novel scene.
* In all **“mismatched tasks”**, our method generally learns targeted concepts and showed decent composing capability, **outperforming Textural Inversion and DreamBooth, and was closer to BAS.**
* We conclude the **'Bind Adjectives' regularisation directs our model to the correct location**. This is evidenced by the **performance diminishes when adjectives are removed**, as seen in the right example of Figure 12.

Reviewer comment (Writing):
The paper is not very clear to read. The idea seems to be straightforward, but the description of the method is a bit ambiguous. I have to read multiple times to make sure I understand the method accurately.

Author response:
* We thank the reviewer for pointing out the clarity issue in our manuscript. In response, we have **simplified certain descriptions in the methods section** to enhance clarity.
* We also acknowledge our methods section combined method descriptions with preliminary motivational experiments, and this design might have disrupted the flow of the method description. To address this, we have **reorganised the methods section, including adjustments to Figure 4**, to ensure a more coherent and seamless presentation of our methods.

Reviewer comment (Experiments):
In experiments, the authors show multiple interesting qualitative results. However, there are very little quantitative results, and it is very hard to compare with other methods and understand the contribution of this effort.

Author response:
* We are grateful to the reviewer for highlighting a potential limitation in our study. As our work is **pioneering** in the field of multi-concept learning **without ground-truth object segmentation**, we acknowledge the **scarcity of directly comparable baselines**.
* **Our original experiments were extensive**, taking about 80 days on a single GPU, a fact recognised by other reviewers. However, following R2's advice, we **added Break-A-Scene (BAS) as an additional benchmark**. We perform **extensive new experiments resulting in an extra seven pages in the revised version.** We have updated Figures 5, 7, 8, 16-23, adding Figures 10-12, and detailing in Sections A.1 and A.2 for a comprehensive comparison. This involved 160 more single GPU runs and around 2 million pairwise comparisons.
* This thorough approach ensures our approach is **quantitatively compared** against **three baselines** (Textural Inversion, MCPL, and BAS) and a **qualitative comparison** against an **additional five baselines** (Figure 11-12), comprehensively covering the scope of the field.
* It worth noting **BAS requires segmentation masks as input and employs separate segmentation models to produce masked objects**, hence integration BAS involves extensive human-in-the-loop pre- and post-processing efforts, as detailed in Section A.1. In contrast, **our method is mask-free at learning and employed its own AttnMask to generate masked objects.**
* In the following results (Figure 8 in table format), our fully regularised method (MCPL-one+CL+Mask) shows **competitive performance against BAS on natural images**. For the **out-of-domain (OOD) medical dataset, BAS leads in the DINOv1 space, but we're comparable in others**. This is **due to our less precise object masks compared to BAS's human-assisted MedSAM segmentation**, shown in Figures 5 and 6.

**Object-level fidelity learned concepts referencing to "ground truth" (natural images)**

| exp_names           | bert          | clip          | dinov1        | dinov2        |
|---------------------|---------------|---------------|---------------|---------------|
| MCPL-all            | 0.235 ± 0.076 | 0.742 ± 0.050 | 0.311 ± 0.115 | 0.983 ± 0.010 |
| MCPL-one            | 0.235 ± 0.073 | 0.760 ± 0.065 | 0.388 ± 0.143 | 0.978 ± 0.014 |
| MCPL-all+CL         | 0.247 ± 0.081 | 0.756 ± 0.051 | 0.346 ± 0.115 | 0.983 ± 0.010 |
| MCPL-all+CL+Mask    | 0.251 ± 0.083 | 0.756 ± 0.057 | 0.362 ± 0.128 | 0.982 ± 0.012 |
| MCPL-one+CL         | 0.265 ± 0.089 | 0.791 ± 0.061 | 0.467 ± 0.128 | 0.986 ± 0.010 |
| MCPL-one+CL+Mask    | 0.271 ± 0.078 | 0.821 ± 0.050 | 0.534 ± 0.112 | 0.991 ± 0.006 |
| BAS                 | 0.276 ± 0.078 | 0.823 ± 0.050 | 0.552 ± 0.097 | 0.995 ± 0.030 |

**Object-level fidelity learned concepts referencing to "ground truth" (medical images)**

| exp_names           | bert          | clip          | dinov1        | dinov2        |
|---------------------|---------------|---------------|---------------|---------------|
| MCPL-all            | 0.240 ± 0.065 | 0.787 ± 0.046 | 0.349 ± 0.069 | 0.977 ± 0.011 |
| MCPL-one            | 0.250 ± 0.069 | 0.776 ± 0.044 | 0.329 ± 0.062 | 0.982 ± 0.010 |
| MCPL-all+CL         | 0.251 ± 0.083 | 0.785 ± 0.040 | 0.339 ± 0.062 | 0.982 ± 0.011 |
| MCPL-all+CL+Mask    | 0.264 ± 0.065 | 0.785 ± 0.045 | 0.353 ± 0.068 | 0.982 ± 0.010 |
| MCPL-one+CL         | 0.247 ± 0.070 | 0.791 ± 0.047 | 0.376 ± 0.095 | 0.981 ± 0.012 |
| MCPL-one+CL+Mask    | 0.262 ± 0.072 | 0.792 ± 0.037 | 0.407 ± 0.081 | 0.983 ± 0.009 |
| BAS                 | 0.267 ± 0.079 | 0.823 ± 0.043 | 0.560 ± 0.080 | 0.987 ± 0.021 |

* **The new t-SNE results in** Figure 7 demonstrate that the learned embeddings from both the mask-based ’ground truth’ and BAS show less disentanglement compared to ours, attributable to their lack of a specific disentanglement objective, such as the PromptCL loss in MCPL.

Reviewer comment (Novelty):
Attention masks and contrastive loss on different concepts seems to be a widely used method. It would be great if the authors can explain a bit more about the novelty of their work.

Author response:
We appreciate the reviewer's inquiry and are pleased to clarify our contributions as follows:

* Firstly, we highlight the consensus among all reviewers on **our work targets on important research areas** of **mask-free text-guided learning of multiple prompts from a single scene**. 
* We introducing Multi-Concept Prompt Learning (MCPL), **the first** to achieve multi-concept learning **without relying on object masks as input**. This novelty has been recognised by all the other reviewers.
* Our approach **differs to the leading mask-based method**, Break-A-Scene (BAS) as follows:
    * BAS learns multiple concepts from images paired with object-level masks but we don’t use a mask;
    * BAS updates both textural embeddings and model weights but we don’t update model weights; 
* We also summarise **other key contributions** our work have made:
    * The **motivational study** introduces the problem and the current limitation with vanilla MCPL. Addressing this, we propose **three innovative regularisation techniques**: Attention Masking, Prompts Contrastive Loss, and Bind Adjectives, to disentangle multiple object concepts and enhance the accuracy of object-level concept learning.
    * **Extensive experiments and analysis** demonstrate superior performance with our method against competitive baselines, delivering results comparable to the leading mask-based method, Break-A-Scene. This **highlights the potential of our mask-free approach**. We also provide detailed visualisations to showcase the practical applications of our method.
    * We have compiled a **new dataset** for comprehensive evaluation. This dataset involves both in-domain natural images and out-of-domain biomedical images, **setting up a new benchmark for the field**.
    * Overall, our approach not only enhances current methodologies but also **paves the way for novel applications**, such as facilitating knowledge discovery through natural language-driven interactions between humans and machines.
* All of the above clarifications and contributions have been reflected in our revised manuscript.