Paper: AlignDiff: Aligning Diffusion Models for General Few-Shot Segmentation (ICLR-2024)

Reviewer comment (Writing):
As few-shot segmentation might refer to semi-supervised segmentation with very few annotated examples and lots of unannotated examples in some literature, different from the setting in this paper, I suggest to state the problem setting in the very beginning of the paper.

Author response:
We thank the reviewer for pointing out the possible confusion of our semi-supervised mask generation method with semi-supervised segmentation. We have revised the introduction section so that the method and task settings are clearly stated.

Reviewer comment (Novelty):
If I recall correctly, using a special adjective token for a specific instance was proposed in [A], but it seems claimed as one of the contribution of this paper.

Author response:
Dear Reviewer cYiM,

Thank you for your reply and further clarification. We unintentionally made an honest mistake in our previous response – we apologize for this. Instead of tens of images, DreamBooth [A] indeed does work with as few as 4-6 images with the desired objects taking up most of the pixels. However, it is still inappropriate for generating data for few-shot segmentation, where the model is required to work with as few as a single image. In contrast, our AlignDiff works with as few as a single image, as shown in Fig. A4 in the appendix.

We have also updated our paper accordingly to cite DreamBooth. Initially, our method was motivated by and more related work textual inversion, which is a predecessor to DreamBooth that also performs image-to-text inversion. Note that we did not intend to claim learning tokens for adjectives as a part of the contributions for the paper. Instead, we discussed it as a detailed implementation and for best reproducibility. Our main contribution of this part is the revised loss term (which we term normalized masked textual inversion). This revised loss term is what allows us to learn with as few as a single image.

We attached the revised paragraph in the updated paper PDF below for your convenience.

> ... serves as the adjective description. This is similar to adjective token learning in DreamBooth (Ruiz et al., 2023) and is different from textual inversion (Gal et al., 2022) where the trainable embedding is the noun.

We hope this clarifies your concern. We are happy to discuss more if you have any remaining questions.

Reviewer comment (Presentation):
This paper does not clearly explain the "copy-paste" process, but simply cites another paper. An example figure would be nice. The images in Figure 2 is too dark to see clearly. If the space is not enough, I suggest to remove the introduction of diffusion models as it is becoming a common sense in this area.

Author response:
We revised the manuscript to explain more details and exact numbers in Section D.2 and refer readers to this section in the method section. In addition, we provide visualized copy-and-pasted images in Fig. A8 in the supplementary material.

Note that we intentionally implement the copy-paste in a simple manner because investigating the details of copy-paste is not a goal of this paper. Our goal is to argue that previous work (i.e., DiffuMask) uses a heavy image augmentation technique to generate samples with more realistic layouts for training because they overlook a simple yet effective method - copy-and-paste - to combine generated data with real data.

The main goal of the preliminary is to relate our revised loss scheme in Eq. 3 with Eq. 2 from the original diffusion work.

Reviewer comment (Evaluation):
The proposed method sounds very expensive. For every category, users need to personalize Stable Diffusion with text-inversion first, and then train the segmentor over and over again to contain the new generated masks into training set. However, the paper seems to only compare the mask generation speed. I would suggest to compare the whole process speed to benefit the community.

Author response:
Timing is measured using a single RTX3090 GPU and with 512 x 512 resolution. AlignDiff operates in two stages. In the first stage, AlignDiff takes ~5 minutes to inverse the embedding for few-shot samples.

In the second stage of sample generation, AlignDiff performs

Diffusion generation: ~4s per image  
Coarse mask: <0.1s per image  
Semi-supervised refinement: <0.15s per image  

We believe that AlignDiff is quite suitable for practical applications with its two-step scheme. The cost of the initial sample generation is amortized over the generation process.

Reviewer comment (Novelty):
The main performance gain seems to come from combining two papers: [B] and [C], which correspond to the contribution 2) and 3) in the introduction, respectively.

Author response:
We respectfully disagree with the reviewer that the performance gain comes from combining papers [B] and [C].

**Relationship with [B]**: though AlignDiff and [B] both use copy-paste [R1], the novelty of our approach does not lie in the fact that we use copy-paste. Instead, we argue the complexity of the approach in previous work such as DiffuMask, where heavy image augmentation techniques are used, is because they overlook real data. Copy-paste is merely an effective solution to relate generated data to real data.

**Relationship with [C]**: our mask generation method has completely no technical similarity with [C]. The one and only reason we mention [C] in our paper is to establish a graceful similarity overlooked by previous work between mask generation and semi-supervised learning. [C] is designed for semi-supervised classification, whereas our method focuses on generating masks for images generated by Stable Diffusion. We refer to [C] only to mention its task setting; whereas the technical contributions from [C] are irrelevant to our approach.

Reviewer comment (Evaluation):
Related to weakness 3), does the "copy-paste" process requires post-process harmonious method? If not, how to make sure the generated image make sense to the layout? Does it harm the performance?

Author response:
Previous work in copy-paste [R1] has found that copy-pasting without using any blending technique shows similar performance with implementations that use blending to harmonize the generated images. To make consistent with these previous works, we also did not implement harmonization technique in our paper and similarly found that even generated images with uncommon layouts yields significant performance gain (example generated images can be found in Fig. A8). An interesting direction would be to further harmonize the image to ensure the consistency of semantic layout, but the focus of this work is to identify the misalignment of diffusion models for generating training examples, so we leave this to future work.

Reviewer comment (Experiments):
Related to 1), with only a few examples of the new instance, how to obtain multi-object layout? Does the layout include the new instance?

Author response:
For more technical details, the current implementation obtains multi-object layout by randomly translating and randomly scaling objects before copy-paste. The instances that are used for copy-pasting include both real samples and samples generated from Stable Diffusion.

Reviewer comment (Novelty):
(Major) The three main contributions of the paper are extensions of existing approaches. This is not a problem, but in all cases the new approach is minor. For example, the copy-paste idea is used out of the box. Similarly, the iterative training of the few-shot segmentation model does not contain any particular innovation. The paper has limited novelty.

Author response:
We respectfully disagree with the reviewer that our work has limited novelty. In particular, we would like to highlight our novelty from two aspects:

(1) **Paradigm-level novelty:** We investigate a novel and interesting task setting in this paper - adapting text-to-image diffusion models to generate training samples (as recognized by reviewers ofKT and bS35). We went beyond naively using generated samples to facilitate the training process. Instead, we identify issues in existing works (e.g., mask generation efficiency and quality) and issues that are overlooked by existing works (e.g., generation for rare categories). 

(2) **Methodology-level novelty:** Previous work, such as DiffuMask, generates image-mask pairs with expensive GPU hours. In contrast, AlignDiff handles both generation for rare categories and generates high-quality masks with great efficiency, which is not possible with previous methods and has great potential for practical applications. We believe this constitutes technical novelty of our method.

Reviewer comment (Writing):
The term "out-of-distribution generation" is also confusing. There is no discussion of what is in-distribution information in terms of prompts or generated images. The paper may refer to the additional image variations given a prompt as OOD. This is not clear. However, OOD here differs from the common use of it in uncertainty estimation.

Author response:
We chose the term ‘out-of-distribution’ to highlight the misalignment of the distribution of generated samples and the distribution of synthetic samples. Out-of-distribution refers to generation of data points that fall outside the true data distribution, which we attempts to fix in this work. To avoid confusion, we revise the introduction section and give a definition of ‘OOD generation’ the first time we use this term.

Reviewer comment (Experiments):
It would be interesting to discuss whether the method is limited to latent diffusion models or generalisable to more diffusion approaches.

Author response:
Our method is not limited to a specific latent diffusion model, such as Stable Diffusion. In particular, our method makes two assumptions of the underlying diffusion model.

The normalized masked textual inversion method assumes that the diffusion model is differentiable from the final synthesized image to the text encoder.
The semi-supervised mask generation process requires generation of coarse masks. In particular, we generate coarse masks by exploiting the cross-attention layer within the diffusion model.

AlignDiff is applicable to any diffusion model that satisfies the above two properties, such as this work [D] that extends latent diffusion models to support synthesis with spatial-temporal consistency.

Reviewer comment (Experiments):
As stated in the limitation section, the proposed method degrades when the gap between the target distribution and the distribution covered by the generative model is large (e.g., medical images). However, I believe the few-shot semantic segmentation would be mostly useful for rare categories that are hard to collect enough instances for training a segmentation model. For common classes (e.g., chairs, sofa, boat) of which plenty of images can be found, it is hard to justify the necessity of using synthetic data. How is the performance of the proposed method on rare objects (e.g., the rare species from the iNaturalist dataset)?

Author response:
Though AlignDiff may not generalize well to domains where domain-specific knowledge is required (e.g., medical images), it is able to work on hard categories that plain text conditioning fails. In fact, this is our intention to evaluate AlignDiff on the FSS-1000 dataset, because FSS-1000 contains many rare categories.

To validate this point, we provide more qualitative samples in the supplementary material, which can be found in Fig. A4 of the supplementary material. Since iNaturalist does not provide segmentation annotations, it does not work with AlignDiff. Therefore, we visualize some rare categories of fine-grained species of animals from the FSS-1000 dataset. Notice how AlignDiff is able to capture the fine-grained details of textures in these species; while text-conditioned Stable Diffusion generates irrelevant samples.

Reviewer comment (Experiments):
Limited comparison on the FSS-1000 dataset

Author response:
## More results on the FSS-1000 dataset

To illustrate AlignDiff is a model-agnostic approach. we introduce an additional baseline FSS model [A] and compare its performance on the FSS-1000 dataset with and without generated data from AlignDiff and GD. AlignDiff consistently improves the performance of [A] on the FSS-1000 dataset under 1-shot setting.

| Method  | Overall IoU |
| - | - |
| HSNet (1-shot) | 86.5 |
| VAT-R101 [A] (1-shot) | 90.0 |
| HSNet + GD (1-shot) | 81.4 |
| VAT-R101 [A] (1-shot) + GD | 84.7 |
| HSNet (1-shot) + AlignDiff (Ours)   | 88.3 |
| VAT-R101 (1-shot) [A] + AlignDiff (Ours) | 90.8 |
| HSNet (5-shot) | 88.5 |
| VAT-R101 [A] (5-shot) | 90.6 |

Reviewer comment (Experiments):
Evidence of the finding that treating the learnable embedding as an adjective leads to a faster and more stable training convergence, is missing

Author response:
## Validating the advantage of learning “adjective” embedding over nouns

We present qualitative and quantitative results that prove the superiority of learning an embedding for adjectives over nouns. As we have mentioned in the paper, when only a limited number of samples are available, learning an adjective embedding leads to more stable performance, as the additional nouns serve as a regularization that keeps the generation process to be related to the semantic concept.

We provide qualitative samples of learning adjective and noun embedding for the same 1-shot example in Fig. A11 in the supplementary material of the revised paper. In particular, we notice that optimizing noun embedding may result in degenerate samples when only a single example with a small object is available; whereas optimizing adjective embedding works in this case.

In addition, we also provide results on a small-scale experiment on Pascal-5-3. 

| Method  | Overall IoU |
| - | - |
| Baseline - learn noun | 39.9 |
| Ours | 41.5 |

Reviewer comment (Experiments):
The evaluation setup describes that DiffuMask without their prompt engineering is used, but it is missing from Table1 and Table2

Author response:
## Evaluation of DiffuMask without prompt engineering

Thanks for the question. The main reason why we did not make a comprehensive comparison with DiffuMask without prompting is because of the expensive GPU hours it required (The inefficient noise learning process in DiffuMask requires thousands of GPU hours to generate samples on the COCO dataset).

As an alternative, we present a smaller-scale study that compares our method with DiffuMask on a split of the Pascal-5i dataset. The table can be found in Table A2 in the supplementary material. We have also updated our paper to refer readers to Table A2 for better clarity. In summary, DiffuMask generates masks of similar quality with AlignDiff on common categories, but it comes at a much higher cost with the costly noise learning process.

Reviewer comment (Experiments):
Do the methods compared with in Table1 and Table2 also use base classes, or is it extra information used in this approach? (for copy-paste and semi-supervised learning parts)

Author response:
## Clarification of base class information use

Thanks for the question. To clarify, to ensure a fair comparison, some other baseline methods also use information from the base classes. For instance, GD trains its zero-shot segmenter using image-mask samples from the base dataset. Also, GAPS+GD uses samples from the base class. We observe that AlignDiff shows improvement against baselines that also use information from the base class, which validates the effectiveness of our method.

Reviewer comment (Experiments):
An ablation on the amount of generated samples would be valuable in practice

Author response:
## Plot of performance w.r.t. number of generated samples

Thanks for the question. We provide a plot of the novel IoU on a split of the Pascal-5i dataset in the revised PDF. The figure can be found in Fig. A12 in the revised supplementary material. Note that the performance saturates around 500-1,000 samples.