Paper: SLiMe: Segment Like Me (ICLR-2024)

Reviewer comment (Experiments):
further, within the appendix, Table 9 shows an improvement from adding WAS-attention, from 62.7 on average to 68.3. I'm a little confused, then, on how this differs from Table 5, as I would have though the last row, setting $\alpha=0$, would also correspond to dropping WAS-attention, and here the average performance is 68.0

Author response:
**Clarification on Tables 5 and 9**

We apologize for the confusion. We clarify that WAS-attention can be employed either solely during inference or during both optimization and inference. When used solely during inference, it has no impact on the optimization process. 

The reference made by the reviewer to Table 5 refers to the scenario where WAS-attention is applied exclusively during inference. On the other hand, Table 9 presents results when WAS-attention is entirely excluded from both optimization and inference steps in the car parts segmentation task. Notably, this exclusion leads to a notable decrease in IoU, dropping from 68.3% to 62.7%.

We ran another experiment with completely removing WAS-attention from both optimization and inference steps on the horse parts segmentation task, where we also see a significant IoU drop (63.3% to 52.4%). 

This table provides a clear breakdown of the values for each body part, comparing the results without and with the Weighted Average Score (WAS).

| Part      | Without WAS   | With WAS      |
|----------------|---------------|---------------|
| Background     | 68.2% ± 5.0   | 79.6% ± 2.5   |
| Head           | 47.8% ± 2.8   | 63.8% ± 0.7   |
| Legs           | 50.1% ± 0.6   | 59.5% ± 2.1   |
| Neck+Torso     | 57.3% ± 13.1  | 68.1% ± 4.4   |
| Tail           | 38.6% ± 1.4   | 45.4% ± 2.4   |
| Average        | 52.4% ± 3.8   | 63.3% ± 2.4   |

Reviewer comment (Novelty):
The proposed method seems to be adapted for the segmentation task based on the work by Hedlin et al for unsupervised semantic correspondence.

Author response:
**Hedlin et al.’s work**

We acknowledge that the high level idea of our work might have similarities to Hedlin et al.’s work, but their objectives are completely different from ours: we perform image segmentation while their task is image correspondence. Furthermore, in terms of architectural similarities, the optimization of cross-attention maps has been previously explored for various downstream tasks, as demonstrated in the work by Hedlin et al. However, our current task differs, and adopting their approach for our segmentation task did not yield effective results. Hedlin et al.’s exploration is confined to the optimization of a single text embedding, while our approach allows optimizing multiple text embeddings, enabling the simultaneous segmentation of various parts or objects. 

We attribute the significantly enhanced performance of our method to the utilization of WAS-attention which we propose in this work, as demonstrated in Table 9 and also the following results on horse part segmentation which we ran to address reviewer Fjuo’s comment. 

| Part      | Without WAS   | With WAS      |
|----------------|---------------|---------------|
| Background     | 68.2% ± 5.0   | 79.6% ± 2.5   |
| Head           | 47.8% ± 2.8   | 63.8% ± 0.7   |
| Legs           | 50.1% ± 0.6   | 59.5% ± 2.1   |
| Neck+Torso     | 57.3% ± 13.1  | 68.1% ± 4.4   |
| Tail           | 38.6% ± 1.4   | 45.4% ± 2.4   |
| Average        | 52.4% ± 3.8   | 63.3% ± 2.4   |

Reviewer comment (Novelty):
Related works which are not cited: [a] Burgert, Ryan, et al. "Peekaboo: Text to image diffusion models are zero-shot segmentors." arXiv preprint arXiv:2211.13224 (2022). [b] Tian, Junjiao, et al. "Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion." arXiv preprint arXiv:2308.12469 (2023).

Author response:
**Related works [a] and [b]**

Thank you for bringing this to our attention. We've noticed that neither of the two works are peer-reviewed. While we appreciate their contributions, it's essential to highlight the distinctions that make them not directly comparable to SLiMe. In summary, Peekaboo requires an object description for each segment, a requirement absent in SLiMe. Additionally, Peekaboo lacks the capability to perform segmentation at the object part level, which is a fundamental feature of SLiMe.

Regarding the second work by Junjiao et al., as we understand it, the method primarily segments images into different parts based on feature similarity, akin to the approach in the Segment Anything paper. This method lacks control over segmentation granularity and cannot perform targeted object/part segmentation. For example, it cannot be directed to segment a specific part of an object. These distinctions emphasize the unique capabilities and advantages of SLiMe in comparison.

Reviewer comment (Experiments):
The number of works that are compared against is limited.

Author response:
**The number of methods that are compared against are limited**

We have conducted thorough comparisons with three _most_ relevant papers to SLiMe. Notably, these papers also provide comparisons with other existing methods, offering readers valuable insights into our performance relative to those cited works (although we didn't include them in our study as their setups differed from ours). In response to a reviewer's suggestion (please refer to our responses to reviewer Fjuo for details), we further conducted experiments on two additional datasets, providing readers with a broader context for evaluating the performance of our work. We are open to suggestions on specific papers that you believe warrant comparison, and we genuinely welcome any recommendations for meaningful comparisons to be incorporated into the final version of the paper. Thank you

Reviewer comment (Experiments):
Is there any specific reason behind not including the SegDDPM results in Tab. 1?

Author response:
**Not including SegDDPM in Table 1**

We conducted a comparative analysis of SLiMe and SegDDPM on two segmentation tasks: _face_ and _horse_ part segmentation (Tables 2 and 3). It's worth noting that the SegDDPM paper did not report results for the _car_ part segmentation task (Table 1). Our attempts to run their method on this task revealed a challenge: for SegDDPM to segment cars into their parts, it requires a DDPM trained to generate car images. Unfortunately, the SegDDPM authors have not provided a DDPM checkpoint for car image generation, preventing us from assessing SegDDPM's performance in car part segmentation.

Reviewer comment (Experiments):
Unconvincing importance of WAS attention: Figure 1 shows a good intuition that we need WAS attention to refine the object boundary. However, in Table 5, the contribution of the WAS attention doesn't seem to be significant. For the without WAS attention results (fig. 2a, 2c), will they be much improved by using GrabCut or other methods for segmentation refinement post-processing?

Author response:
**WAS-attention's effect** 

We would like to clarify that WAS-attention can be employed either solely during inference or during both optimization and inference. When used solely during inference, it has no impact on the optimization process. 

The reference made by the reviewer to Table 5 refers to the scenario where WAS-attention is applied exclusively during inference. On the other hand, Table 9 presents results when WAS-attention is entirely excluded from both optimization and inference steps in the car parts segmentation task. Notably, this exclusion leads to a notable decrease in IoU, dropping from 68.3% to 62.7%.

We ran another experiment with completely removing WAS-attention from both optimization and inference steps on the horse parts segmentation task, where we also see a significant IoU drop (63.3% to 52.4%). 

This table provides a clear breakdown of the values for each body part, comparing the results without and with the Weighted Average Score (WAS).

| Part      | Without WAS   | With WAS      |
|----------------|---------------|---------------|
| Background     | 68.2% ± 5.0   | 79.6% ± 2.5   |
| Head           | 47.8% ± 2.8   | 63.8% ± 0.7   |
| Legs           | 50.1% ± 0.6   | 59.5% ± 2.1   |
| Neck+Torso     | 57.3% ± 13.1  | 68.1% ± 4.4   |
| Tail           | 38.6% ± 1.4   | 45.4% ± 2.4   |
| Average        | 52.4% ± 3.8   | 63.3% ± 2.4   |

=====================

**Post-processing e.g., GrabCut** 

When it comes to post-processing techniques like traditional GrabCut, it's important to note that not only do they introduce additional computations during inference, but they may also necessitate user intervention for refining segmentations in complex cases (refer to Figure 5 in the GrabCut paper). These methods often struggle to generalize in scenarios where there is no distinct color difference between object and background, or in cases with subtle color/texture transitions between different parts of an object. For instance, in face part segmentation, distinguishing noise from the actual face becomes challenging due to their almost identical texture and color.

Reviewer comment (Experiments):
Lack of comparison on benchmark datasets like ADE-Bedroom-30 (used by segDDPM) and FSS-1000 (used in segGPT). The current quantitative results are only on horse/car/face datasets used in ReGAN. The result can be more solid with evaluation on diverse types of objects/parts.

Author response:
**Results on ADE-Bedroom-30 and FSS-1000**

We would like to emphasize that while the SegGPT model was tested on unseen samples/datasets, its foundational base model is trained on extensive supervised segmentation data—comprising images and corresponding segmentation masks. This means that SegGPT's underlying supervised model may have encountered categories or samples resembling the test data. However, it's important to note that we do not utilize a base supervised segmentation model. Nevertheless, when applied to the FSS-1000 dataset, our method yielded a reasonable IoU:

SLiMe (ours): 73.53 %
Painter: 61.7%
SegGPT: 85.6% 

Painter is cited in the SegGPT paper as a similar technique which also leverages a supervised segmentation base model (as SegGPT does). Notably, our SLiMe outperforms Painter.

For the ADE-Bedroom-30 dataset, we obtained an average mIoU of 32.3 using the same settings as SegDDPM, which is comparable to the 34.6 achieved by segDDPM. We identified a slight underperformance in SLiMe, attributed to its imperfect handling of small and featureless objects, such as man-made items like lamps in the ADE-Bedroom dataset. This limitation arises from the fact that these objects become tiny or even vanish in the cross-attention maps we employ, sized at 16x16 and 32x32, while the input images for this dataset are at a resolution of 256x256. Addressing this issue could be a valuable improvement for SLiMe in future work.

Reviewer comment (Evaluation):
For the learned text embedding, are they interpretable? e.g. one can use the data-driven approach to find text tokens whose embedding are closest to the optimized ones.

Author response:
**Interpretability of learned embeddings** 

We calculated cosine distance between some of the learned text embeddings with our SLiMe and the words/embeddings in the Stable Diffusion dictionary. The closest text tokens in the Stable Diffusion’s dictionary to the learned ones using SLiMe do not necessarily match. For instance, the 10 closest words in the Stable Diffusion dictionary to the optimized embedding of 'bear' segmentation are:

```
{brasil, nightout, mccall, boudo, confirmation, oldschool, stockholm, regardless, beacon, tik}
```

which do not correspond to the concept of 'bear.' However, a future work could involve a deeper exploration of the initial and the evolution of the word embeddings during optimization, which may lead to improved segmentation results or more interpretable tokens.

Reviewer comment (Writing):
The introduction of the text prompt is quite abrupt. In the Introduction, SLiMe is described as requiring only an image and a corresponding mask to achieve segmentation of any granularity. However, immediately after, the author talks about fine-tuning text embeddings. What is the definition of 'text' in this task? How are text embeddings obtained? And do different granularities correspond to the same text? The author is encouraged to provide further clarification.

Author response:
**Clarification on prompt, tokens, and embedding**

We appreciate the reviewer's insight regarding the potentially confusing tokenization step. To address this concern, we plan to include a new figure similar to this: https://imgur.com/a/0dutj8r that provides a clearer illustration of how each part relates to tokens for granular level segmentation. As illustrated in the figure, each part (p) in the image corresponds to a token (t) in the embedding, which is initially set as a null text (i.e., "") to be later optimized. Consequently, we do not utilize any prompt as input. The inputs to SLiMe include empty token placeholder(s), an image, and a corresponding segmentation mask. 

We utilize Stable Diffusion's text tokenizer to specify text tokens (i.e., breaking the sentence into tokens). Since Stable Diffusion is configured to operate with 77 tokens, if the length of the tokenized text is less than 77, the tokenizer pads it with a special token (e.g., end of string: EOS). If its length exceeds 77, the tokenizer crops it to a length of 77. We initialize the tokens with null text (i.e., ""). So they are like placeholders in the embedding.

We hope this clarification resolves any confusion. Please feel free to let us know if any ambiguity persists.

Reviewer comment (Reproducibility):
The role of "text prompt" in the method. The authors claim that "our novel WAS-attention map to fine-tune the text embeddings, enabling each text embedding to grasp semantic information from individual segmented regions". However, I haven't found evidence from the main text to illustrate the correspondence between "text embedding" and "individual segmented regions", especially in the arbitrary granularity situation, which is one of the most important parts of this paper. Moreover, how to construct the text is also ignored.

Author response:
Thank you for your response. We present another example using a real sample [here](https://imgur.com/a/TDzUT2c), illustrating how each part is initially initialized as null text. Following optimization, each of these null texts transforms into an embedding that corresponds to a specific part in the single segmentation mask we provide as input.

If the question relates to whether the embeddings for each part represent sensible or semantically meaningful words—the short answer is not necessarily. We conducted an experiment and identified the closest words in the Stable Diffusion space to the learned embedding. The learned embeddings do not match the closest words.  Please see our response to reviewer Fjuo under *Interpretability of learned embeddings*.