Paper: ISCUTE: Instance Segmentation of Cables Using Text Embedding (ICLR-2024)

Reviewer comment (Novelty):
Utilizing the combination of two powerful models, CLIPSeg and SAM, may be effective but not novel.

Author response:
We are sincerely grateful for your constructive comments and suggestions, as they play a crucial role in helping us improve our paper. 
We would like to draw attention to the fact that the novelty of our method lies in the adapter model we introduced. As discussed extensively in Sections 1 and 2.2, CLIPSeg and SAM are not capable of performing one-shot instance segmentation out-of-the-box, with the former one unable to perform instance segmentation at all and the latter requiring a laborious process of manual prompting. Furthermore, quoting from the SAM paper (section 8: limitations) “While SAM can perform many tasks, it is unclear how to design simple prompts that implement semantic and panoptic segmentation.” Another limitation of SAM is that it does not introduce any method to filter masks we do not need. The main novelty we introduce, overcomes both these limitations by presenting a simple way to prompt it, as well as a method to filter out masks that we do not require.
Another point we wish to highlight is that we use the output of the PENULTIMATE layer of CLIPSeg and not the final semantic segmentation it generates. Our presented prompt encoder network transforms this embedding space directly into SAM’s prompt embedding space, without the need to sample individual pixels.

Reviewer comment (Writing):
The design motivation in the 3.1.2 section (i.e., MLP, cross-attention, self-attention) is missing.

Author response:
Q1) We wish to thank you for pointing out a valid deficiency in our paper. We have carefully incorporated your suggestions in the revised paper. We have added a reference to Appendix C (Ablation study) in section 3. This appendix discusses the design motivation for each individual block, as well as the impact of removing it from the presented model. We wish to note that all the additions and changes were made based on extensive experiments carried out in each configuration.

Reviewer comment (Experiments):
Few baselines. The only other method mentioned is RT-DLO. Considering the author is leveraging strong semantic segmentation methods, including SAM, they should compare their method with those segmentation methods.

Author response:
Q2) The RT-DLO method compares DLO-specific instance segmentation methods with other non-DLO specific baselines and shows a major improvement in comparison. We present a method that exceeds all the previously presented DLO-specific instance segmentation models, including RT-DLO, mBEST, FASTDLO, etc. (refer to table 2). 
Additionally, it's crucial to highlight that the CLIPSeg model does not execute instance segmentation, as detailed in Section 1. Similarly, SAM lacks a direct mechanism for DLO instance segmentation without manual point prompting. The generation of instance masks using this method heavily depends on the precision of point prompts, as depicted in Figure 2 of our paper. This direct application is also impractical in real-world scenarios for the same reasons.
Another notable concern is the shortage of DLO instance segmentation baselines for meaningful comparisons. Due to the novelty of this field, there is a lack of robust datasets available for comprehensive research. This scarcity underscores the importance of the dataset we introduce alongside our method, as we anticipate it will expedite progress in DLO instance segmentation research.

Reviewer comment (Writing):
Please explain the motivation for components proposed in the 3.1.2 section.

Author response:
- The motivation for each independent network has been discussed in the paper as well as our comments. The proposed prompt encoder network consists of just 4 single-layer blocks, each performing a subprocess in the main process of point prompt filtering. The proposed classifier network is initially motivated by similar classifier networks introduced in works such as DETR[1] and MaskFormer[2], where trainable tokens were used to classify N+1 object classes.

Reviewer comment (Experiments):
Please compare the proposed method with SAM, CLIPSeg, and other strong segmentation models by inference on DLO benchmarks.

Author response:
Q2) The RT-DLO method compares DLO-specific instance segmentation methods with other non-DLO specific baselines and shows a major improvement in comparison. We present a method that exceeds all the previously presented DLO-specific instance segmentation models, including RT-DLO, mBEST, FASTDLO, etc. (refer to table 2). 
Additionally, it's crucial to highlight that the CLIPSeg model does not execute instance segmentation, as detailed in Section 1. Similarly, SAM lacks a direct mechanism for DLO instance segmentation without manual point prompting. The generation of instance masks using this method heavily depends on the precision of point prompts, as depicted in Figure 2 of our paper. This direct application is also impractical in real-world scenarios for the same reasons.
Another notable concern is the shortage of DLO instance segmentation baselines for meaningful comparisons. Due to the novelty of this field, there is a lack of robust datasets available for comprehensive research. This scarcity underscores the importance of the dataset we introduce alongside our method, as we anticipate it will expedite progress in DLO instance segmentation research.

Reviewer comment (Experiments):
Please conduct ablation studies, including quantitative and qualitative analysis.

Author response:
Q3) Thank you for your constructive comments and suggestions, and they are exceedingly helpful for us to improve our paper. We have carefully incorporated them in the revised paper. Please find the updates in Appendix C “Ablation Study”.

Reviewer comment (Evaluation):
It seems that the proposed method relies on the assumption that if properly prompted, SAM can always provide correct cable segmentation masks. As the authors claimed, the performance upper-bound is limited by SAM and CLIPSeg. I wonder what is the exact upper-bound of these two methods, and how close can the proposed method reach this bound?

Author response:
Q1) This is a great question, and it needs some more exploration. From our experiments (results in table 1 “Oracle”), we observed that with the proposed architecture and using both the models in their frozen state, the method achieves an mIoU of 92.5% and 92.1% with and without augmentation respectively. We trained this method multiple times (by increasing the representation power and depth of the prompt encoder network as well) and tested it under the oracle setup. The variance of the results was very low. In table 2 as well, the “A+O” and “B+O” configurations have very close results across the board, leading us to conclude that this is an upper bound on the performance our method can achieve, which can only be traced back to the frozen foundation models.

Reviewer comment (Writing):
Typo in Section 2.2: ” ... that have historically have been difficult to segment ...”

Author response:
We appreciate your valuable feedback and suggestions; they greatly contribute to enhancing our paper. The typos mentioned in the comments have been fixed in the updated PDF.

Reviewer comment (Theory):
What is the exact upper-bound of these two methods, and how close can the proposed method reach this bound?

Author response:
Q1) This is a great question, and it needs some more exploration. From our experiments (results in table 1 “Oracle”), we observed that with the proposed architecture and using both the models in their frozen state, the method achieves an mIoU of 92.5% and 92.1% with and without augmentation respectively. We trained this method multiple times (by increasing the representation power and depth of the prompt encoder network as well) and tested it under the oracle setup. The variance of the results was very low. In table 2 as well, the “A+O” and “B+O” configurations have very close results across the board, leading us to conclude that this is an upper bound on the performance our method can achieve, which can only be traced back to the frozen foundation models.

Reviewer comment (Evaluation):
How much time does the proposed method need to take for DLO segmentation?

Author response:
Q2) The runtime of our method is 330[ms], per image (1920x1080) on a single Nvidia RTX 2080 GPU; and 250[ms] on a single Nvidia A5000 GPU. This can mainly be attributed to the image encoder in SAM. We would like to add that unlike other baselines mentioned in our paper, our model’s runtime is independent of the number of DLOs or DLO intersections in the image. The method we propose focuses mainly on quality improvement of the instance segmentation of DLOs as well as the generalization capabilities of the model, using classical computer vision methods

Reviewer comment (Experiments):
Although the experiments have shown that the proposed approaches work well in many cases, it is still unclear how well they can generalize. The dataset used for evaluation consists of only 4 colors and all images are high resolution (1920x1080), with cables placed at a similar distance to the camera and limited appearance variation. This simplifies the problem significantly and may affect model performance when tested on more realistic scenarios, such as "cables in the wild" with diverse colors, twisted shapes, greater distances from the camera, or larger appearance variance in training and testing sets. It would be helpful to see how the model performs under such cases to better understand its behavior on different data.

Author response:
The generated data indeed shares a similar nature, featuring cables selected from a color range commonly found in industrial environments, including hues like yellow and black. Each image adheres to a standardized resolution of 1920x1080 and has served as the basis for our training. The results presented in Table 1 presents the outcomes derived from this dataset.
Nevertheless, we wish to highlight the findings outlined in Table 2. The datasets referenced as C1, C2, and so forth, originate from the RT-DLO and mBEST papers. Notably, our model has not been exposed to or trained on these datasets that exhibit scenarios, colors, and shapes of DLOs unlike our training dataset. The "C[i]" datasets maintain a resolution of 640x360, while the "S[i]" datasets contain images of size 896 × 672. The colors of DLOs in these images as well as the backgrounds and scenarios of these images also differ from those we generated, as depicted in figures 4 (a), (b), (c), 5, and 11.
Additionally, in figure 6, we present results pertaining to "cables in the wild" scenarios, showcasing genuine cable images captured using a smartphone camera in our laboratory. Employing our trained model, we conducted a "zero-shot" transfer to all the aforementioned scenarios.

Reviewer comment (Experiments):
The text prompts evaluated are quite limited. Only 3 choices are used, which seem not to have enough coverage. Moreover, all 3 text prompts are single words functioning as class labels. Given that both the CLIPSeg and SAM models are very strong at recognizing a broad range of textual concepts, it would be valuable to know how the proposed model reacts to more complex, detailed text prompts, and whether this could improve or reduce model quality.

Author response:
As mentioned in section 6, we are currently in the process of experimenting the effectiveness of our method to standard rigid-object instance segmentation. This involves training a general instance segmentation model with a wide variety of text-image pairs, as done in CLIP. Another direction we are currently exploring is the application of our method to more specific prompts such as “blue HDMI cable”, “black LAN cable”, etc. The presented research is aimed to solve one of the major problems in robotics research (DLO instance segmentation). We harness the power of SAM and CLIPSeg, with novel adapter model to control SAM’s segmentation output and to prompt it effectively.
Furthermore, although CLIPSeg is capable of processing more complicated text prompts, it is most effective on simple “class-like” text prompts as you can see in the Figure 14 of our updated PDF.

Reviewer comment (Evaluation):
In page 7, the authors claim "we observe that the model generalizes better if it is trained using augmentations in the dataset." However, the improvement is very marginal on some data (e.g., from 97.01 to 97.71) and larger on other datasets. An explanation for this variation would be helpful.

Author response:
The influence of augmentation on generalizability is closely tied to the types of augmentations employed in our approach. During training, we adopt a randomized selection of augmentations in the pixel domain, encompassing techniques like grayscale, Gaussian blur, color jitter, and more. These augmentations play a crucial role in improving the model's ability to generalize across images with varying resolutions and diverse colors of DLOs, up to the upper limit set by SAM's performance. Notably, results without augmentation already closely approach this upper bound, indicating limited additional improvement with augmentation.
However, the challenge lies in generalizing to the intricate shapes of DLOs, particularly those with numerous loops, which cannot be addressed through image-based augmentations. This presents an opportunity to highlight the complexities in segmenting DLOs compared to more rigid objects.
Additionally, these findings can indicate that the dataset we generated may be reasonably homogeneous and reasonably representative of the data distribution in question. The details of this dataset, as outlined in the paper, will be made available upon publication.

Reviewer comment (Writing):
Some sentences in the text are broken:  
- In Section 2.2, "However,Struggingle to" should be corrected.  
- In Section 2.2, "its prompt encoder with a single capable" should be corrected.

Author response:
Thank you for pointing it out. These typos have been fixed.