Paper: Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervision (ICLR-2024)

Reviewer comment (Experiments):
While the Multiscale Combinatorial Grouping (MCG) approach was proposed in 2016, it might be beneficial to consider the use of more recent methods, such as the Segmentation Attention Module (SAM), to enhance the generation of higher-quality masks for this problem. The integration of SAM into the existing framework could potentially improve the performance and accuracy of mask generation.

Author response:
We thank the reviewer for posing this idea. We unfortunately couldn’t find a canonical reference for the suggested “segmentation attention module”, closest works are [1] and [2] - both of which are trained on supervised, closed-world annotated images and hence not applicable to generate part-masks in an open-world setting.

In case you meant Segment Anything Model (SAM)[3], we note that foundational models like SAM are reliant on millions of supervised images during training, making direct comparison of their open-world capabilities with ours difficult. Particularly, SAM's training data lacks category labels, making it infeasible to construct an evaluation set for categories not included in SAM's training data. However, UDOS is perfectly compatible to incorporate advances like SAM by replacing the initial super-pixel supervision with that generated by SAM, opening up exciting possibilities for enhanced open-world segmentation that builds upon UDOS in the future.

[1] Jiang, Junzhe, et al. "DSA: Deformable Segmentation Attention for Multi-Scale Fisheye Image Segmentation." Electronics 12.19 (2023): 4059.

[2] Gou, Yuchuan, et al. "Segattngan: Text to image generation with segmentation attention." arXiv preprint arXiv:2005.12444 (2020).

[3] Kirillov, Alexander, et al. "Segment anything." ICCV  (2023).

Reviewer comment (Experiments):
In order to provide a comprehensive evaluation of the proposed approach, it would be valuable to compare it with relevant open-world panoptic segmentation techniques, such as ODISE (Open-vocabulary DIffusion-based panoptic SEgmentation). The inclusion of a comparative analysis with ODISE would enable a thorough assessment of the strengths and weaknesses of the proposed method and offer insights into its effectiveness in handling open-world scenarios.

Author response:
Thank you for bringing this prior work to our notice. We note that ODISE and UDOS are fundamentally different tasks. While ODISE is designed for open-vocabulary panoptic segmentation which requires labeling new classes in-the-wild, UDOS is focused on open-world instance segmentation which aims at detecting categories unseen during training (that is, objects with no masks during train time). Further, ODISE still employs seen and annotated instances for training the mask module using binary classification loss, which we already demonstrate does not generalize as well to open-classes as UDOS (MaskRCNN baseline in Tab 2 and Tab 4). We will include a discussion around this difference with ODISE in the updated version of the paper.

Reviewer comment (Experiments):
This paper generate candidate object regions through unsupervised segmentation methods. However, it cannot be guaranteed that these unsupervised methods can generate object regions that cover all regions. Especially when the number of categories increases, I question the performance of the unsupervised segmentation methods. The author should provide: 1) the specific performance of the unsupervised segmentation methods, 2) experimental comparison with existing methods when categories are more, like COCO to LVIS.

Author response:
Thanks for raising this pertinent question! We show the segmentation performance of using only the unsupervised segmentation methods in Tab 2 of our paper. Specifically, the most competitive unsupervised method MCG yields 23.6 box AR100 compared to the most competitive MaskRCNN baseline of 25.6, indicating its effectiveness in detecting open-set instances.

Also, we do agree that the unsupervised methods do not necessarily cover all the objects in the region, but as shown in our qualitative illustrations in Fig 5 in main paper and Fig 3,4 in the supplementary, UDOS does manage to detect many objects missed by MaskRCNN. This property is also reflected in the SOTA performances achieved by UDOS across the board (Tab 2 and Tab 4), with upto 5% improvements in maskAR100 compared to next best method.

> **Effectiveness of UDOS on many-class settings**

We note that the UVO dataset used in our paper is exhaustively annotated to cover every object in the image across several types of categories and scenes on a large-scale. It is notable that many of these categories are not even covered in the 1.2k sized LVIS taxonomy at all, as shown in [1]. On this dataset, UDOS achieves SOTA results outperforming all prior methods (COCO -> UVO, Table 4), effectively demonstrating UDOS's capabilities in handling open-world scenarios where a wide range of unseen objects may be present.

Reviewer comment (Evaluation):
The author should provide more result metrics with previous methods. For example, LDET also provides AP, AR10. The author should provide related performance comparisons to provide more comprehensive results.

Author response:
Using AP/AR10/AR100, UDOS gives 2.8/15.0/31.6 while LDET gives 5.0/16.3/27.4 on the VOC to NonVOC setting. However, note that on COCO with non-exhaustive annotations, evaluating precision (AP) or low AR@K (like AR10), may unfairly punish detecting valid, yet un-annotated objects. We remark that other prior works in open-world segmentation (including OLN and GGN) also often choose not to report AP on cross-category generalization for this reason. We will add this point to the paper, along with the AP result.

Reviewer comment (Experiments):
[A] also proproses a CLN (region proposal generation algorithm). What's about performance comparision with this work.

Author response:
We thank the reviewer for bringing this paper to our attention. While the region proposal generation algorithm in CLN employs label fusion from multiple closed-world datasets to mimic an open-world setting, this approach may not guarantee true open-world capabilities, as classes not included in any training datasets could still be overlooked. In contrast, our annotation-free and vocabulary-free segmentation method utilizes bottom-up low-level grouping to achieve state-of-the-art open-world capabilities with a remarkably simple framework. We will add this citation and relevant discussion to the paper.

Reviewer comment (Novelty):
What's about the details about Refinement module? I feel that this is all about previous methods, no matter the objectness ranking and inference.

Author response:
Our refinement method follows the mask and box prediction heads from previous methods only in design, but significantly differs in the purpose served. Specifically, prior methods feed the region proposals for mask prediction, while we feed the grouped part-masks as input to the refinement module for correcting noisy part-mask groupings. The significance of our refinement module is highlighted qualitatively in Tab.5a (+1% improvement in AR100) and qualitatively in Fig. 1 and Fig 2 in the supplementary through several visualizations.  It is also notable that just adding this refinement head to baseline MaskRCNN (_MaskRCNN_sc_ in Tab 2 and Tab 4) without the bottom-up grouping significantly underperforms UDOS by atleast 8% in AR value.

Reviewer comment (Experiments):
Is there any time-consuming experiments on the cluster in the grouping module? Because the similarity is calculated two-by-two.

Author response:
Our UDOS framework utilizes a fast and efficient agglomerative clustering algorithm from scikit-learn, *ensuring minimal time and memory overhead* even when handling hundreds of part-masks. While standard MaskRCNN training on 8 Nvidia A10 GPUs takes about 22 hours, UDOS with its grouping and refinement module takes less than 24 hours. As already noted in the paper (sec 4.4), UDOS takes 0.13sec/image (7 FPS) during inference compared to MaskRCNN's 0.09 sec/image (11 FPS), while delivering remarkably higher accuracy.

Reviewer comment (Experiments):
As we know, the segment anything (SAM[1]) has high generalizability in category-free segmentation task. It is a foundation model pretrained in many data, but its zero-shot ability is strong without fine-tune in specific datasets in category-free segmentation task, so I think the comparison is necessary. Can this have higher recall that SAM? If not, please discuss on the contribution.

Author response:
This is a great question, thanks for raising this comment! Although foundational models like SAM expand the potential for segmentation, their reliance on vast amounts of supervised images during training impedes a fair comparison of their open-world capabilities with ours. Particularly, SAM's training data lacks category labels, making it infeasible to construct an evaluation set for categories not included in SAM's training data. However, note that UDOS is perfectly compatible to incorporate advances like SAM by replacing the initial super-pixel supervision with that generated by SAM, opening up exciting possibilities for enhanced open-world segmentation. We will add a citation to SAM, along with this discussion, in the final version.

Reviewer comment (Experiments):
Why exclude part masks from U that overlap with any ground truth mask in S with an IoU greater than 0.9? Please discuss on it with experiments.

Author response:
We remove part masks from U with significant overlap with those in the annotated set S to avoid possibly redundant supervision. Removing these masks using a threshold of 0.9 made no change to the resulting AR values, hence is optional. We will add this remark to the paper.

Reviewer comment (Experiments):
How about the grouping result on these situations: two same-category instances are close (or overlap), two instance with similar color, two hierarchical categories (e.g. clothes and person).

Author response:
This is a great point raised by the reviewer! We thank you for suggesting a novel protocol to construct evaluations, we believe that following these suggestions and conducting large-scale studies on failure modes of existing systems in the open-world can be a new contribution in itself! In current setting, despite the difficulty in quantitative evaluations, we still aim to provide understanding of these cases qualitatively in the main paper and the supplementary through several visualizations where UDOS shows its prominence in all those cases, such as:

- _Similar category objects close to each other_: The glasses on the table (Fig 5 in main paper) and the people on the elephant (Fig 1 in the supplementary) are segmented with separate masks even when they are close together.
- _Similar colored instances_: The two cabs (Fig 1 in the supplementary) are segmented separately.
- _Hierarchical categorization_: The dress and the boy (Fig 1 in the supplementary), the uniform and the baseball player (Fig 5 in main paper) as well as the truck and its wheel (Fig 1 in the supplementary) are segmented together. Note that it is in general difficult to resolve this ambiguity in an open-world setting without the knowledge of real world categorization of the class hierarchies.

Reviewer comment (Experiments):
The bottom-up module is quite complex, involving a few components. Although the authors conducted ablation experiments to justify some design choices, it is not clear why part-segmentation and grouping work better than other baseline approaches. Part-segmentation + grouping appeared in the literature a long time ago and researchers abandoned this idea. Current experiments in this paper do not convince me that this is actually a better idea for open-world segmentation.

Author response:
> **why should part-segmentation and grouping be preferred compared to class-agnostic training?**

Firstly, the suggested class-agnostic instance segmentation is *already included as one of our baselines* (named _MaskRCNN_, and related variants), for cross-category setting (using VOC-COCO annotations) in Table 2 and cross-dataset setting (using all COCO annotations) in Table 4. Despite being a strong baseline, the class-agnostic models still suffered from weaker performance in open-world instance segmentation literature as also shown by other works in this field in OLN, LDET and GGN. We emphasize that UDOS *outperforms this exact baseline suggested by the reviewer by 8.4%* in cross-category and *6% in cross-dataset* settings. We refer the reviewer to Table 2, Table 4 and section 4.1 where we explicitly describe the baselines, including the class-agnostic model, a few improved variants of it as well as the state-of-the-art methods from recent literature.

> **Prior works in part-segmentation + grouping**
1. Reviewer mentions that part-segmentation and grouping methods appeared in literature a long time ago but failed to provide any specific references. Thus, we cannot provide a detailed comparison. We do, however, like to refer reviewers to our related work section on how our work is related to prior works for segmentation.
2. In addition, the reviewer mentions that the idea of part-segmentation + grouping has been attempted before and has been abandoned. We respectfully disagree that this is a weakness and a justification for the low rating of our work. Firstly, we emphasize that our approach stands out from prior works due to two novel design choices: (i) *employing weak supervision from bottom-up part segments* for handling unseen instances and (ii) *integrating a refinement module* within an end-to-end framework for improving noisy part-mask groupings. These contributions *are validated both quantitatively (Table 5a and 5c) and qualitatively (Figure 5 and supplementary Figures 3-4)* in our paper. Second, we do not believe that unsuccessful prior attempts with conceptually similar ideas diminishes the value of new works. In fact, our strong empirical performance suggests that combining part segmentation and grouping indeed achieves strong performance in the open-world. 

> **Current experiment setting is not convincing**

We thank the reviewer for noting their concern. We note that the current experiments and baselines in the paper are very comprehensive, covering diverse datasets (COCO, LVIS, UVO, ADE and OpenImages) and transfer settings (cross-category and cross-dataset), which were also used in all prior works in open-world instance segmentation (LDET, OLN, GGN). *UDOS comprehensively outperforms all these methods on all the settings*, thus highlighting the applicability of our approach for open world segmentation. In addition, the ablations clearly justify several of our design choices, such as the use of part-mask supervision or the grouping and refinement modules.

Reviewer comment (Experiments):
Even for top-down methods, Mask2former etc. will be a much better choice. I see that the authors might argue that the proposed method can use any other top-down method to replace Mask RCNN. But still, why choose MaskRCNN in the first place? Using a more recent method will make the experiment results more convincing.

Author response:
> **Choice of MaskRCNN as the backbone**

We employ MaskRCNN to enable fair comparisons with prior works such as LDET, GGN and OLN - all of which use MaskRCNN for open-world segmentation. We also highlight that despite employing a basic top-down architecture like MaskRCNN and superpixel method like MCG, our *UDOS framework already outperforms the best prior method* by 2% maskAR100 for cross-category (Tab 2) and upto 5% maskAR100 in cross-dataset settings (Tab 4). Moreover, as rightly noted, UDOS framework is designed to integrate seamlessly with any segmentation architecture, enabling it to incorporate recent and future advancements to potentially improve results further.