Paper: SOHES: Self-supervised Open-world Hierarchical Entity Segmentation (ICLR-2024)

Reviewer comment (Theory):
Regarding the generation of pseudo-labels, I am inquisitive about why the visual feature of merged patches is computed as the sum of original features rather than an average or any other operation?

Author response:
- In fact, summing and averaging the features when merging two regions are equivalent. The reason is that, we adopt the *cosine similarity* when measuring the semantic closeness between two regions, which normalizes the features’ magnitude in computation. The sum of patch-level features within a region leads to the same cosine similarity as the average of patch-level features.

- We chose the sum over average due to its calculation and implementation simplicity. Equivalent to computing and storing the feature sum $f_k=f_i+f_j$, we can also use the feature average $\bar{f_k}=\frac{N_i\bar{f_i}+N_j\bar{f_j}}{N_i+N_j}$, where $N_i,N_j$ are the numbers of patches in region $i$ and $j$, respectively. The sum formation does not require keeping track of the patch numbers.

Reviewer comment (Experiments):
Concerning the second local re-clustering step, I'm curious about whether it has an impact on the results for large entities and how the threshold for small regions affects the final results.

Author response:
- We designed the local re-clustering to refine the pseudo-labels for small entities, because they are the most challenging to be discovered. The local re-clustering has minimal impact on the large entities. As shown in Table 2 of the original manuscript, the average recall for large entities (AR-Large) is slightly improved from 17.2 to 17.5 by the local re-clustering step. The reason for this change is that “large” entities considered by the MS-COCO evaluation metric are regions that are larger than $96\times 96$ pixels (https://cocodataset.org/#detection-eval), which could still fall under our relative area threshold $\theta_\text{small}=\frac{1}{1024}$ in some high-resolution images from SA-1B (e.g., $1500\times 8000$). Visually, these entities are not “large” compared to the whole image, but are still counted by MS-COCO style AR$_L$.

- To better understand the impacts of the hyper-parameter $\theta_\text{small}$, we run additional ablation study experiments. We observe that a threshold larger than $\frac{1}{1024}$ does not further significantly improve the coverage of small entities but would introduce more processing time.
| $\theta_\text{small}$ | 0 (No local re-clustering) | $\frac{1}{2048}$ | $\frac{1}{1024}$ | $\frac{1}{512}$ |
| --------------------- | -------------------------- | ---------------- | ---------------- | --------------- |
| AR$_S$            	| 1.9                    	| 4.3          	| 5.1          	| 5.1         	|
| Time/Image (sec)  	| 0.0                      	| 6.0          	| 8.4          	| 10.0        	|

- We appreciate the reviewer’s feedback and will integrate the explanation for the impact of $\theta_\text{small}$ into the revision.

Reviewer comment (Experiments):
In the ablation study, there is no information regarding how the ancestor prediction head contributes to the final results. Further analysis of the hierarchical architecture is needed for clarification.

Author response:
- We proposed the ancestor prediction head to perform the additional task of hierarchical segmentation, which could be considered as an add-on extending the Mask2Former segmentation model. We did not expect it to help with mask prediction, since this ancestor prediction head operates in parallel to the mask prediction head. In fact, we trained a SOHES model without the ancestor prediction head, and its performance is very close to the standard SOHES model (e.g., on SA-1B, 33.0 Mask AR w/o ancestor prediction vs. 33.3 Mask AR w/ ancestor prediction).

Reviewer comment (Novelty):
The authors claimed that “Existing segmentation models cannot predict the hierarchical relations among masks.” However, methods like Groupvit[2] already can predict the hierarchical relations among masks.

Author response:
2. Hierarchical segmentation model

- We would like to first clarify that in the original manuscript, we stated that “existing segmentation models cannot predict the hierarchical relations among masks” **in the context** of general segmentation model architectures like Mask2Former (Section 3.2). This drove us to design a novel module that could predict hierarchical structures among entities. Indeed, some *specialized* segmentation models (e.g., GroupViT [R6]) are capable of making hierarchical predictions, but do not directly fit into our context.

- More importantly, although our SOHES and GroupViT both aim to produce hierarchical segmentation masks, there are some vital differences between the two approaches:
	- SOHES directly learns from unlabeled raw images, but GroupViT is weakly supervised by text.
	- SOHES can produce a variable number of hierarchical levels, but GroupViT is limited to pre-defined levels (e.g., 2 levels).
	- SOHES can produce a variable number of whole entities, parts, and subparts, but GroupViT is limited to a fixed number of pre-defined group tokens (e.g., 64 and 8 group tokens in the first and second grouping stages, respectively).

- We appreciate the reviewer’s feedback and will revise this sentence in Section 3.2 to more precisely describe the context about the widely-applied segmentation architectures. In addition, we will cite GroupViT [R6] and include the discussion above in the revision.

Reviewer comment (Experiments):
The proposed method introduced too many hyperparameters such as Theta_{merge}, Theta_{small}, Theta_{cover}. There is not explanation for the selection of some hyperparameters, such as Theta_{cover}.

Author response:
3. Selection of hyper-parameters

- We selected these hyper-parameters for simple reasons:
	- $\theta_\text{merge}$: This hyper-parameter determines the stopping criterion in the clustering step. We chose $0.1, 0.2, 0.3, 0.4, 0.5, 0.6$ to cover as many entities as possible without exceeding a practical computation constraint. The reason for this choice was detailed in Appendix C of the original manuscript.
	- $\theta_\text{small}$: This hyper-parameter is the threshold for small pseudo-labels that are processed via local re-clustering, which aims to improve the coverage of small entities. We chose $\frac{1}{1024}$ because a) the commonly adopted MS-COCO style evaluation defines small objects as objects whose area is smaller than $32\times 32$ pixels (https://cocodataset.org/#detection-eval), b) images are resized to $1024\times 1024$ in our pseudo-label generation phase, and c) $\frac{32\times 32}{1024\times 1024}=\frac{1}{1024}$.
	- $\theta_\text{cover}$: This hyper-parameter controls the hierarchical relation test between a pair of masks. We chose 90% (rather than 100%) for better robustness. For example, even if one pixel of pseudo-label $i$ is not in pseudo-label $j$, it is still possible that $j$ is an ancestor of $i$ -- we just require that $j$ covers most of $i$’s pixels.

- To better understand the impacts of hyper-parameters $\theta_\text{small}$ and $\theta_\text{cover}$ ($\theta_\text{merge}$ has been discussed in Table 2 and Appendix C of the original manuscript), we run additional ablation study experiments.
	- $\theta_\text{small}$: We observe that a threshold larger than $\frac{1}{1024}$ does not further significantly improve the coverage of small entities but would introduce more processing time.
| $\theta_\text{small}$ | 0 (No local re-clustering) | $\frac{1}{2048}$ | $\frac{1}{1024}$ | $\frac{1}{512}$ |
| --------------------- | -------------------------- | ---------------- | ---------------- | --------------- |
| AR$_S$            	| 1.9                    	| 4.3          	| 5.1          	| 5.1         	|
| Time/Image (sec)  	| 0.0                      	| 6.0          	| 8.4          	| 10.0        	|
	- $\theta_\text{cover}$: We are not able to quantitatively evaluate different hierarchical relations among pseudo-labels, due to the lack of ground-truth annotations. However, we can examine the robustness of hierarchical relations when we alter this hyper-parameter $\theta_\text{cover}$. We compare the relations induced by varying $\theta_\text{cover}$ vs. $\theta_\text{cover}=0.9$, and count the ratio of changed relations. Indeed, when $\theta_\text{cover} \in [0.6, 0.95]$, the relations are relatively stable.
| $\theta_\text{cover}$                        	| 0.6  | 0.7  | 0.8  | 0.85 | 0.9  | 0.95 | 1.0  |
| ------------------------------------------------ | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| Relative Change w.r.t. $\theta_\text{cover}=0.9$ | 9%   | 7%   | 4%   | 2%   | 0%   | 4%   | 36%  |

- We appreciate the reviewer’s feedback and will integrate the explanation and ablation study for these choices of hyper-parameters into the revision.

Reviewer comment (Experiments):
Although the authors proposed the ancestor prediction head for the hierarchical segmentation. However, the mask prediction of masks at each hierarchy is independent. I am curious if the relation modeling of hierarchy can contribute to better mask predictions. I am also curious about the effect of just predicting the hierarchical relationships. It seems that there is no ablation to verify if the ancestor prediction head can bring the performance improvements.

Author response:
4. Ablating the ancestor prediction head

- We proposed the ancestor prediction head to perform the additional task of hierarchical segmentation, which could be considered as an add-on extending the Mask2Former segmentation model. We did not expect it to help with mask prediction, since this ancestor prediction head operates in parallel to the mask prediction head. In fact, we trained a SOHES model without the ancestor prediction head, and its performance is very close to the standard SOHES model (e.g., on SA-1B, 33.0 Mask AR w/o ancestor prediction vs. 33.3 Mask AR w/ ancestor prediction).

- By “just predicting the hierarchical relationships,” we can only create a hierarchical structure of the queries in Mask2Former. However, these queries do not have any physical meaning before we use the mask prediction head to instantiate their corresponding masks in the given image. Thus, we can only predict the hierarchical relations in conjunction with the mask prediction, but not using the ancestor prediction head alone.

---

We appreciate your time in reviewing our work and participating in the discussion. We are glad that our responses have addressed most of your issues.

Regarding the ancestor prediction head, we would like to kindly emphasize the novelty and significance in proposing this module: Although it does not greatly increase the Mask AR score, the ancestor prediction head has accomplished its key function – to produce hierarchical relations between segmentation masks, leading to a structured analysis of entities and their constituent parts in the given image. **By proposing the ancestor prediction, we enable a brand new capability of hierarchical segmentation which did not exist in the current self-supervised object discovery literature [R1-5].**

Reviewer comment (Novelty):
[Technical Contributions] The three phases proposed in this work are very similar to the Cut-and-Learn pipeline proposed by CutLER [1]. Self-exploration is pretty similar to the MaskCut stage in CutLER, which also leverages DINO feature for pseudo-label generation. Self-instruction is the same as the LEARN process of CutLER, which trains a model on pseudo-labels. And, the Self-correction stage can be viewed as a variant of CutLER's multi-round self-training, but with a teacher-student framework. I agree that while there are some implementation differences between the stages in SOHES and CutLER, their core concepts are largely analogous.

Author response:
1. Technical contributions

- We agree with the reviewer’s perspective, but we would like to point out that such a multi-phase learning paradigm (e.g., discovering pseudo-labels and then learning a detection/segmentation model) is a general pipeline widely-adopted in the existing self-supervised object localization/discovery literature. Indeed, not only does CutLER [R5] (as suggested by the reviewer), but all recent methods also follow this pipeline including LOST [R1], TokenCut [R2], FreeSOLO [R3] (as mentioned by Reviewer vWUT), and FOUND [R4]. That means, the contributions of these different methods do not lie in the introduction of a new pipeline, but instead in proposing unique components to improve this established pipeline for self-supervised object localization/discovery.

- Our SOHES similarly adopts this general multi-phase pipeline, but makes significant contributions through **novel designs for each phase** involved in the learning process, including a global-to-local clustering algorithm for pseudo-label generation, a hierarchical relation learning module, and a teacher-student self-correction phase. In particular, none of the previously mentioned methods [R1-5] consider hierarchical structures among objects/entities, while learning such structures is involved throughout the learning process of SOHES. Given these technical novelties and advancements and our substantial performance improvements over CutLER, we respectfully disagree with the reviewer that the distinctions between our SOHES and CutLER are merely minor implementation differences.

- We appreciate the reviewer’s feedback and will revise the Related Work section and the beginning of the Approach section, to better reflect this general pipeline and our novel components proposed in this work with a hierarchical perspective.

Reviewer comment (Evaluation):
[Model performance] SOHES performs much worse than CutLER (9.8 vs. 2.1 on COCO and 3.6 vs. 1.9 on LVIS) in terms of the mask AP. This work shows stronger performance than the previous SOTA CutLER and SAM on some benchmarks, however, the main results are AR (averaged recall). Recall is important, however, for many downstream tasks, the AP is still the most valuable evaluation metric.

Author response:
2. Model performance

We would like to first clarify that, **regarding the open-world segmentation task on MS-COCO/LVIS**, the computed AP number is misleading and cannot accurately reflect the model’s true segmentation performance. **The observed lower AP in our results compared with CutLER’s on MS-COCO/LVIS is a consequence of limitations in the annotations within these datasets**. In contrast, on the SA-1B dataset which mitigates these annotation limitations, our AP (12.9) significantly surpasses CutLER’AP (7.8). Below we explain in detail.

- We chose average recall (AR) over average precision (AP) for the following reason: When evaluating open-world segmentation models (which try to segment “everything”) on datasets with closed-world annotations (such as MS-COCO/LVIS which only include a pre-defined, limited set of entity classes), AP would penalize model predictions that are actually valid entities, but just not annotated by the dataset. Therefore, when the dataset annotations cannot cover all entities, AP becomes misleading for judging the performance of open-world models. Note that this choice of AR over AP is also commonly adopted in the open-world literature. Examples include (but are not limited to) detection [R6], segmentation [R7], and tracking [R8].

- As an example (shown in Table 5), SAM’s AP is lower than CutLER on MS-COCO (6.1 vs. 9.8) and PartImageNet (3.4 vs. 5.3), which definitely cannot imply SAM is a model inferior to CutLER. The lower AP of SAM is due to insufficient annotations in these two datasets, not the capability of SAM. Meanwhile, as the annotated entities increase, AP becomes more and more trustworthy. For instance, AP on SA-1B (the most densely annotated dataset evaluated in our work) matches qualitative comparison and follows the same trend as AR (FreeSOLO < CutLER < SOHES < SAM).

Reviewer comment (Experiments):
[Unfair comparison] The main baseline CutLER used Cascade Mask RCNN as the segmentation model, while this work used Mask2Former, a stronger segmentation model, as the base model. This makes the performance comparison unfair.

Author response:
3. Unfair comparison

- We chose Mask2Former as our segmentation model mainly for making hierarchical predictions. In Cascade Mask R-CNN, each proposal is predicted independently, and therefore, analyzing hierarchical relations between entities would be challenging in Cascade Mask R-CNN. In contrast, the attention modules in Mask2Former allow information exchange among queries, so we can build our ancestor prediction head on Mask2Former.

- For a more comprehensive comparison with CutLER, we train a Cascade Mask R-CNN segmentation model using SOHES without the entity hierarchy. This model still significantly outperforms CutLER with the same segmentation model architecture.
| Method | Segmentation Model | LVIS | Entity | SA-1B |
| ------ | ------------------ | ---- | ------ | ----- |
| CutLER | Cascade Mask R-CNN | 20.2 | 23.1   | 17.0  |
| SOHES  | Cascade Mask R-CNN | 27.0 | 29.2   | 31.9  |
| SOHES  | Mask2Former    	| 29.1 | 33.5   | 33.3  |