Paper: AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation (ICLR-2024)

Reviewer comment (Experiments):
How does the model correct wrongly segmented object instance? For example, in Figure 6, if the initial segmentation wrongly groups two chairs into one instance, how can a later click fix this mistake?

Author response:
Sorry if this was unclear. If the model produces a wrongly segmented object instance, users are expected to provide assistive clicks on the error regions. Then the model will take as input the newly added clicks together with the previous clicks and the 3D point cloud and produce an updated segmentation mask. Each click $c_k = ${$p_k, o_k $}  (where $k=1, 2, …$ is the click index) has two attributes, one is its coordinates $p_k$, and another one is its region index $o_k$. $o_k = 0$ indicates the click comes from the background, $o_k=m (m>0)$ indicates the click comes from object $m$.

In Figure 6, **If** the user starts with one click $c_1= ${$p_1, o_1=1$} on chair 1 but the model wrongly groups chair 1 and chair 2 into one instance, then the error region is chair 2 and the user is expected to provide a second click $c_2 = ${$p_2, o_2=2$} on chair 2. After that, $c_1$ and $c_2$ are converted to queries and go through the click attention module and each click produces one region mask logits:

-$c_1$ produces $\mathcal{R}_{1} \in \mathbb{R}^{ {N}'}$, ${N}'$ is the number of points of the point cloud. 

-$c_2$ produces $\mathcal{R}_{2} \in \mathbb{R}^{ {N}'}$ 

-We also have learned background mask logits $\mathcal{R}_{0} \in \mathbb{R}^{ {N}'}$.

The above three mask logits are concatenated and softmaxed. Finally, an argmax will give a corrected segmentation map assigning each point a label belonging to [$0$ (background),$1$ (chair 1), $2$ (chair 2)].

Reviewer comment (Evaluation):
In Table 9, results 2 and 7 show minor performance drops without the C2C attention and temporal encoding components. Does this indicates that modeling the relations between clicks is less important for the proposed method?

Author response:
Thanks for the reviewer’s careful observation. We agree that the self-attention between click queries (C2C attention) has a less pronounced effect compared to the cross-attention between click queries and point features, i.e., C2S attn and S2C attn. However, it does not mean that C2C attention is not important or can be safely removed. The ablation results in Table 9 are conducted in a setting where the model is trained on ScanNet40-train and evaluated on ScanNet40-val (smaller domain gap). We empirically find that C2C attn has a significant effect in the setting ScanNet40 (Training) -> KITTI-360 (Testing), **e.g. 37.4 (with C2C) vs 33.8 (wo. C2C)** on  $\overline{\textrm{IoU}}@\overline{\textrm{5}}$. Similarly, we find that the temporal encoding (TE) has an obvious effect on KITTI-360, especially for long click sequences, **e.g. 40.5 (with TE) vs 42.3 (wo. TE) on $\overline{\textrm{IoU}}@\overline{\textrm{10}}$**. This indicates that **C2C and temporal encoding play crucial roles in improving the model’s generalization ability**. **We have updated the manuscript in page 9 to highlight this point**.

Reviewer comment (Presentation):
Discussion of limitations/drawbacks should be put in the main body of the manuscript instead of the appendix for a fairer portrayal.

Author response:
Thank you for the suggestion. We have now moved the limitations from the appendix to the main paper.

Reviewer comment (Evaluation):
More failure cases need to be showcased and analyzed if any, otherwise the paper seems to focus too much on the advantages of the presented method and does not always give the whole picture.

Author response:
Thank you, we agree that we did not pay enough attention to this. We have added failure cases and an accompanying discussion in the main paper, please see the updated manuscript for details. We hope the analysis will motivate further research into the under-explored topic of interactive 3D segmentation.

Reviewer comment (Novelty):
Lack of Novelty: For the proposed methods, the novel point is the multi-object segmentation, and the fast inference is also a novel point to some degree. And for efficiency, the submission does not present a technical point on how to improve efficiency. Multi-object segmentation is also achieved by the previous methods (InterObject3D) via some small improvements. Actually, the performance on the multi-object segmentation is not the best, according to Table 4.

Author response:
We thank the reviewer for spending time reviewing our paper, and for the advice regarding related work and failure cases. **We do, however, disagree with some of the other comments and believe they must be due to misunderstandings**. We therefore respond below to every individual question/concern:

**1. Lack of Novelty**
> For the proposed methods, the novel point is the multi-object segmentation, and the fast inference is also a novel point to some degree.

We are not sure what is meant here. The reviewer starts by saying, in their first sentence, that our multi-object segmentation and fast inference are novel. But then goes on to criticize a lack of novelty. We can only point out once more that

(1) as far as we know, our work is the first to do interactive multi-object segmentation of 3D point clouds.

(2) our architecture is significantly different from state-of-the-art (see Fig. 1).

(3) our method yields high-quality instance masks with less human effort (fewer clicks), and also offers faster inference; thus making the labeling of 3D segmentation masks much more efficient.

> And for efficiency, the submission does not present a technical point on how to improve efficiency

**We respectfully disagree, this is not correct.**  The efficiency of our method comes from two sources: (1) an architecture designed with fast inference in mind; and (2) more importantly, fewer user clicks due to multi-object handling. In detail: 

(1) our proposed **architecture** minimizes computational cost, as described in the introduction and illustrated in Fig.1: the feature backbone only needs to be run once for a point cloud, whereas clicks are processed separately, so that for every interaction one must only run the lightweight decoder (not the complete forward pass, like for instance InterObject3D). The comparison in Sec. 4.2, respectively Tab. 5 confirms that our **inference is 2× faster** than the baselines.

(2) The **multi-object formulation** allows for concurrent segmentation of several objects (as opposed to prior work that processes one object at a time). Our scheme therefore needs fewer clicks than the baselines to achieve high IoU. In interactive segmentation, the crucial point is to **gain efficiency by saving user clicks**. 

In this context **we note that our view is stand by all other reviewers**:

-reviewer **PRcH**: *“The model ... champions efficiency, demanding fewer user inputs and providing swifter inference times”, “detailed computational cost comparisons”, “diminished user interaction and expedited inference ... is empirically substantiated.”*

-reviewer **3vqw**: *“the proposed approach can reduce annotation time”*

-reviewer **Gxfu**: *“the author performed Efficiency Comparison, which is necessary for user-interactive tasks.”*

>Multi-object segmentation is also achieved by the previous methods (InterObject3D) via some small improvements.

**Also here we disagree**. Perhaps there was misunderstanding because we adapted InterObject3D so that it could serve as a baseline? Since there isn’t any prior work about multi-object interactive segmentation, we had no baseline to compare to. Therefore, we upgraded InterObject3D, the state-of-the-art for single-object segmentation, such that it could be evaluated by our protocol. The “baseline” paragraph in Sec. 4 explains our modifications. But although we modified InterObject3D for evaluation on our multi-object benchmark **it can, by design, only predict a binary mask for a single object** in every pass through the network, and those masks must be merged into a multi-object segmentation either heuristically or by hand.

>Actually, the performance on the multi-object segmentation is not the best, according to Table 4.

**This is not true**. We are not sure how the reviewer arrived at that conclusion. The fact is that **AGILE3D outperforms previous methods** by a significant margin reaching “*top-tier results*” [reviewer **PRcH**] and “*state-of-the-art performance*” [reviewer **Gxfu**]. Table 4 shows 6 metrics for each of the 3 datasets. Among the 18 comparisons in total, AGILE3D outperforms standard InterObject3D **in all cases**, and outperforms our enhanced InterObject3D++ in **17 out of 18 cases**. Please do not only check the metrics in isolation but overall for KITTI-360, in that particular case AGILE3D surpasses the baseline by **a factor of 4** and the enhanced baseline by **a factor of 2** on $\overline{\textrm{IoU}}@\overline{\textrm{5}}$.

Reviewer comment (Evaluation):
The running time should be reported in a fair manner; for table 5, it is not clear how to perform the comparison since the manuscript claims efficiency as a contribution. Or, the user study is a good choice to compare the efficiency. In some challenging cases, there are a lot of occlusions between the segmented object and other objects in the scene.

Author response:
**3. The running time should be reported in a fair manner**

We fully agree, and did our best to ensure a fair comparison. Run times for all methods were measured in the **same setting** on the **same hardware** (a single TITAN RTX GPU).

>for table 5, it is not clear how to perform the comparison since the manuscript claims efficiency as a contribution.

The paragraph “efficiency comparison” in Sec. 4.2 describes this. **Inference time is measured after all objects in a given scene have received 5/10/15 clicks on average**. E.g., for a scene with 6 objects the t@$\overline{5}$ is the total time spent on forward passes through the network after 6×5=30 clicks. Throughout, our method is **2× times faster** (1.0s vs 2.4s at t@$\overline{10}$; 1.6s vs 3.5s at t@$\overline{15}$; etc.).

> Or, the user study is a good choice to compare the efficiency. 

We fully agree, that is why we have conducted a real user study, see Tab. 7. Each user was asked to annotate 5 scenes with, in total, 30 objects. We report the final number of clicks ($\overline{\textrm{NoC}}$, average of all objects), the intersection-over-union achieved with those clicks ($\overline{\textrm{IoU}}$, average of all objects), and **the time spent by the user** to obtain that result ($\overline{\textrm{t}}$, average of all scenes). In the study, users achieved higher IoU with significantly fewer clicks, and in shorter times, with our proposed multi-object segmentation approach. I.e., the **user study confirms better efficiency with AGILE3D**.

>In some challenging cases, there are a lot of occlusions between the segmented object and other objects in the scene.

We are not sure what the question is. Regarding missing data due to sensor occlusions: yes, the data has the characteristics of real scans. Regarding occlusions in the visualization: with our interactive tool users can use the mouse to rotate/translate/zoom in/zoom as needed to minimize occlusions. **All users quickly got familiar with the tool and we did not receive any feedback about occlusion problems**.

Reviewer comment (Novelty):
The significant difference between interobject3D and interobject3D++ should be discussed comprehensively since the published paper is very relative to the submission.

Author response:
**4. The difference between interobject3D and interobject3D++**

The difference is explained in the “baselines” paragraph in Sec. 4. **InterObject3D is the only recent published work** about (single-object) interactive 3D segmentation. **InterObject3D++ is our enhanced version** (i.e., a harder baseline) that we have derived from it. The reason for introducing it is that the iterative training strategy that we propose in our paper can be readily applied also to InterObject3D. **We found that, indeed, iterative training significantly improved InterObject3D. We call the enhanced version InterObject3D++**, and include it for a complete and fair comparison that properly disentangles our model design from the training strategy. **AGILE3D beats also InterObject3D++, indicating that its advantage is due to the multi-object model design and not only the result of a better engineered training strategy**.

Reviewer comment (Presentation):
From Figure 1, the input only presents the XYZ + RGB, i.e., the point cloud and point color. Why does the visualization have the faces? Could you use the faces as extra information to enhance the performance of the proposed methods? From the interactive demo, there is still the face visualization.

Author response:
**5. Visualizations have the faces**

For all visualizations of ScanNet, e.g., Fig. 1 and for the interactive demo (Fig. 12), we visualize the meshes. This is common practice, and only serves to make the renderings easier to read. E.g., [1,2,3] also plot faces, but do not use them in their computational methods. We also do not use face information since not every 3D dataset contains faces. It is of course true that one could potentially calculate normals from the faces and use them as additional input to the network, but that would mean that the network could no longer be applied to scan data that does not contain faces (like S3DIS, KITTI-360, etc.).

[1] Peng, Songyou, et al. OpenScene: 3D Scene Understanding with Open Vocabularies. CVPR. 2023.

[2] Schult, Jonas, et al. Mask3D for 3D Semantic Instance Segmentation. ICRA. 2023.

[3] Chibane, Julian, et al. Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation using Bounding Boxes. ECCV, 2022.

Reviewer comment (Novelty):
The related works are very sparse, especially for the 3D instantce segmentations. I suggest the author refer to some surveys about the 3D segmentation and discuss more papers on it, especially for the pointnet and pointnet++.

Author response:
**6. Related works**

Thank you for the suggestion. We have rewritten the related work part for 3D instance segmentation, including more discussions on point-based and voxel-based learning backbones and more details on existing methods on 3D instance segmentations, as well as references to surveys. Please check the red-marked content in our revised manuscript.

Reviewer comment (Evaluation):
For the comparison, it is not clear to me: Are the fewer user clicks the same for the different methods?

Author response:
**7. Are the fewer user clicks the same for the different methods?**

For comparison using simulated clicks, only the first click is the same for all methods. This serves as a fair, common starting point. The following clicks will be sampled from regions where a method makes mistakes, which obviously differ between methods. 

In the study with real user clicks, even the first click could be different, as it would defeat the purpose of a realistic user study to enforce specific click locations for everyone. We did, on purpose, also not give explicit instructions how to click, in order to obtain realistic and representative user behavior.