Paper: OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation (ICLR-2024)

Reviewer comment (Experiments):
The mask proposal module is adapted from an existing instance segmentation model Mask3D. Although the authors remove the class-specific information, essentially Mask3D is trained with a close set of categories. Hence the mask proposal module is not class-agnostic and the whole system is not an open-vocabulary system, as this system cannot handle the irregular point cloud clusters that don't belong to the indoor object classes. To evaluate this, I suggest the authors adapt their system to outdoor driving scenarios such as nuScenes, to see whether their approach can generate mask proposals of objects on a road such as traffic cones and barriers.

Author response:
It is widely acknowledged that a substantial domain gap exists between indoor and outdoor scenes in 3D. This is why many attempts at 3D Foundation Models ([1],[2],[3])  and Unified 3D representation [4][5] models still require training separate weights for indoor and outdoor scenes.

It is true that generalizing the indoor scene trained Mask Proposal Module (MPM) to the outdoor scene would be a remaining challenge. However, we believe that within the same scene space, i.e., indoor or outdoor, the MPM possesses a decent level of generalization capability. To substantiate this claim, we conducted zero-shot experiments with OpenIns3D, trained on ScanNet, across four diverse types of indoor datasets. The visual results are presented in the following anonymous repository: https://anonymous.4open.science/r/rebuttal_image-6CBE/README.md.

Reviewer comment (Experiments):
The authors need to compare with CLIP^2 [1] in Table 1, which can also generate open-vocabulary instance segmentation results. 

[1] Zeng et al. CLIP$^2$: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data. CVPR 2023.

Author response:
We thank the review for bringing our attention to this good work. Indeed, CLIP$^2$ can also generate instance-level results with the help of aligned 2D images. The results are rather good on ScanNetV2. We will update the paper and add this to the literature review.

Integrating the results into Table 1 might be challenging, though, as CLIP$^2$, unlike other 3D instance segmentation work, uses the top-1 accuracy metric rather than the Average Precision metrics. It would not be so cohesive to put it in Table 1.

Speaking of Table 1, an interesting insight that might be worth sharing is that, upon slight fine-tuning and testing (better mask proposal and better SNAP images), the performance of OpenIns3D on the novel 17 classes on ScanNet v2 can be massively improved compared to the original reported results. As shown in the table below. We think this indicates that the "Mask-Snap-Lookup" approach has huge potential that can be explored in the future.

|                      |   AP  |  AP50 |  AP25 |
|:--------------------:|:-----:|:-----:|:-----:|
|  Mask3D + PointCLIP  |  3.20 |  4.50 | 14.40 |
| OpenIns3D (reported) | 16.80 | 28.70 | 38.90 |
|    OpenIns3D (new)   | **28.25** | **38.44** | **46.30** |

Reviewer comment (Reproducibility):
As the method consists of multiple steps, the authors should provide more training details for all steps in the main text or appendix.

Author response:
## More details in the paper
As per our response to reviewer (rePE), we will include more information about Mask Scoring Module, as well as the Snap module in the revision (will be added later). More details about these modules can be found in the reply to the reviewer (rePE).

Reviewer comment (Experiments):
Will the performance of 2D Open-world Detector influence the performance of the OpenIns3D? The authors seems not to provide experimental results in ablation study.

Author response:
## Will the 2D performance influence the performance of 3D? 

Indeed. This is because we completely outsource the detection tasks to 2D model. We provided a small piece of comparison results in Table 2, but here we offer a direct comparison of OpenIns3D using two different backbones, as shown in the table below.

|Methods|Rendering|2Dbackbone|AP50|AP25|
|-|-|:-:|:-:|:-:|
|Mask-P-CLIP|Mask|CLIP|4.5|14.4|
|OpenIns3D|Scene-level|Grounding-DINO|18.8(+14.4)|29.8(+15.4)|
|OpenIns3D|Scene-level|ODISE|28.7|38.9|

An important note here is that once the rendering is switched to the scene level, the general performance improves significantly compared to that of the object/mask-level rendering, regarding less the 2D model used.

A further note is that relying on the 2D detector, on the other hand, is also a good strategy, as their evolving speed is generally faster. This ensures that OpenIns3D can also evolve with the 2D detector. This design also offers more special abilities, such as understanding complex sentences when the 2D detector is integrated with the LLM. We demonstrated this in the paper in Figure 1 and 6.

Reviewer comment (Experiments):
The authors should provide visual comparisons with SOTA.

Author response:
## Questions:
- Due to time constraints in the rebuttal period, we might not be able to include a comparison with other state-of-the-art models, but we will certainly include this as future work.

Reviewer comment (Theory):
At its core, the method is built on top of a somewhat flawed assumption. How can we obtain RGB point clouds, without actually having aligned RGB images? Of course there might be LiDAR point clouds without aligned RGB images, but at that point we can also not create synthetic RGB images from an uncolored point cloud, to feed into a 2D model expecting RGB images. While I still see some potential benefit, like being able to render novel images that are more focused on certain objects or better suited for the downstream model, this aspect is not explored here. I can't really imagine another case, where we have RGB point clouds, but for some reason we had to delete each and every underlying RGB frame used for colorization. As such I don't see the true merit of this approach.

Author response:
The reviewer argues that since RGB point clouds are derived from RGB images (a point we acknowledge), developing a method that intentionally excludes the use of these 2D images for 3D scene understanding may not be considered meaningful (a viewpoint we strongly disagree with). Following this line of reasoning, it implies that the majority of point cloud processing methods, ranging from PointNet to PointTransformer, SpareNet, and Mask3D, which solely utilize 3D RGB point clouds as input without incorporating 2D images, do not offer a clear advantage over methods that demands both modalities as input. This argument is not valid logically.

When OpenIns3D will be useful: 
Imagine you've been given only a RGB 3D point cloud/mesh and need to conduct 3D open-world instance understanding. OpenIns3D remains easily applicable, while other methods may suffer.
More use cases, are explained in the introduction session, here we provide a condensed summary. 
- LiDAR point cloud often exists without matched 2D images. 
Example: Semantic3D, Tornoto3D, Paris-carla-3d, etc.
- Photogrammetry point cloud often exists without depth maps. [without depth maps it is hard to build point-pixel correspondence]	
Examples: Sensaturban, Campus3d, STPLS3D, etc.
- Point cloud generated from the registration of multiple scans or converted from 3D simulations/CAD models typically lacks 2D images.
Example: SynthCity, etc.

(Note: Example datasets are publicly available dataset under these described circumstances.)

In real Application, we may loss of pose, intrinsic, and depth information in the transferring process; we may remove of 2D images to reduce storage requirements. Example: ScanNetv2, 3D scan (3MB), image package (3GB). Clearly saving a 3MB mesh is easier than saving entire 3GB images.

Reviewer comment (Novelty):
Ignoring the above issue, the "mask" module is not novel, it's very similar to the one used in OpenMask3D with some mask filtering from SAM, the "lookup" module is fairly simple matching and not super interesting as far as I can tell. Leaving the "snap" module to be the core novelty of this approach. The way it is described initially: "Multiple synthetic scene-level images are generated with calibrated and optimized camera poses and intrinsic parameters." gives the impression that the camera poses and intrinsics might somehow be optimized on a scene by scene basis, to create the best possible synthetic images for a given scene, resulting in the best scores. However, I understand this module simply places a predefined number of cameras around the scene, looking at the center and optimizing the focal length according to some heuristics. The slightly more involved "local enforced lookup" is mostly explained in the supplementary, raising the question what the real novelty of this paper is supposed to be? Finally, the rendered images are still just points splatted to the camera? According to "Challenge 4" there is a domain gap between projected and natural images, which I agree with, but as far as I can tell, the paper does nothing to truly bridge this gap, apart from aiming virtual cameras in a certain way?

Author response:
Before addressing any questions related to this item, we wish to emphasize our belief that good and meaningful research lies not in proposing sophisticated methods but in suggesting simple yet effective approaches that yield sustainable gains for the tasks and new insights for the community.

- Substantial Gains:
OpenIns3D has proposed a 2D input-free framework, which is not only more flexible to use but has also achieved a wide range of state-of-the-art results in indoor and outdoor, instance segmentation and object detection tasks. This success is evident even when compared with methods using original 2D images.

- New Insights: One of the main insights of "Mask-Snap-Lookup" is that scene-level rendering scheme is beneficial for 3D open-world understanding. Because scene-level images enable all masks to be rendered and comprehended at once, massively reducing inference time while providing robust results. Consider having 100 masks in a scene; other approaches that handle each mask one by one need to render/crop 200-400 images, while scene-level rendering approach only requires 8-16 images, which provide a significant benefits in overall runtime. Meanwhile, this new pipeline can seamlessly evolve with 2D open world models without retraining, and handle super complex text input when integrated with an LLM-powered 2D detector.

The Mask module indeed leverages the strengths of both Mask3D and SAM, and our scoring and filtering design makes it work really well, providing a solid foundation for further steps. 
As previously explained, the Snap-Lookup module is designed to enable scene-level rendering. The lookup approach is not a simple match. The key idea behind the Lookup module is the construction and use of a mask2pixel map. 2D models work best when the image contains more background context and mask2pixel map allows us to search precisely in a global image for a specific 3D mask. We believe this design would be very useful for many 3D open-world models intending to leverage 2D image results. 

Gap with the natural image: Point clouds are rendered into images using ray tracing add proper lighting rather than simple point projection.

As explained in the appendix (Fig 9), when rendering images for each mask individually—whether using PointClip rendering or LAR rendering—these images are challenging for 2D models to comprehend. Transitioning to scene-level rendering significantly enhances the ability of 2D detectors to recognize objects in the image. This narrowing of the gap is strongly supported by experiments on outdoor datasets STPLS3D, where sparse point clouds with scene-level rendering also yield good results with 2D open-world models.

The Local Enforced Lookup is arranged in the appendix due to space constraints. We prioritise presenting the complete flow of the Mask-Snap-Lookup module, which is proved to be effective and is what distinguishes OpenIns3D.

With these explanations, we hope the reviewer could appreciate the merits of OpenIns3D a bit better. We will address other comments in the next response. Thanks

Reviewer comment (Experiments):
An actual ablation about how real images compare to synthetic images for this would have been very valuable. An interesting setup could have been to use a set of real images from the dataset and compare it to rendered synthetic images using the same camera pose and intrinsics.

Author response:
## Item 5 Using original image's pose and intrinsic

Thank you for the suggestion; however, could the reviewer kindly elaborate on the intention behind this setting? **In scenarios where 2D images are not accessible—how could we get to know the pose and intrinsic parameters of those images?** Through our numerous attempts with a 2D image-free approach, we've discovered that one of the main challenges is how to determine effective and efficient pose and intrinsic. Directly taking pose and intrinsic from the original 2D image might not be so logical. OpenMask3D demonstrates a similar setup in the paper in its latest revision, but we are unsure the intention on this.

Reviewer comment (Writing):
Certain parts of the text make it seem like it's a bad thing to use a pre-trained network such as 3DETR or DETR, clearly stating that MPM is trained from scratch (start of chapter 5). This feels a bit hypocritical given that this whole method relies on pretrained 2D VL models trained on huge amounts of data. In general I must sadly say the text seems to contain a lot of "fluff" for me, introducing new fancy names for all kinds of stuff.

Author response:
## Item 6 3D DETR

The concern regarding the use of pre-trained 3D models lies in their tendency to bypass the challenges associated with proposing bounding boxes, which are crucial tasks in 3D scene-level understanding.

It's important to emphasize that there is a lack of widely adopted 3D pre-trained models. To our knowledge, in their context, a 3D pre-trained model generally refers to being trained on the same dataset used for testing, supervised by all ground truth labels and classes during training. In fact, this practice is commonplace in the field of 3D visual grounding, because all benchmarks in that domain, are originated from the same dataset, ScanNetv2. However, we think this is not a good practice.

Several works in the field, such as the PLA family, Openmask3D, OpenScene, and OpenIns3D, demonstrate a dedicated focus on the challenges of 3D **scene-level understanding**. What sets these works apart is their simultaneous consideration of both proposal and understanding challenges. Therefore, we believe it is crucial to distinguish between works that solely rely on pre-trained box proposals and those that address both aspects.

However, we agree that using 2D DETR is reasonable, as they normally have good generalisation capability. We will modify the text accordingly.

**Lastly, we wish to emphasize that in the composition of this paper, we have adhered to a rigorous and integral mindset. While we are eager to share the novel and encouraging findings derived from numerous attempts, we have also been transparent in acknowledging the weaknesses and limitations in our method. All the results in the paper are easily reproducible and achieve significant improvements compared to the current SOTA. We have also invested time and effort to communicate the ideas as clearly as possible. These efforts, thankfully, have been well-received and appreciated by all other reviewers.**

**Regarding the reviewer's observation that our paper may contain a lot of "fluff," introducing new fancy names for various concepts, we recognize the subjective nature of such critiques. We would greatly appreciate it if the reviewer could provide more specific and practical suggestions for improvement. Without concrete recommendations, we believe such comments may not significantly contribute to enhancing the paper's quality.**

Reviewer comment (Presentation):
I think that the references to Table 5 and 6 in the text are swapped.

Author response:
1. Thanks for point this out, we will update this in new revision.

Reviewer comment (Writing):
The introduction states "and point clouds generated by photogrammetry frequently lack depth maps", how is this relevant here? And is a point projected to an image not a sparse depth map?

Author response:
2. There are two layers of argument here that we would like to convey, and the second argument is more important. 1) Yes, it is possible to project and get depth, but it is rather inconvenient 2) In our experience with various 2D-3D dataset alignments, it's highly probable that a depth map generated in such scenarios will lack accuracy and may not be very useful. **The precision of 2D-3D alignment heavily relies on the original execution of 3D reconstruction.** In datasets like ScanNet, where the reconstruction is notably good, there are only minor errors in the depth map and pose information. On the flip side, datasets like ArkitScene exhibit a much higher level of inaccuracy in 2D-3D alignment, making it challenging to link both modalities seamlessly. This introduces additional challenges for approaches that rely on both types of data. **While the assumption that directly projecting points to create a depth map is valid in the presence of perfect 2D-3D alignment, in reality, achieving such perfection is rarely the case.**

Reviewer comment (Experiments):
As suggested above, an actual evaluation on ScanNet200 would make a major difference.

Author response:
3. "As suggested above, an actual evaluation on ScanNet200 would make a major difference."
As suggested, we have now included ScanNet200 results. Please let us know if your concerns have been addressed.

Reviewer comment (Reproducibility):
The Mask Proposal Module is trainable using IoU as a form of supervision. It is strongly recommended to include comprehensive training details in the main draft of the paper.

Author response:
Thanks for your suggestion. We have explained the details as follows and will include this in the revision (will update later).
- Mask scoring: For each of the $N$ proposed masks, we utilized the Hungarian Match to match it with $n$ GT masks and calculated the IOU value between the each pairs (n). For the remaining masks $(N-n)$ that did not match with GT, we set the IoU value as zeros. With this method, a $N$ GT IoUs for all proposed masks are obtained. During training, a simple two-layer MLP is applied to process mask queries and predict their IoU value. This simple two-layer MLP is supervised by the $L_2$ loss between predicted IoU and GT IoU, as depicted in formula (1) in the main paper.

Reviewer comment (Reproducibility):
Further clarification is needed regarding the adjustment of camera parameters in the Camera Intrinsic Calibration process.

Author response:
- Intrinsic parameter calibration: 
Once the pose matrix $Pose$ is obtained via the $Lookat$ function (A.2 in appx), we initialize the Intrinsic with an ordinary intrinsic matrix. With this parameter, we project all points into the images. This will lead to randomly positioned scene images. Once this is obtained, we uniformly rescale the values of fx, fy, cx, and cy in the initialised intrinsic matrix by the same factor. This process merely repositions and rescales the projected image. To illustrate, if the original projected point was located in image coordinates within the range of [-1000, -192] in x, our calibrated intrinsic metrics transform it to [0, 2000] in x.

Reviewer comment (Experiments):
It is advisable to incorporate a comparative analysis that includes segmentation results obtained from multiple pseudo-projected images. Given that your method heavily relies on prior knowledge from 2D models, solely comparing it with point-based methods may not provide a fully equitable evaluation.

Author response:
We will answer Weakness 3 and Question 2 in this section as they are closely related

- Comparing with non-point-based methods: 
In general, most 3D open worlds rely on 2D images, making it challenging to categorize them as truly point-based. If considered solely from an inference perspective, PLA-family and OpenScene might be point-based methods, although both heavily rely on 2D images during training. Methods like PointClip, PointClipV2, and OV-3DET are derived from multiple projections/croppings, and their comparisons are included in the paper. We present these comparisons in Table 7 (OVOD), Table 3 (outdoor dataset), ablation studies on Table 2, and pre-class results on Tables 10 and 11.

-	Pros and Cons when comparing with pure-image projection-based methods:  For various pseudo-image projection methods, we showcase six attempts we made in Table 12 and compare them based on runtime and performance. Our scene-level rendering also proves to be robust in both performance and fast inference time. The downside, though, as we stated in the weakness session [point 3], is that it might not work well on small objects, as the visual information of small objects is diluted or lost in the 3D reconstruction process. For a more detailed analysis of this point, we compared our method with OpenMask3D on ScanNet200, as shown in the table below. OpenIns3D achieves competitive performance in the Head category (top 1/3 classes, generally medium size object) but shows weaknesses in common and tail categories. OpenMask3D achieves strong results in common and tail categories, at the cost of requiring well-aligned 2D image as input during inference. Compared with OpenScene, **in cases where only 3D is available during inference, OpenIns3D surpasses OpenScene in all categories.**

|Model|Require Aligned Images in training|Require Aligned Images in Inference|Head AP|Common AP|Tail AP|AP|AP50|AP25|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|OpenScene(2DFusion O-Seg)|Yes|Yes|13.4|11.6|9.9|11.7|15.2|17.8|
|OpenScene(2D/3DEns.)|Yes|Yes|11.0|3.2|1.1|5.3|6.7|8.1|
|OpenScene(2DFusionL-seg)|Yes|Yes|14.5|2.5|1.1|6.0|7.7|8.5|
|OpenMask3D|Yes|Yes|17.1|14.1|14.9|15.4|19.9|23.1|
|OpenScene (3DDistill)|Yes|**No**|10.6|2.6|0.7|4.8|6.2|7.2|
|OpenIns3D|**No**|**No**|**16.0**|6.5|4.2|8.8|10.3|14.4

Reviewer comment (Evaluation):
The performance gain from 2D models or framework design should be separately discussed.

Author response:
We conducted ablation experiments on various 2D models and evaluated their performance on the ScanNet20 dataset. This is included in Table 2. The following is a breakdown version of it. Notably, when we switched the rendering method to scene-level rendering, there is a significant increase in performance. This effect underscores the usefulness of scene-level rendering for 3D open-world understanding.
|Methods|Rendering|2Dbackbone|AP50|AP25|
|-|-|:-:|:-:|:-:|
|Mask-P-CLIP|Mask|CLIP|4.5|14.4|
|OpenIns3D|Scene-level|Grounding-DINO|18.8(+14.4)|29.8(+15.4)|
|OpenIns3D|Scene-level|ODISE|28.7|38.9|