Paper: Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detection (ICLR-2024)

Reviewer comment (Novelty):
The main idea of this paper is essentially an object cut-paste augmentation, which has been exploited as a common practice in most 3D object detectors. For example, starting from Point R-CNN, those 3D object detectors leverage object cut-paste to augment the 3D driving scenes. The proposed object anchor selection, normalization, and placement share very similar spirits with Point R-CNN [1] and Back-to-Reality [2].

Author response:
Thanks for your reminder, we clarify the differences with the existing data augmentation methods in the overall rebuttal section and have updated the related discussion in the revised paper.

Reviewer comment (Novelty):
The contrastive learning strategy has also been studied by many relevant papers. For example, CLIP^2 [3] leverages contrastive learning to align the text and point cloud representations for open-vocabulary 3D object detection.

Author response:
Contrastive learning itself is a very versatile learning strategy, widely used in a variety of tasks. However, how to design suitable contrastive learning objectives and construct appropriate positive and negative learning samples for different scenarios is an important issue. In our paper, we address the domain gap problem introduced by multi-object dataset training through a carefully designed category-level contrastive learning loss. However, in CLIP^2, contrastive learning is employed to align text and point cloud representations.

Reviewer comment (Evaluation):
Text prompts are generated by pre-defined rules and lack generalization ability.

Author response:
Our generated prompts are indeed rule-based and to some extent lack generative ability. However, the purpose of generating prompts is to align the point clouds and texts of objects in scenes, thereby achieving open-vocabulary. Experimental results validate the effectiveness of our prompt generation. Of course, we believe that more complex and natural language descriptions will have better generalization capabilities and bring improved point cloud-text alignment results. We will consider introducing LLM or some other tools in the future to generate more natural prompts for experiments.

Reviewer comment (Novelty):
The proposed detection framework didn't change much compared to the existing detectors. The detector is trained with supervised learning and doesn't rely on open-vocabulary foundation models such as CLIP. Hence it is hard to say whether the proposed method has the open-vocabulary detection ability, as the detection vocabulary is actually constrained by the 3D object datasets.

Author response:
We offer the related statement in the overall rebuttal section. Thanks for your question.

Reviewer comment (Evaluation):
Evaluation is also questionable: The authors split the datasets into seen and unseen categories, but those unseen categories are not really ``unseen", since the 3D object datasets they used can have those categories. Hence it is an unfair comparison with the baseline methods.

Author response:
We offer the related statement in the overall rebuttal section.

Reviewer comment (Novelty):
Essentially this paper is not an open-vocabulary 3D detection paper. It is 3D detection with categories augmentation by inserting new objects from 3D object datasets. The detected categories are bounded by the used object datasets.

Author response:
We restate the definition of open-vocabulary 3D detection in the overall rebuttal section.

Reviewer comment (Experiments):
May I know the scale of inserted 3D objects and generated prompts? For example, the number of them? Since contrastive training like GLIP usually requires large-scale objects and prompts.

Author response:
For OV-ScanNet20 and OV-SUN RGBD20 benchmark , the inserted object datasets cover 6w 3D objects with 190 categories. As for the generated prompts, since they are generated randomly and dynamically, based on our statistics, the actual number of prompts generated during the training process is approximately 500,000.

Reviewer comment (Presentation):
As mentioned in the last rebuttal, "we will add and discuss the cut and mix augmentation methods, such PointCutMix, in our final version." I could not find the discussion.

Author response:
Thanks for your reminder, we have added PointCutMix in the related work section in our updated version.

Reviewer comment (Experiments):
While using existing 3D object datasets can provide annotated objects across more categories for model training, the training process may still be limited by the vocabulary of these datasets. Learn semantics beyond these datasets’ object categories pose a challenge for the model.

Author response:
**2.While using existing 3D object datasets can provide annotated objects across more categories for model training, the training process may still be limited by the vocabulary of these datasets. Learn semantics beyond these datasets’ object categories pose a challenge for the model.**

Yes, it's true that learning semantics beyond these datasets’ object categories poses a challenge for the model during the training. However, once the number of object categories included in the training data is large enough, like Detic using ImageNet21K datasets to broaden the object concept, then the model's open-vocabulary capability can cover most objects in daily scenes.

Reviewer comment (Reproducibility):
The “Object Placement” process is central to Object2Scene but needs further clarification. How are injection regions determined? How is physical reasonability checked during placement? A pseudo code on this process could enhance understanding.

Author response:
**3.The “Object Placement” process is central to Object2Scene but needs further clarification. How are injection regions determined? How is physical reasonability checked during placement? A pseudo code on this process could enhance understanding.**

Thanks for your advice, we have added this pseudo code in the appendix of our updated paper for reference.

Reviewer comment (Experiments):
Lack of comparison on the larger OV-ScanNet200 benchmark.

Author response:
**4.Lack of comparison on the larger OV-ScanNet200 benchmark.**

|  | head |  common | tail |
| --- | --- | --- | --- |
| OV-PointCLIP | 2.76 | 0.56 | 0.0 |
| Detic-ModelNet | 1.56 | 0.12 | 0.0 |
| L3Det | 13.1 | 10.1 | 3.4 |