Paper: OpenPatch: a 3D patchwork for Out-Of-Distribution detection (ICLR-2024)

Reviewer comment (Experiments):
In the experiment part, the authors only compare OpenPatch with a few related literatures. However, from section 2, there are many related work that are not included in the experiment part. The authors are encouraged to provide more robust experiment comparison to convince the effectiveness of the proposed method.

Author response:
Most of the OOD detection literature covered by our sec 2 is tailored to 2D data and considers methods that require a training or at least a finetuning phase on the support set that contains samples of known classes. Our approach is instead learning-free. OpenPatch exploits a pre-trained architecture to extract data representations and perform sample-to-class comparisons to detect novelty in 3D data. Despite the different scenarios, the right part of Fig. 2 shows that OpenPatch is competitive with the state-of-the-art in 3D OOD detection methods (according to [Alliegro et al. (2022)]) that train or finetune on the support set.

Reviewer comment (Presentation):
In figure 4, the concept of "Patch" is not clearly demonstrated. It would be nice the the authors can show the "Patch"-level visualizations.

Author response:
In Figure 4, we present a depiction of patch assignments using a backbone based on Farthest Point Sampling (FPS). For this instance, we extracted the coordinates of the patch features and subsequently assigned each point in the original point cloud to its nearest patch feature. In the initial column, we color-coded based on the class assignment of the underlying patch embedding, while in the second column, each point was colored using heatmap colors corresponding to the relative embedding distance from the memory bank of the underlying patch embedding.
We recognize that discerning the different patches could be difficult due to the color similarity, to counteract this, we added an additional figure in the supplementary material where we color each patch a different color and compare the patch dimension when changing the layer of extraction.

Reviewer comment (Experiments):
The task of out-of-distribution detection is closely related to the open-set and open-world task. Nonetheless, the paper does not compare to methods that perform open-world tasks. "Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence (2012)" "Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (2015)"

Author response:
Out of Distribution Detection aims at labeling a new sample as belonging or not to the training distribution. This is a binary problem. Open Set Recognition refers instead to a class set, so its goal is identifying if a new sample belongs to one of the known classes or to an unknown one. This is a multiclass problem and more precisely, it combines the multiclass recognition of the known classes with the binary task of separating known from unknown data. Open World builds over Open Set, so that once a new class is identified it is incrementally added to the original model.
Our focus is on Out of Distribution Detection since we are only interested in the binary task of separating known from unknown data. As in our specific case the unknown data is novel because it belongs to a new semantic class, we also used the expression ‘semantic novelty detection’.
We are aware that the names might be confusing at first sight, but they have been explained in the first paragraph of our sec. 2.
Having clarified the frameworks it also becomes clear that the papers cited by the reviewers do not match our setting.

Reviewer comment (Novelty):
The motivation for a 3D framework is somehow weak since the paper does not explain the challenges in the 3D domain compared to the direct adaptation from 2D.

Author response:
The work [Alliegro et al. (2022)] extensively discussed the different challenges of 3D OOD detection and Open Set Learning with respect to those of the corresponding 2D domain.

Reviewer comment (Experiments):
There no proper ablation studies.

Author response:
The component analysis presented in section 4.3 explains in detail the role of each component of our approach. We would be happy to provide further details if the reviewer could specify the meaning of ‘proper ablation’ for our model.

Reviewer comment (Writing):
Using "detection" in the title is confusing since the paper is only doing classification and semantic segmentation.

Author response:
Out of Distribution Detection is a standard name to indicate the task of predicting whether a test sample belongs or not to the nominal data distribution. Unfortunately there is a clear misunderstanding, we are not performing neither object detection nor semantic segmentation.

Reviewer comment (Writing):
How is out-of-distribution detection different from open-set and open-world?

Author response:
Out of Distribution Detection aims at labeling a new sample as belonging or not to the training distribution. This is a binary problem. Open Set Recognition refers instead to a class set, so its goal is identifying if a new sample belongs to one of the known classes or to an unknown one. This is a multiclass problem and more precisely, it combines the multiclass recognition of the known classes with the binary task of separating known from unknown data. Open World builds over Open Set, so that once a new class is identified it is incrementally added to the original model.
Our focus is on Out of Distribution Detection since we are only interested in the binary task of separating known from unknown data. As in our specific case the unknown data is novel because it belongs to a new semantic class, we also used the expression ‘semantic novelty detection’.
We are aware that the names might be confusing at first sight, but they have been explained in the first paragraph of our sec. 2.
Having clarified the frameworks it also becomes clear that the papers cited by the reviewers do not match our setting.

Reviewer comment (Writing):
What are the challenges/differences in the 3D domain compared 2D?

Author response:
The work [Alliegro et al. (2022)] extensively discussed the different challenges of 3D OOD detection and Open Set Learning with respect to those of the corresponding 2D domain.

Reviewer comment (Novelty):
The importance of determining if the point cloud of an object is out-of-domain for a 3d classifier is not well-presented. No prior works nor possible applications are shown for this work.

Author response:
Regarding the “no prior work” claim, we kindly refer the reviewer to [Alliegro et al (2022)] in our references. This work introduced the task of 3D OOD detection and ran an extensive benchmark of methods originally designed for 2D data, showing that their performance does not necessarily transfer to 3D. 
Regarding the “no possible application” claim, we highlight the essential need for methods able to manage novelty in real-world applications involving 3D information. We refer the reviewer to the following works that appear among our references [Masuda et al. (2021); Bhardwaj et al. (2021); Cen et al. (2021); Wong et al. (2019); Riz et al. (2023)]. Besides autonomous driving where unexpected objects might create critical safety issues, other relevant scenarios are related to industrial robotics applications where agents are supplied with only a few clean 3D object templates to use as reference and they should be able to mitigate potential operational hazards due to novel objects.
Both prior works and applications are mentioned in the paper's introduction and related work section.

Reviewer comment (Reproducibility):
The authors proposed a patch-based method for determining if a point cloud is OOD but failed to mention how are those patches obtained. I guess how to break a whole point cloud into patches has a significant impact on the OOD recognition accuracy. In this sense, the proposed method is not even complete.

Author response:
Regarding “failed to mention how are those patches obtained” we point the reviewer to sec 3 (in particular 3.1) of our submission. There, we explained that representations extracted from internal layers of networks that elaborate on point clouds describe different sized 3D shape portions that we indicate as patches. There is no manual segmentation involved here, the patches are a byproduct of the chosen architecture and the only choice to operate is deciding from which layer to extract them: deeper layers provide larger patches.

Reviewer comment (Experiments):
In Table 1 and Table 2, compared with methods such as EVM and Mahalanobis that have been proposed for many years, if the latest research such as [1] Semantic Novelty Detection via Relational Reasoning, [2] Detecting out-of-distribution examples with Gram matrices, and [3] Delving into Out-of-Distribution Detection with Vision-Language Representations are added to the comparison, can OpenPatch maintain its advantage?

Author response:
Concerning the mentioned references we can state that:
[1] leverages ad hoc pre-training and a very specific architecture designed to be used for semantic novelty detection on 2D data. Instead, our work targets efficiency in design and learning effort: OpenPatch builds on large-scale 3D models trained with standard classification or contrastive objectives and well-known backbones. 
We remark that, despite 3D transformer models exist, adapting the relational module and the learning pipeline of [1] to the 3D setting is not straightforward.
The method proposed in [2] was shown in [1] to be worse than the Mahalanobis baseline that we adopted. Therefore, although adding it to our analysis might improve completeness, our conclusions would remain unchanged.
The approach in [3] requires the availability of a textual description for the known data object category. This cannot be given for granted in industrial settings where in-distribution objects are custom CAD models created by users and are possibly identified by numerical codes without any semantic meaning.

Reviewer comment (Experiments):
As mentioned in 3.2, "The banks' cardinality may quickly increase, significantly impacting the computational cost of the method. To mitigate this effect and address redundancy we adopt a greedy coreset subselection mechanism.” Can the impact of the greedy coreset subselection mechanism on computational cost reduction be quantitatively reflected in the experiment?

Author response:
The main bottleneck of OpenPath lies in the nearest neighbor search which depends on the number of patches stored in the memory banks. This total count of patches is determined by the number of samples in the support set and the layer used for feature extraction in the pretrained model encoder. In our specific experiments, we observed negligible performance gains from implementing a more aggressive coreset reduction. We attribute this to the low number of samples in the support set, as defined by the 3DOS benchmark, and our decision to use larger patches rather than numerous smaller local patches. Our main concern for implementing the coreset selection is mainly the limited memory of devices in industrial scenarios.