Paper: CSI: Enhancing the Robustness of 3D Point Cloud Recognition against Corruption (ICLR-2024)

Reviewer comment (Novelty):
The novelty of the proposed density-aware sampling may be limited as similar ideas have already been explored by previous methods [1,2], but the authors do not provide any comparisons with them.

Author response:
We appreciate the reviewer’s concerns regarding the novelty of our proposed Density-Aware Adaptive Sampling (DAS) and the challenges in directly comparing it with methods presented in references [1] and [2]. The method in [1] fundamentally differs from ours in terms of output format and integration with subsequent processing layers. Specifically, our DAS outputs a format of (S, S, H) and is intricately linked with spherical voxel convolution layers, making it incompatible as a direct replacement for FPS or DAS in PCT architectures. Therefore, a direct comparison is not feasible due to these structural and functional differences.

Regarding the method in [2], we faced the challenge of no open-sourced code being available. Despite this, we endeavored to reproduce their method to the best of our ability. Due to time constraints, we implemented a version of Grid Density Sampling (GDS) with a grid size of 8, as larger grid sizes would have necessitated significantly longer training times. The results of this implementation are presented:

|ModelNet40-C | mER| Occlusion | Lidar | Density Inc. | Density Dec. | Cutout | Uniform	Gaussian |	Impulse	Upsampling	| Background |	Rotation |	Shear |	FFD |	RBF |	Inv. RBF|
|-| ---| --------- | ----- | ------------ | ------------| ------ | ---------------- | -------------------   | ------------ | --------- | -------| --- | ------| -----| 
PCT+CSI (GDS)| 21.96| 54.54 |	59.85 |	11.46 |	14.4 |	13.1 |	11.65 |	11.3 |	22.43 |	13.48 |	45.28 |	19.04 |	14.6 |	13.32 |	12.58 |	12.46 |
PCT+CSI (DAS) | 18.38 | 56.23 |	60.2 |	12.37 |	12.73 |	11.97 | 10.83 |	10.73 |	12.3 |	11.63 |	11.19 |	17.41 |	11.98 |	12.05 |	12.41 |	11.65 |

|PointCloud-C |mCE| Scale | Jitter | Drop-G | Drop-L | Add-G | Add-L | Rotate |
|-| --| ----- | ------ | ------ | ------ | ----- | -----| ------ |
PCT+CSI (GDS) | 0.996 | 1.17 |	0.528 |	0.815 |	1.043 |	1.166 |	1.244 |	1.009 |
|PCT+CSI| 0.757| 1.128 | 0.472 | 0.56 | 0.889 |0.329 | 0.924 | 0.995 |

While it is true that the underlying motivation of our DAS may share similarities with these previous works, the implementation and the operational context significantly differ. A key distinction of our DAS is that it is not a grid-based approach, unlike GDS. We believe that grid-based operations can potentially disrupt the inherent structure of point clouds. Our ablation studies on ModelNet40-C and PointCloud-C demonstrate that our non-grid-based approach is more effective, supporting the unique contribution and novelty of our DAS methodology in preserving the integrity of point cloud structures while enhancing sampling efficiency.

Reviewer comment (Writing):
The writing contains inconsistent usage of section references. For instance, Section 3.2 (SELF -ENTROPY MINIMIZATION) uses different references like $\S 3.1$, $\S 1$, and *Section 2*; $d_a$ and $d_e$ in equation 4.

Author response:
We greatly appreciate your feedback regarding the readability of our manuscript. We recognize that clear and accessible writing is crucial for effective communication of our research. To address your concerns, we will undertake a comprehensive review of the manuscript to simplify and shorten complex sentences, ensuring they are more reader-friendly. Additionally, we will systematically revise the usage of section references to maintain consistency throughout the text. Our goal for the final version is to enhance overall clarity and ease of reading, making our research more accessible to a broader audience. We are committed to improving these aspects to ensure our work is communicated as clearly and effectively as possible.

Reviewer comment (Experiments):
In Table 1, `PCT+CSI` achieves an ER of 18.4 in ModelNet40-C, which is clearly inferior to `PCT+PointCutMix-R` (16.3) and `PCT+PointCutMix-K` (16.5). These two configurations are from the method [3] that originally introduces the ModelNet40-C dataset. The authors perform the experiment with `PCT+PointCutMix-R+CSI` in Table 2, but the comparison is unfair because it involves comparing `A+B+C` against `A+B`, and the overall improvement is minimal (0.4%).

Author response:
The major limitation of data augmentation is that different data augmentation techniques have varying degrees of effectiveness against distinct types of corruption. This is because the process of augmenting data often relies on heuristic approaches, which may not always align with the underlying data distribution. Motivated by the robustness can also be improved from the model architecture perspective, our proposed CSI aims to exploit the distribution shift in unseen data itself rather than in a heuristic way. However, when combined with data augmentation, the margin of improvement is relatively modest. This can be attributed to the inherent complexity of data augmentation methods in 3D point clouds, which often entail the blending of point clouds from different classes. This amalgamation creates a level of ambiguity that makes the identification of a critical subset in the augmented point cloud challenging. We appreciate you pointing out the limitation of CSI when combined with the augmentation. This certified solution remains unexplored and it is a potentially promising avenue for our future work.

Reviewer comment (Evaluation):
Similar observations can be found in Table 2 of the Appendix, where `PCT+PointCutMix-R+CSI` fails to outperform `PCT+WOLFMix`.

Author response:
Just keep aligned with the metric in Table 9 as listed in [PointCloud-C paper](https://arxiv.org/pdf/2202.03377.pdf). We will change the metric from mOA to ER in our final version. The results of ER are listed below:

| Model (\%) $\downarrow$ | Clean | ER | Scale | Jitter | Drop-G | Drop-L | Add-G | Add-L | Rotate |
|-----------------------|-------|-----|-------|--------|--------|--------|-------|-------|--------|
| DGCNN                 | 7.4   | 23.6  | 9.4   | 31.6   | 24.8   | 20.7   | 29.5  | 27.5  | 21.5   |
| PointNet              | 9.3   | 34.2  | 11.9  | 20.3   | 12.4   | 22.2   | 87.9  | 43.8  | 40.9   |
| PointNet++            | 7.0   | 24.9  | 8.2   | 37.2   | 15.9   | 37.3   | 18.1  | 27.3  | 30.2   |
| RSCNN                 | 7.7   | 26.1  | 10.1  | 37.0   | 20.0   | 31.4   | 21.0  | 31.7  | 31.8   |
| SimpleView            | 6.1   | 24.3  | 8.2   | 22.6   | 30.8   | 28.1   | 29.0  | 23.2  | 28.3   |
| GDANet                | 6.6   | 21.1  | 7.8   | 26.5   | 19.7   | 18.5   | 25.7  | 28.5  | 21.1   |
| CurveNet              | 6.2   | 22.1  | 8.2   | 22.9   | 17.6   | 21.2   | 39.7  | 27.5  | 17.4   |
| PAConv                | 6.4   | 27.0  | 8.5   | 46.3   | 24.8   | 20.8   | 32.0  | 35.7  | 20.8   |
| RPC                   | 7.0   | 20.5  | 7.9   | 28.2   | 12.2   | 16.5   | 27.4  | 27.8  | 23.2   |
| PCT                   | 7.0   | 21.9  | 8.2   | 27.5   | 13.1   | 20.7   | 23.0  | 38.1  | 22.4   |
| **PCT+CSI**               | 7.3   | 16.3  | 10.5  | 15.2   | 13.9   | 18.4   | 9.5   | 25.3  | 21.3   |
| DGCNN+OcCo            | 7.8   | 23.4  | 15.1  | 20.6   | 22.4   | 21.5   | 42.6  | 23.3  | 18.0   |
| Point-BERT            | 7.8   | 30.7  | 8.8   | 39.8   | 17.1   | 23.8   | 57.0  | 39.6  | 28.5   |
| PN2+PointMixUp        | 8.5   | 21.5  | 15.7  | 22.5   | 19.9   | 37.5   | 13.5  | 16.9  | 24.3   |
| DGCNN+PW              | 7.4   | 19.1  | 8.7   | 27.3   | 24.5   | 18.1   | 23.8  | 21.0  | 10.3   |
| DGCNN+RSMix           | 7.0   | 16.1  | 12.4  | 27.6   | 16.2   | 12.2   | 8.3   | 17.3  | 18.7   |
| DGCNN+WOLFMix         | 6.8   | 12.9  | 9.3   | 22.6   | 17.3   | 11.9   | 8.4   | 11.4  | 9.7    |
| PointNet+WOLFMix      | 11.6  | 25.7  | 19.9  | 15.0   | 14.3   | 22.4   | 65.7  | 19.3  | 23.2   |
| PCT+WOLFMix           | 6.6   | 12.7  | 9.4   | 27.0   | 9.4    | 10.2   | 8.8   | 13.9  | 10.5   |
| GDANet+WOLFMix        | 6.6   | 12.9  | 8.5   | 27.9   | 13.2   | 11.4   | 9.0   | 11.4  | 8.8    |
| RPC+WOLFMix           | 6.7   | 13.5  | 9.5   | 30.6   | 10.5   | 10.6   | 9.8   | 13.2  | 10.3   |
| **PCT+CSI+PointCutMix-R** | 7.2   | 12.9  | 11.3  | 11.5   | 11.3   | 15.9   | 8.2   | 12.2  | 16.3   |

Reviewer comment (Writing):
The paper could improve the clarity of exposition. Some parts of the paper, particularly in the methodology section, are not explained in a clear and concise manner, which may impede the reader's understanding.

Author response:
We appreciate the feedback on the clarity of our exposition, especially in the methodology section. We understand that clear and concise communication is crucial for the reader's comprehension. To address this, we will undertake a thorough revision of our manuscript, with a specific focus on enhancing the clarity and conciseness of the methodology section. We aim to ensure that our final version presents our methods and findings in a more accessible and understandable manner. This revision will include refining complex explanations, simplifying technical jargon where possible, and providing additional context or examples to facilitate better understanding.

Reviewer comment (Experiments):
I think the method proposed in ModelNet40-C and PointCloud-C should also be compared and analyzed.

Author response:
Thank you for emphasizing the importance of comparative analysis with ModelNet40-C and PointCloud-C. We have indeed conducted such comparisons and included them in our manuscript. Specifically, Table 1 in the main text and Tables 1 and 2 in the appendix present a comprehensive comparison of our method against others mentioned in the original papers of ModelNet40-C and PointCloud-C. These tables detail the performance metrics and demonstrate how our method stacks up against existing methods under similar conditions. We believe this thorough comparison effectively illustrates the strengths and limitations of our proposed approach in the context of these well-established datasets.

Reviewer comment (Theory):
Please explain the statement “local density of a point positively correlated with its significance”? Previous work has indicated that significant points are usually outward points in the point cloud, which are generally sparse. Also, traditional sampling methods like FPS tend to find points with low density as representations.

Author response:
>Please explain the statement “local density of a point positively correlated with its significance”? Previous work has indicated that significant points are usually outward points in the point cloud, which are generally sparse[1]. Also, traditional sampling methods like FPS tend to find points with low density as representations.

The observation that significant points are predominantly located at the outer regions of the point cloud does not hold true in scenarios involving data corruption, as demonstrated in Figure 3. In such cases, Farthest Point Sampling (FPS) tends to retain an excessive number of noisy data points. This limitation of FPS is a key motivator for our proposal of the Density-Aware Sampling (DAS) method. DAS is specifically designed to mitigate the adverse effects caused by FPS, particularly its tendency to sample a higher number of noisy points in the presence of various forms of corruption, such as background noise and global noise.

Reviewer comment (Experiments):
The authors should compare with other SOTA train-time methods under the same settings in the main experiment, such as data augmentation or modules addition to the model.

Author response:
>The authors should compare with other SOTA train-time methods under the same settings in the main experiment, such as data augmentation or modules addition to the model.

We acknowledge the suggestion to benchmark our approach against other state-of-the-art (SOTA) train-time methods under equivalent experimental conditions, including factors like data augmentation and module additions. In response, we direct attention to Table 2 in the main manuscript, which provides a comprehensive comparison where all evaluated methods, including ours, are integrated with the data augmentation technique PointCutMix-R. Additionally, for a more extensive comparison, Table 2 in the supplementary material, ranging from the entry 'DGCNN+OcCo' to 'PCT+CSI+PointCutMix-R', offers a detailed analysis of various other train-time methods. This inclusion ensures a thorough and fair evaluation of our method against current SOTA techniques under consistent experimental settings.

Reviewer comment (Experiments):
The experiments should consider including datasets not based on ModelNet40.

Author response:
>The experiments should consider including datasets not based on ModelNet40.

In addressing the recommendation to include datasets beyond ModelNet40 in our experiments, we emphasize our focus on point cloud recognition within this work. ModelNet40, along with its two corrupted variants, ModelNet40-C and PointCloud-C, are extensively recognized and employed in related research. These datasets are particularly relevant and suitable for the objectives of our study. Our decision to utilize these datasets is grounded in their widespread acceptance in the field, ensuring that our findings are comparable and relevant to current standards in point cloud recognition research.

Reviewer comment (Evaluation):
Tables 1 and 2 in Supplementary show that CSI cannot effectively handle point dropping (e.g., occlusion) and transformations (e.g., rotation), and may even be harmful. Does this indicate the limitations of CSI in terms of generalizability? Therefore, the reviewer points out that the results derived from the ModelNet40-C can be misleading.

Author response:
>Tables 1 and 2 in Supplementary show that CSI cannot effectively handle point dropping (e.g., occlusion) and transformations (e.g., rotation), and may even be harmful.

We appreciate the reviewer's observation regarding the performance of CSI under conditions of point dropping and transformations, as shown in Tables 1 and 2 of the supplementary material. We acknowledge that these results may highlight a limitation in the generalizability of the CSI approach, particularly in handling occlusions and rotations. This limitation is indeed an area that warrants further investigation and improvement in future iterations of our work. Regarding the use of ModelNet40-C, we understand the concern that the results derived from this dataset might present a skewed perspective. We will consider this point critically in our analysis and discussion, ensuring that we clearly communicate the potential limitations of our findings in the context of different datasets and conditions. However, we argue that it is hard for a method to cure all corruption types. This feedback is invaluable for guiding our ongoing research efforts to refine and enhance the robustness and applicability of our approach.

Reviewer comment (Novelty):
SEM is mostly based on previous knowledge that entropy minimization helps classification robustness, which slightly undermines the significance of the proposed techniques. Nonetheless, the paper provides a detailed discussion of how entropy minimization should be applied to transformer-based point classifiers in both attention layers and the classification head, accompanied by sufficient ablation studies.

Author response:
Although previous work has investigated how entropy minimization helps classification robustness, in their works, they merely investigated how to minimize entropy during inference. For example, tent [1] minimizes Shannon Entropy on classification logits through several iterations. The biggest difference is that we give a thorough analysis of how to apply entropy minimization during training and which features contribute most by conducting ablation studies not only on transformer self-attention maps but also on intermediate features.

[1] Wang, Dequan, et al. "Tent: Fully test-time adaptation by entropy minimization." arXiv preprint arXiv:2006.10726 (2020).

Reviewer comment (Experiments):
It is not clear how general DAS is and how it affects the classifier's robustness to more types of corruptions other than global noise addition shown in Figure 3. Table 1 and Table 2 in the supplementary material ablate CSI as a whole so they can not show the effect of DAS. It would be better if DAS could be individually studied on different types of corruption.

Author response:
More visualization results are available in the appendix of our revised manuscript. One can observe the effectiveness of DAS compared with FPS (Farthest Point Sampling) and RS (Random Sampling). Besides, our quantitative experimental results have also validated that DAS has outperformed other baseline methods in improving corruption robustness.