Paper: 3D Feature Prediction for Masked-AutoEncoder-Based Point Cloud Pretraining (ICLR-2024)

Reviewer comment (Experiments):
The results on the ScanObjectNN dataset indicate that Point-MA2E outperforms MaskFeat3D significantly; however, this comparison is not included in the paper.

Author response:
We are grateful to the reviewer for mentioning Point-MA2E, which introduces an intriguing concept of improving the masking strategy with affine transformation. However, it is important to clarify the idea of Point-MA2E is orthogonal to our method. Point-MA2E is dedicated to refining the masking strategy, while our work focuses on studying the pre-training pretext task. We speculate that Point-MA2E's mask strategy might improve our method, and we will provide a discussion on this in our revised manuscript.

Additionally, **the Point-MA2E paper has not yet been published, and its supplementary material remains unavailable**. This restricts our ability to fully grasp their evaluation procedures and conduct further systematical comparisons.

Reviewer comment (Experiments):
To assess the effectiveness of the decoder, it is recommended to include results obtained by combining the block features and masked queries, and feeding them into self-attention blocks of the same depth, similar to the MAE approach, with points used as the positional embeddings.

Author response:
We are thankful to the reviewer for their insightful suggestion. In response, we conducted this comparison experiment. We removed the cross-attention module, directing both masked queries and block features into the self-attention blocks, which is similar to MAE approach. The results of this comparison are presented in the table below, showing that our method achieves consistent improvements over the suggested alternative design.

| Method         | Target Feature             | ScanObjectNN PB-T50-RS |
| -------------- | -------------------------- | ---------------------- |
| Self-attention | normal                     | 85.9                   |
|                | surface variation          | 86.2                   |
|                | normal + surface variation | 86.9                   |
| MaskFeat3D     | normal                     | 86.5                   |
|                | surface variation          | 87.0                   |
|                | normal + surface variation | 87.7                   |

We hypothesize that the removal of the cross-attention module might have increased the complexity of the learning process. This complexity arises as the decoder needs to simultaneously process its own previous outputs and the encoder’s outputs. Such a scenario could potentially lead to information confusion. The cross-attention mechanism serves a crucial role in distinctly separating information originating from the decoder itself and that from the encoder. Blending these two types of information could hinder the model's ability to differentiate between relationships in the encoder features and query points.

Reviewer comment (Writing):
Analysis of 2D/3D masked autoencoders. In the manuscript, the authors commented that "These designs make an intrinsic difference from 2D MSMs, where there is no need to recover masked pixel locations." It is true. I understand the analysis by the authors. When we think of the vanilla MAE, it also takes a masked image as an input and predicts the color information, not its pixel location. However, when we think of the nature of the point cloud, it is sparse, irregular, and unordered. Even, I would say raw point cloud naturally does not involve color information. Accordingly, it is not feasible to extend the concept of the MAE for the 2D image into the MAE for the 3D points. In my opinion, the authors should have written such a clear understanding of MAE for 3D points.

Author response:
We are grateful for the insightful suggestion and deeply appreciate the reviewer's accurate understanding of our insight. We agree that, unlike grid-organized 2D image data, point cloud data is irregular and possibly noisy, making direct recovery of point positions less effective in capturing the intrinsic features of 3D shapes. We will clarify our statement more explicitly in our revision.

Reviewer comment (Experiments):
Details in computing surface normal and surface variance on scene-level experiments. While the various object-level datasets, such as shapenet, are synthetically created, the real-world points are captured by the sensors. Due to such difference, raw point cloud from the real world naturally involves lots of noise, which could be an issue when computing surface normal using PCA. So I wonder how the authors solve this issue when conducting experiments on Sec. 4-4 in the manuscripts.

Author response:
We thank the reviewer for this valuable feedback. During our experiments, we also observed that surface normals in scene data are noisy and often lack consistent orientation due to incomplete point clouds. However, we found that surface variation remains robust in these conditions. Therefore, as mentioned in Section 4.4, we took the rgb color signal and surface variation as the target features, rather than the surface normal.

Reviewer comment (Theory):
Insightful analysis. I truly agree that the proposed experiments demonstrate that the surface normal and surface variance are important measurements for self-supervising learning using 3D points. Technically, I also agree with such an observation. However, I wonder why such an approach brings performance improvement. Is there any geometric analysis? Based on the manuscript, this approach can be viewed as a naive extension of the Point-MAE that additionally uses other geometric measurements. I want to know the author's own analysis of such problem setup and insights.

Author response:
We appreciate the reviewer for raising this important concern. We would like to argue that our method is not merely a simple extension of Point-MAE.

Firstly, as elaborated in Section 1 and illustrated in Figure 1, in addition to the insight mentioned by the reviewer in W-1 (which we believe is also an important contribution), we introduce a general pre-training framework with a novel attention-based decoder, thereby enhancing the pre-training model. Our model is not restricted to any specific encoder design and is easily adaptable to various task settings, including those in challenging indoor environments. Also, by disregarding point positions and focusing on intrinsic 3D shape features, our model learns robust features suitable for diverse downstream tasks.

Secondly, as mentioned in Table 4 of the main paper (referenced below), a naive extension of Point-MAE with additional features does not yield the same level of significant improvement as our method. This finding further proves the validity of our insights and the effectiveness of our decoder design.  

----



| Method     | Target Feature                 |      | ScanObjectNN PB-T50-RS |
| ---------- | ------------------------------ | ---- | ---------------------- |
| PointMAE   | position only                  |      | 85.2                   |
|            | position + normal∗             |      | 85.7                   |
|            | position + surface variation∗  |      | 85.9                   |
|            | position + normal + variation∗ |      | 86.0                   |
| MaskFeat3D | normal                         |      | 86.5                   |
|            | surface variation              |      | 87.0                   |
|            | normal + surface variation     |      | 87.7                   |

**Table 4: Ablation study on different features.** ∗ uses position-index matching Zhang et al.[1] for feature loss computation.

----

**W-3-2: More analysis.**

In the supplementary material, we provide further experimental analysis. In Section 2.3, we visualized the point features of our model and other MAE-based approaches. Our approach shows more discriminative features than other methods. This observation proves that learning to reconstruct high-order geometric features robustly enables the encoder to extract more distinctive and representative features.

In Section 2.4 of the supplementary material, we investigated how the learned features of our model assist in zero-shot correspondence learning, showing that points nearest in the feature space exhibit similar semantic information, even across different objects within the same class. We hope these analyses further highlight the robustness and effectiveness of our pre-training strategy.

[1] Zhang, Yabin, et al. "Masked surfel prediction for self-supervised point cloud learning." *arXiv preprint arXiv:2207.03111* (2022).

Reviewer comment (Experiments):
Is there any reason that authors did not conduct experiments on the S3DIS dataset using 3D semantic segmentation? If there are some reasonable and meaningful results, I can convince the efficacy of this work. Otherwise, this work could be understood as naive extension.

Author response:
We thank the reviewer for pointing this out. We did conduct semantic segmentation task on S3DIS Area 5 using the PointNeXt [1] encoder backbone. These results are detailed in Table 4 of the supplementary material and summarized below. We observed a significant improvement of +0.9/+1 in mean Intersection over Union (mIoU) and Overall Accuracy (OA) compared to the baseline training from scratch. This outcome underlines the potential efficacy of our pre-training approach.

----

| Method     | S3DIS Area 5 | Semantic Seg |      |      | S3DIS Area 5 | Detection   |
| ---------- | ------------ | ------------ | ---- | ---- | ------------ | ----------- |
|            | mIoU         | OA           |      |      | mAP$_{0.25}$ | mAP$_{0.5}$ |
| PointNeXt† | 70.8         | 90.7         |      |      | -            | -           |
| MaskFeat3D | **71.7**     | **91.7**     |      |      | -            | -           |
| FCAF3D†    | -            | -            |      |      | 66.7         | 45.9        |
| MaskFeat3D | -            | -            |      |      | **71.6**     | **49.2**    |

**Supp Table 4: Area 5 Semantic segmentation and detection results on S3DIS.** † represents the *from scratch* results and MaskFeat3D in the same section represents the *fine-tuning* results using pretrained weights under same backbone.

----



To further validate the effectiveness of our method, we have conducted additional experiments. As we claimed that our method is encoder-agnostic, we experimented with another commonly used encoder backbone, Sparse-UNet [2]. During pre-training, we maintained identical settings, only replacing the encoder backbone with Sparse-UNet.  The results of this experiment are as follows:

| Method             | Backbone        | S3DIS Area 5 mIoU | S3DIS 6-Fold mIoU | ScanNet mIoU |
| ------------------ | --------------- | ----------------- | ----------------- | ------------ |
| Train from scratch | Sparse-UNet     | 68.2              | 73.6              | 72.2         |
| PointContrast      | Sparse-UNet     | 70.9              | -                 | 74.1         |
| DepthContrast      | Sparse-UNet     | 70.6              | -                 | 71.2         |
| **MaskFeat3D**     | **Sparse-UNet** | **72.3**          | **76.4**          | **74.7**     |

These results, spanning S3DIS Area 5, S3DIS 6-Fold, and ScanNet datasets, consistently show improvements over training from scratch and outperform other pre-training strategies under the same encoder backbone. This evidence supports the effectiveness of our method for more challenging downstream tasks.



[1] Qian, Guocheng, et al. "Pointnext: Revisiting pointnet++ with improved training and scaling strategies." In NeurIPS 2022.

[2] Choy, Christopher, JunYoung Gwak, and Silvio Savarese. "4D spatio-temporal convnets: Minkowski convolutional neural networks." In CVPR 2019.