Paper: Multi-View Representation is What You Need for Point-Cloud Pre-Training (ICLR-2024)

Reviewer comment (Writing):
The **masking ratio** is *only discussed* in the ablation study, which is very confusing. Some more details about this masking operation should be discussed in the method section. For example, why do you use this operation? Why can it benefit the proposed method?

Author response:
We express our gratitude to the reviewer for the meticulous attention in pointing this out. We intend to provide a more detailed explanation in our revised manuscript. Our method involved masking the input point cloud guided by the corresponding RGB image. Following the conventional practice in masked autoencoder-based pre-training [1], we divided the image into regular non-overlapping patches. Then we sample a subset of patches and mask (i.e., remove) the remaining ones. The remaining part is projected onto the point cloud, forming the input for our method. We employ this masking operation as a data augmentation technique, which assists in the development of more robust features during pre-training. As evidenced in Table 4(b) (referenced below), an appropriate masking operation during pre-training could enhance performance in downstream tasks.

| Mask Ratio (%) | ScanNet  |
| -------------- | -------- |
| 10             | 74.8     |
| 30             | **75.6** |
| 60             | 74.2     |
| 80             | 73.5     |

Reviewer comment (Novelty):
There are several **important recent related works** [1, 2] that should be discussed in Related Works and compared in the tables in the main paper.

Author response:
We are thankful to the reviewer for bringing these related works to our attention. We will include a comprehensive discussion of these works in our revised manuscript. Mask3D introduces a method of pre-training in 2D by harnessing 3D priors, with the aim of enhancing 2D task performance. Our approach, conversely, seeks to apply 2D knowledge to augment 3D tasks, establishing a reversed transfer of knowledge. Given the differing objectives, a direct comparison with Mask3D is not straightforward.

MM-3DScene has proposed an innovative masking model for point cloud pre-training. We have conducted a comparative analysis with their method on the object detection task, and the results are presented in the table that follows. Our method outperforms MM-3DScene in most cases.

| Method             | Architecture | ScanNet mAP @0.25 | ScanNet mAP @0.5 | SUN RGB-D mAP @0.25 | SUN RGB-D mAP @0.5 |
| ------------------ | ------------ | ---------------- | --------------- | ------------------ | ----------------- |
| Train from scratch | VoteNet      | 58.7             | 35.4            | 57.7               | 32.9              |
| MM-3DScene         | VoteNet      | 63.1             | **41.5**        | 60.6               | 37.3              |
| MVNet              | VoteNet      | **64.0**         | **41.5**        | **62.0**           | **39.6**          |


We also compare our method with MM-3DScene on the semantic segmentation task. Since MM-3DScene employs Point Transformer as the encoder backbone, we ensured a fair comparison by implementing Point Transformer as the backbone during our method's pre-training phase. The results, which are presented in the subsequent table, demonstrate that our method outperforms MM-3DScene under the same experimental setting.

| Method             | Backbone          | ScanNet mIoU | S3DIS Area5 mIoU |
| ------------------ | ----------------- | ------------ | ---------------- |
| Train from scratch | Point Transformer | 70.6         | 70.4             |
| MM-3DScene         | Point Transformer | 72.8         | 71.9             |
| MVNet              | Point Transformer | **75.2**     | **73.5**         |

Reviewer comment (Reproducibility):
The proposed method requires the **pre-processing** on RGB-D scans to generate point clouds. This operation may bring extra processing time and disk usage. How long will it take? How much extra storage will this need? This should be at least discussed in the supplementary material. Also, this may be a limitation of the proposed method, which should be mentioned in the limitation section.

Author response:
We extend our gratitude to the reviewers for bringing this to our attention. We acknowledge the significance of this point and will address it, adding a corresponding note to the limitations section in our revised manuscript. The process of projecting RGB-D scans to point clouds requires approximately 10 hours on a 64-core CPU machine. Storaging the generated point cloud would require  ~600G extra space.

Reviewer comment (Presentation):
Some **qualitative/visualization results** of different tasks such as 3D segmentation, and detection should be provided in the supplementary material.

Author response:
We are thankful to the reviewer for the suggestion. In response, we have included additional qualitative results for the semantic segmentation task in the updated supplementary material.

Reviewer comment (Experiments):
Given that the model can generate dense features containing rich semantics, I strongly recommend incorporating zero-shot experiments, including zero-shot classification experiments on datasets such as ModelNet and LVIS, as well as zero-shot semantic segmentation experiments on ScanNet and S3DIS.

Author response:
We extend our thanks to the reviewer for highlighting this aspect. Our method, not being designed to align learned features with the language domain, faces challenges in direct application to zero-shot experiments like PointCLIP [1] or ULIP [2]. Nevertheless, to demonstrate the effectiveness of our approach, we have included results from few-shot learning experiments conducted on ModelNet40. These results can be found in Table 3 of the Supplementary Material. The table below also presents additional results pre-trained on ShapeNet for a comprehensive view.

| Model        | Pre-train dataset | 5-way 10-shot  | 5-way 20-shot  | 10-way 10-shot | 10-way 20-shot |
| ------------ | -------------- | -------------- | -------------- | -------------- | -------------- |
| Transformer† | - | 87.8 ± 5.2     | 93.3 ± 4.3     | 84.6 ± 5.5     | 89.4 ± 6.3     |
| OcCo         | ShapeNet | 94.0 ± 3.6     | 95.9 ± 2.3     | 89.4 ± 5.1     | 92.4 ± 4.6     |
| PointBERT    | ShapeNet | 94.6 ± 3.1     | 96.3 ± 2.7     | 91.0 ± 5.4     | 92.7 ± 5.1     |
| MaskDiscr    | ShapeNet | 95.0 ± 3.7     | 97.2 ± 1.7     | 91.4 ± 4.0     | 93.4 ± 3.5     |
| PointMAE     | ShapeNet | 96.3 ± 2.5     | 97.8 ± 1.8     | 92.6 ± 4.1     | 95.0 ± 3.0     |
| MVNet-B       | ShapeNet | 96.9 ± 1.9 | 98.1 ± 1.7 | 93.1 ± 3.5 | 95.4 ± 3.4 |
| MVNet-B       | Objaverse | 97.2 ± 1.8 | 98.3 ± 1.8 | 93.4 ± 3.4 | 95.8 ± 3.2 |

To further prove the efficacy of our method, we conducted the label-efficient learning experiments on the scene-level SUN RGB-D object detection task. During these experiments, we trained our model using varied proportions of the dataset, specifically 20%, 50%, and 100%, and subsequently evaluated the performance on the same test dataset. We reported the mAP @0.5 results in the table below. Our observations indicate that our pre-training method yields more significant gains with reduced amounts of labeled training data. Notably, with only 50% of the training data, our method surpasses the performance achieved by training from scratch using the full 100% dataset.

| Method             | 20%      | 50%      | 100%     |
| ------------------ | -------- | -------- | -------- |
| Train from scratch | 18.8     | 24.7     | 32.9     |
| PointContrast      | 24.5     | 29.2     | 37.5     |
| 4DContrast         | 26.3     | 31.5     | 38.2     |
| MVNet              | **28.5** | **33.2** | **39.6** |

We hope that these relevant experiments will effectively address the reviewer's concerns.

[1] Zhang, Renrui, et al. "Pointclip: Point cloud understanding by clip." In CVPR 2022.

[2] Xue, Le, et al. "ULIP: Learning a unified representation of language, images, and point clouds for 3D understanding." In CVPR 2023.

Reviewer comment (Presentation):
Some papers are repeatedly cited in the bib, such as: "Charles R. Qi, Or Litany, Kaiming He, and Leonidas J. Guibas. Deep hough voting for 3d object detection in point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019a."

Author response:
We thank the reviewer for pointing this out! The issue has been fixed in the updated submission.

Reviewer comment (Presentation):
The result of ScanObjectNN in Figure 1 is different from the result in Table 3.

Author response:
We thank the reviewer for highlighting this observation. The result for ScanObjectNN presented in Figure 1 reflects the classification accuracy of MVNet-L in the ScanObjectNN PB-T50-RS task, which is 91.0. This is indeed consistent with the result reported in Table 3. We will clarify this in our revised manuscript.

Reviewer comment (Theory):
Why not directly use the transformation matrix of two views as training targets within the multi-view consistency module?

Author response:
We appreciate the reviewer for raising this intriguing question. In our experiments, we did explore the possibility of predicting the transformation matrix.  However, we found that predicting dense correspondences yielded more accurate results than predicting the transformation matrix directly. This is likely due to the fact that the correspondences are more straightforward for the network to learn from the visual cues than the SE(3) transformation parameters. Furthermore, pre-training with the prediction of the transformation matrix demonstrated inferior results compared to predicting correspondence. We included an ablation study on the ScanNet semantic segmentation task to illustrate this point. The mIoU results, presented in the table below, clearly indicate that predicting dense correspondences during pre-training captures superior features.

| Multi-view Consistency Module | ScanNet mIoU |
| ----------------------------- | ------------ |
| Transformation matrix         | 75.0         |
| Dense corrspondences          | 75.6         |

Reviewer comment (Reproducibility):
Is voting strategy employed in the classification of object point clouds?

Author response:
We did not apply any voting strategies on point cloud classification tasks for fair comparison.

Reviewer comment (Evaluation):
I am curious to understand why SAM yields inferior performance as a 2D encoder compared to DINOv2.

Author response:
We thank the reviewer for the question! Unfortunately, we are unable to provide a theoretical explanation to the difference in performance. However, we highlight that existing works [3] [4] have also successfully recovered dense correspondences from DINO and DINOv2.



[3] Amir, Shir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. "Deep vit features as dense visual descriptors." arXiv preprint arXiv:2112.05814 2, no. 3 (2021): 4.

[4] Zhang, Junyi, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. "A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence." arXiv preprint arXiv:2305.15347 (2023).

Reviewer comment (Experiments):
The experimental comparison with previous point cloud pre-training approaches is problematic because the adopted backbone models differ. To evaluate the effectiveness of the proposed pre-training framework, the authors should make sure that the performance gains are not from the stronger backbone.

Author response:
**Q: The experimental comparison with previous point cloud pre-training approaches is problematic.** 

We are grateful to the reviewer for highlighting this aspect. However, we maintain that we have diligently ensured a fair comparison in our study.

In the scene-level semantic segmentation task, as shown in Table 1 of the main paper, we selected SR-UNet as the encoder backbone, which is also commonly used in most recent works [3] [4] [5]. For other more earlier related works used DGCNN as the backbone, we show the relevant results in the following table.

| Method             | Backbone | S3DIS Area 5 mIoU | S3DIS 6-Fold mIoU | ScanNet mIoU |
| ------------------ | -------- | ----------------- | ----------------- | ------------ |
| Train from scratch | DGCNN    | 47.9              | 56.1              | 50.2         |
| Jigsaw             | DGCNN    | 52.1              | 56.6              | -            |
| OcCo               | DGCNN    | 55.4              | 58.5              | -            |
| STRL               | DGCNN    | 51.8              | 57.1              | -            |
| MVNet              | DGCNN    | **61.5**          | **66.2**          | **63.0**     |

For the scene-level detection task, our methodology is consistent with the current standard in the field, as outlined in [3] [4] [6]. We utilize SR-UNet as the encoder, coupled with VoteNet for the detection architecture. This configuration enabled us to demonstrate notable improvements under an equivalent setup. However, as we mentioned in Section 4.2, *"VoteNet’s performance lags behind the state-of-the-art. Therefore, we also evaluate our approach on CAGroup3D, which offers a stronger decoder architecture and delivers state-of-the-art 3D detection performance."* It is important to note that in the CAGroup3D experiment, the only modification was the decoder architecture; the encoder backbone remained SR-UNet. We will provide further clarification on this aspect in our revised manuscript.

In the shape-level task, as presented in Table 3, all the methods compared utilize the same Transformer as the encoder backbone. To ensure a fair comparison, we also used the same Transformer as the encoder backbone, referred to as 'MVNet-B'. Under this consistent setting, our method demonstrates significant improvement.

In conclusion, we believe that our comparisons across different tasks have been conducted with fairness and in alignment with current standards in the field.

[3] Xie, Saining, et al. "Pointcontrast: Unsupervised pre-training for 3d point cloud understanding." In ECCV 2020.

[4] Zhang, Zaiwei, et al. "Self-supervised pretraining of 3d features on any point-cloud." In ICCV 2021.

[5] Hou, Ji, et al. "Exploring data-efficient 3d scene understanding with contrastive scene contexts." In CVPR 2021.

[6] Chen, Yujin, Matthias Nießner, and Angela Dai. "4dcontrast: Contrastive learning with dynamic correspondences for 3d scene understanding." In ECCV 2022.

Reviewer comment (Experiments):
For object-level experiments, the pre-training dataset Objaverse (with 800K objects) is much larger than ShapeNet/ModelNet used in previous pre-training methods. Therefore, the performance comparison is unfair.

Author response:
**Q: Objaverse (with 800K objects) is much larger than ShapeNet.**

We are grateful to the reviewer for highlighting this aspect. In response, we pre-trained our model on the ShapeNet dataset, and show the ScanObjectNN PB-T50-RS classification results in the table below. Our model demonstrates superior performance in this context.

| Method             | Backbone    | Pre-train dataset | ScanObjectNN PB-T50-RS |
| ------------------ | ----------- | ----------------- | ---------------------- |
| Train from scratch | Transformer | -                 | 77.2                   |
| PointBERT          | Transformer | ShapeNet          | 83.1                   |
| Point-MAE          | Transformer | ShapeNet          | 85.2                   |
| ULIP               | Transformer | ShapeNet          | 86.4                   |
| MVNet-B            | Transformer | ShapeNet          | **86.7**               |

Reviewer comment (Experiments):
For different experiments, the authors are also suggested to report their results of training from scratch, such that readers can better observe the relative gains after pre-training the backbone using the proposed method.

Author response:
**Q: Report training from scratch results.**

We thank the reviewer for this insightful suggestion! We have included the results of training from scratch in Table 3 and will incorporate the training from scratch results in Tables 1 and 2 in our revised manuscript.