Paper: SPOT: Scalable 3D Pre-training via Occupancy Prediction for Autonomous Driving (ICLR-2024)

Reviewer comment (Novelty):
Some of the contributions highlighted by the authors appear to be not novel enough, e.g., the class-balancing strategies. Existing studies [1, 2], have showcased similar strategies. It would be good to acknowledge, cite, and compare their work with these prior studies.

Author response:
A1:  Thanks for your comments, both methods present excellent self-training category balancing methods. In the process of self-training, Class-balanced self-training (CBST) [1] sets different thresholds for different classes, and then balances classes according to the confidence of false tags, so that false tags with low confidence of rare classes can participate in the training. Based on the CBST, Class-range-balanced Self-training (CRB-ST) [2] adds range-based considerations,  and sample pseudo-labels both from both category and range dimensions. Unlike them, we are mainly from a global perspective to alleviate the problem of class imbalance in the pre-training process. First, to ensure the integrity of occupancy labels in the scene, class sampling is based on frame-level. Secondly, considering that foreground information is more important than background information, and a large amount of background information can easily drown out scarce foreground information, we only rebalance for foreground categories, which can bring benefits to both downstream detection and segmentation tasks.

Reviewer comment (Experiments):
In Sec. 4.1, the authors claim that "our experiments are under label-efficiency setting." However, contemporary dataset subsampling techniques seem to encompass more than just the "randomly selected" method utilised in this paper. In fact, drawing from semi-supervised methodologies, the "uniform sampling" technique appears to be more prevalent. Moreover, Li et al. introduced the ST-RFD method, which aims to extract a more diverse subset of training data frame samples. I believe it would be beneficial for the authors to explore different sampling techniques and consider integrating the ST-RFD method to potentially achieve better downstream subset(s).

Author response:
A2:  We strongly agree with you that better sampling for downstream datasets leads to better performance. Since we do not use unlabeled data in the downstream dataset, we only perform fine-tuning experiments and fair comparison experiments on the same batch of randomly sampled data. Also, we have added some experimental results with uniform sampling. In the future, we will also integrate different sampling methods, such as the ST-RFD method [3], into the fine-tuning pipeline for better performance.

Table 19. Different sampling strategies on downstream task.

| Method | Sampling Strategy | F.D.A.  | mIOU |
| --- | --- | --- | --- |
| From Scratch (Cylinder3D) | random  | 10% NuScenes | 53.72 |
| SPOT (Cylinder3D) | random  | 10% NuScenes | 56.10 |
| From Scratch (Cylinder3D) | uniform  | 10% NuScenes | 52.96 |
| SPOT (Cylinder3D) | uniform  | 10% NuScenes | 56.14 |

Reviewer comment (Novelty):
While the authors introduce a two-stage approach of first pre-training followed by fine-tuning, I'm still uncertain about the main advantages of that idea. For instance, current research employing semi-supervised or active learning techniques for point cloud semantic segmentation [3, 4] seems to achieve superior results on 10% of the SemanticKITTI dataset (mIoU: 62.2 [3], mIoU: 60.0 [4]). Furthermore, Unal et al. [2] obtain a 61.3 mIoU on ScribbleKITTI (weakly annotated with 8% of the SemanticKITTI points). These methods also seem to efficiently utilise labels and alleviate the burden of extensive labelling.

Author response:
A3:  [2, 3, 4] are excellent semi-supervised or weakly supervised methods in the field of 3D point cloud segmentation, and they use the labeled data and other pseudo-labels of unlabeled data to achieve excellent performance, but most of these semi-supervised methods are task-specific or dataset-specific, e.g., they can only work in segmentation tasks or detection tasks. In contrast, SPOT focuses on the pre-training part, aiming to bring better transferable prior knowledge to all other tasks and datasets through just one pre-training process, which is decoupled from the downstream datasets and tasks. Moreover, we have not used downstream unlabeled data, meaning that we can combine SPOT with various semi-supervised or weakly-supervised methods in downstream fine-tuning to further improve the performance, by using SPOT as the downstream initialization. We think it's going to be a very effective combination.

Reviewer comment (Novelty):
Lack of related work, e.g., [1-4].

Author response:
A4:  Thank you for pointing these out! We has added all of them in the revision.

Reviewer comment (Experiments):
Unfair comparisons in Table 1 and 2: SPOT is compared against BEV-MAE and AD-PT, which are self-supervised and semi-supervised pre-training methods, respectively. This is an unfair comparison, as SPOT is a supervised pre-training method that benefits from human-labeled data. A more fair comparison would be to compare SPOT against other supervised pre-training methods.

Author response:
**Q1: Unfair comparisons with BEV-MAE and AD-PT**

**A1:** The motivation of SPOT is to pre-train a general backbone for different datasets and tasks, which is not achieved by BEV-MAE and AD-PT. For BEV-MAE, they are only able to pre-train and fine-tune on the same dataset. Tables 11, 12, 13 and 14 below show this phenomenon. For AD-PT, fine-tuning on semantic segmentation task harm the performance. Tables 12 and 13 show that when fine-tuning on LiDAR semantic segmentation task, AD-PT brings smaller improvement or even harm the performance.

To make more fair comparisons to AD-PT and also demonstrate the scalability of SPOT, we perform semi-supervised pre-training with SPOT. Results are provided in Tables 11, 12, 13 and 14 below. It can be found that pre-training with 5% sequence-level labeled data and 15% sequence-level unlabeled data achieves comparable downstream fine-tuning performance as that of pre-training with 20% sequence-level labeled data. Although 20% data in Waymo is much less than that used in AD-PT, it can be found that SPOT still achieve comparable or surpass AD-PT in various downstream datasets and tasks. We are also running experiments with more unlabeled data but due to the time limitation the experiments are not finished yet. We would update the results as soon as they finish. However, scalability of SPOT can also be discovered when we compare pre-trainings with different amount of unlabeled data (5% and 15%). It can be found that more unlabeled data brings better downstream performance. These results verify that (1) SPOT can learn general representations for different downstream datasets, tasks and architectures with small part of labels. (2) When incorporating more unlabeled data into pre-training phase, SPOT also shows scalability. 

Table 11. Comparison to different pre-training methods. The downstream task is 3D object detection on NuScenes. We provide additional semi-supervised pre-training via SPOT here.

| Backbone    | Method | Pre-training Setting | Pre-training Data Amount | F.D.A.  | mAP  | NDS |
| --- | --- | --- | --- | --- | --- | --- |
| SECOND | From Scratch | - | - | 5% Nuscenes | 32.16 | 41.59 |
| SECOND | BEV-MAE | Unsupervised | 100% | 5% Nuscenes | 32.09 | 42.88 |
| SECOND | AD-PT | Semi-supervised | 100% | 5% Nuscenes | 37.69  | 47.95 |
| SECOND | SPOT (SEMI) | Semi-supervised | 20% | 5% Nuscenes | 39.81 | 51.51 |
| SECOND | SPOT | Fully-supervised | 20% | 5% Nuscenes | 39.63 | 51.63 |
| CenterPoint | From Scratch | - | - | 5% Nuscenes | 42.37  | 52.01 |
| CenterPoint | BEV-MAE | Unsupervised | 100% | 5% Nuscenes | 42.86  | 52.95 |
| CenterPoint | AD-PT | Semi-supervised | 20% | 5% Nuscenes | 44.99  | 52.99 |
| CenterPoint | SPOT (SEMI) | Semi-supervised | 20% | 5% Nuscenes | 45.18 | 54.98 |
| CenterPoint | SPOT  | Fully-supervised | 20% | 5% Nuscenes | 44.94 | 54.95 |

Table 12. Comparison to different pre-training methods. The downstream task is 3D object detection on KITTI. We provide additional semi-supervised pre-training via SPOT here.

| Backbone    | Method | Pre-training Setting | Pre-training Data Amount | F.D.A.  | mAP  |
| --- | --- | --- | --- | --- | --- |
| SECOND | From Scratch | - | - | 20% KITTI | 61.70 |
| SECOND | BEV-MAE | Unsupervised | 100% | 20% KITTI | 63.45 |
| SECOND | AD-PT | Semi-supervised | 100% | 20% KITTI | 65.95 |
| SECOND | SPOT (SEMI) | Semi-supervised | 20% | 20% KITTI | 65.45 |
| SECOND | SPOT | Fully-supervised | 20% | 20% KITTI | 66.45 |
| PV-RCNN | From Scratch | - | - | 20% KITTI | 66.71 |
| PV-RCNN | BEV-MAE | Unsupervised | 100% | 20% KITTI | 69.91 |
| PV-RCNN | AD-PT | Semi-supervised | 20% | 20% KITTI | 69.43 |
| PV-RCNN | SPOT (SEMI) | Semi-supervised | 20% | 20% KITTI | 70.86 |
| PV-RCNN | SPOT  | Fully-supervised | 20% | 20% KITTI | 70.85 |

Table 13. Comparison to different pre-training methods. The downstream task is LiDAR semantic segmentation on NuScenes. We provide additional semi-supervised pre-training via SPOT here.

| Backbone    | Method | Pre-training Setting | Pre-training Data Amount | F.D.A.  | mIOU |
| --- | --- | --- | --- | --- | --- |
| Cylinder3D | From Scratch | - | - | 5% Nuscenes | 45.85 |
| Cylinder3D | BEV-MAE | Unsupervised | 100% | 5% Nuscenes | 46.94 |
| Cylinder3D | AD-PT | Semi-supervised | 100% | 5% Nuscenes | 45.61 |
| Cylinder3D | SPOT (SEMI) | Semi-supervised | 20% | 5% Nuscenes | 48.84 |
| Cylinder3D | SPOT | Fully-supervised | 20% | 5% Nuscenes | 47.84 |

Reviewer comment (Novelty):
Supervised occupancy prediction is not new, and it is hard to argue scalability when pre-training is based on a labeled dataset. On the other hand, self-supervised occupancy pretraining was demonstrated in previous works, e.g. [1], which had shown unlabeled occupancy pretraining via MAE work.

Author response:
**Q3: Supervised occupancy prediction is not new, and It is hard to argue scalability when pre-training is based on a labeled dataset, and self-supervised occupancy pretrainig was demonstrated in previous works.**

**A3:**  We agree with the reviewer that, supervised occupancy prediction is indeed not a new emerging field, but we are the first to build a unified pre-trained model by treating it as a pre-training task and directly adapt it to different downstream tasks and datasets. In contrast, current work such as Occupancy-MAE [1] can only be fine-tuned on the current dataset by pre-training it on the current dataset, which means that for each downstream dataset, pre-training needs to be performed again. Such a one-to-one experimental setting (perform the pre-training and fine-tuning on the same dataset) causes that the pre-trained backbone is hard to be deployed to different tasks and scenes.

Reviewer comment (Experiments):
Another critical limitation on scalability is that the pre-training dataset must have a more expensive lidar setup (64-beam with high density) compared to the fine-tuning dataset.

Author response:
**Q4: Another critical limitation on scalability is that the pre-training dataset must have a more expensive lidar setup (64-beam with high density) compared to the fine-tuning dataset.**

**A4:**   Building occupancy labels does not require a very expensive lidar setup, as NuScenes only has 16beam compared to Waymo's 64-beam, while the NuScenes dataset also has dense occupancy labels. Furthermore, our semi-supervised experiments also demonstrate that SPOT's reliance on occupancy labels is secondary, and more pre-training labels can be built using simple pseudo-labeling and consistent gain across all downstream tasks. Finally, we conducted pre-training experiments on the Waymo dataset with only 16-beam as input, and cross-domain experiments on the NuScenes dataset, and the experimental results show that SPOT can still achieve a good performance improvement.

Table 16. Pre-training on the Waymo dataset with only 16beam as input and fine-tune the pre-trained backbones for 3D object detection on NuScenes dataset. 

| Method | P.D.A. | F.D.A.  | mAP | NDS |
| --- | --- | --- | --- | --- |
| From Scratch (CenterPoint) |  -  | 5% NuScenes | 42.37 | 52.01 |
| SPOT_5% (CenterPoint) | random 5% | 5% NuScenes | 44.42   | 54.42 |
| SPOT_5%_16beam (CenterPoint) | random 5% | 5% NuScenes  | 44.24   | 53.45 |

Reviewer comment (Experiments):
Low train from scratch baseline performance: As shown in Fig 7, with 100% nuScenes, training from scratch leads to mAP ~ 50, which is far from SOTA (>70) [2].

Author response:
**Q5: Low train from scratch baseline performance: As shown in Fig 7, with 100% NuScenes, training from scratch leads to mAP ~ 50, which is far from SOTA (>70).**

**A5:** Thanks for pointing this out. Our motivation is to improve label-efficiency performance on various downstream tasks and datasets via **one** pre-trained model. We totally agree that it is also important to improve SOTA models. Most models that achieve >70 NDS performance on NuScenes Leaderboard is multi-modality methods with camera inputs but we focus on LiDAR perception. To demonstrate SPOT can improve SOTA model, we perform fine-tuning experiments with 100% full data on the VoxelNext [2] model, which is one of SOTA LiDAR-only models on NuScenes benchmark. And the results in Table 17 prove that SPOT can still achieve gains on SOTA models.

Table 17. Fine-tune the pre-trained backbones for VoxelNext on 100% NuScenes 3D object detection dataset. 

| Method | P.D.A. | F.D.A.  | mAP | NDS |
| --- | --- | --- | --- | --- |
| From Scratch (VoxelNext) |  -  | 100% NuScenes | 60.53 | 66.65 |
| SPOT  (VoxelNext) | 100% | 100% NuScenes | 61.05 | 66.90 |

Reviewer comment (Experiments):
Given the low train from scratch performance, it is possible that the baseline model is undertrained or using a weaker data augmentation.

Author response:
**Q6: Could the baseline model be undertrained or that it is using a weaker data augmentation?**

**A6:**  The data augmentations we used all follow the OpenPCDet settings, including gt sampling, random flip/rotate/translate, etc., which is a consistent approach for current 3D detection models. Besides, we conduct experiments on extending the training schedule on NuScenes dataset, and the results are shown as follows. From the results, it can be seen that even when the training schedule is extending by 5 times at least, the initialization by our pre-trained model can achieve higher performance. Especially for the Transformer-based DSVT detector, the result of just 20 epochs is even superior to the result of 150 epochs.

Table 18. Extending training schedule on NuScenes dataset. 

| Method | P.D.A. | F.D.A.  | Training Schedule | mAP | NDS |
| --- | --- | --- | --- | --- | --- |
| From Scratch (SECOND) |  -  | 5% NuScenes | 30 epochs | 32.16  | 41.59 |
| From Scratch (SECOND) |  -  | 5% NuScenes | 150 epochs | 36.79  | 51.01 |
| SPOT (SECOND) | 20% | 5% NuScenes | 30 epochs | 39.63 | 51.63 |
| SPOT (SECOND) | 100% | 5% NuScenes | 30 epochs | 42.57  | 54.28 |
| From Scratch (CenterPoint) | - | 5% NuScenes | 30 epochs | 42.37 | 52.01 |
| From Scratch (CenterPoint) | - | 5% NuScenes  | 150 epochs | 41.01  | 53.92 |
| SPOT (CenterPoint) | 20% | 5% NuScenes | 30 epochs | 44.94 | 54.95 |
| SPOT (CenterPoint) | 100% | 5% NuScenes | 30 epochs | 47.47  | 57.11 |
| From Scratch (DSVT) |  -  | 5% NuScenes | 20 epochs | 49.78 | 58.63 |
| From Scratch (DSVT) |  -  | 5% NuScenes | 150 epochs | 54.30 | 63.58 |
| SPOT (DSVT) | 20% | 5% NuScenes | 20 epochs | 56.65 | 63.52 |

Reviewer comment (Evaluation):
The improvement is marginal (only +0.25 NDS) for a stronger baseline (VoxelNext), as shown in Table 17.

Author response:
**Q1: The seemingly marginal improvement for VoxelNext.**

First of all, in our pre-training, both 3D and 2D (BEV) backbones are used for occupancy prediction but VoxelNext [1] only uses 3D backbone for encoding. Thus, we are only able to load weights of 3D backbone when fine-tuning on NuScenes, which losses useful information from pre-training. Under such circumstances, we also achieve 0.52\% mAP improvement. It is a well-known phenomenon in previous works [2,3,4] that improving over SOTA models with 100% downstream labels is hard and 0.52\% mAP is non-negligible improvement.

To further demonstrate the ability of SPOT to improve SOTA performance on NuScenes, we perform two additional experiments: (1) training VoxelNext on train and validation set and then test it on **Test** set. (2) fine-tuning on CenterPoint [5], which is another SOTA model on NuScenes, and it has both 3D and 2D backbone for embedding point clouds, leading to less information loss.

(1) We train VoxelNext on train and validation set and then test it on **Test** set, which is a standard setting for **test set**. It can be seen from the following table that, although only pre-trained parameters of 3D backbone are loaded, SPOT can boost the performance of VoxelNext on **test set** by 1.0\% mAP and 0.7\% NDS, respectively. More than 200\% improvement are gained compared to previous results and this improvement is significant on SOTA models as stated in previous works [2,3,4].

Table. Fine-tune the pre-trained backbones for VoxelNext on 100% NuScenes 3D object detection trainval set and **evaluate on test set.** 

| Method | F.D.A.  | mAP | NDS |
| --- | --- | --- | --- |
| From Scratch (VoxelNext) | 100% NuScenes trainval Set | 64.5 | 70.0 |
| SPOT  (VoxelNext) | 100% NuScenes trainval Set | **65.5** | **70.7** |

(2) Furthermore, to avoid information loss, we employ the CenterPoint [5] (voxel_size=0.075), which utilizes both 3D and 2D backbones for encoding, for 3D object detection task on NuScenes. We load pre-trained weights for both 3D and 2D backbones from SPOT and fine-tune CenterPoint with full training data. In the Table below, it can be found that compared to training from scratch, initializing CenterPoint by SPOT brings 2.63\% mAPs and 1.25\% NDS improvement. Thus, without loss of pre-trained information, that is loading the full amount of pre-training parameters, SPOT is able to achieve far more significant performance improvement on SOTA models.

Table. Fine-tune the pre-trained backbones for CenterPoint (voxel_size=0.075) on 100% NuScenes 3D object detection dataset. 

| Method | F.D.A.  | mAP | NDS |
| --- | --- | --- | --- |
| From Scratch (CenterPoint) | 100% NuScenes train set | 59.28 | 66.60 |
| SPOT  (CenterPoint) | 100% NuScenes train set | **61.91** | **67.85** |

We hope these discussions and experiment results can cover your concerns on the improvement on SOTA models. Thanks.

Reviewer comment (Novelty):
As reviewer y7L9 suggested, "the improvements and results showcased appear anticipated."

Author response:
**Q2:** As reviewer y7L9 suggested, `the improvements and results showcased appear anticipated`.

We provide discussion on this concern in **Q2** of [Response to Reviewer y7L9](https://openreview.net/forum?id=9zEBK3E9bX&noteId=ctrbWDv7Xw) and we are glad to provide a further discussion here on why we believe that purely using large-scale annotated dataset for pre-training to bring the improvements is **not anticipated.**

Firstly, it has been demonstrated in previous works [6,7] that directly using Waymo annotated data ****cannot**** help improve the performance of other datasets such as NuScenes, which is mainly due to huge domain differences between different 3D datasets,  especially for the NuScenes dataset. Also, the works [8,9] also shows that the pre-trained model on the Waymo dataset has very limited performance when transferring to other datasets. Cross-domain adaptation between different datasets is still one of the challenges in the field of 3D outdoor point cloud perception. We cite the results reported from [6] here. From the following table, it can also be observed that, large-scale annotated Waymo dataset does not improve the model performance on NuScenes dataset, but brings ****a significant performance drop**** on NuScenes. The performance drop mainly is due to the domain differences (including class definitions, sensor setting and scenarios differences) from both datasets.

Table. Domain adaptation performance under Waymo→NuScenes with different amounts of target-domain labels between Waymo and NuScenes. **Note that the results here are evaluated and reported only in the car category [6], which is different from the overall performance reported in our experiments.**

| Training Method | 5% NuScenes (mAP/ NDS on the category of CAR) | 10% NuScenes (mAP/ NDS on the category of CAR) | 20% NuScenes (mAP/ NDS on the category of CAR) | 100% NuScenes (mAP/ NDS on the category of CAR) |
| --- | --- | --- | --- | --- |
| Only NuScenes | 61.0 / 53.2 | 65.6 / 58.2 | 70.2 / 63.0 | 78.4 / 69.9 |
| Using extra large-scale Waymo annotated data | 57.7 / 58.0 | 59.4 / 58.9 | 63.1 / 61.1 | 66.5 / 63.5 |

Secondly, SPOT demonstrates the superiority of occupancy prediction as pre-training tasks to learn general 3D representations. By “general”, we mean that SPOT only pre-train the backbones for one time and these pre-trained weights are able to improve different baselines on various downstream datasets and tasks. In comparison, pre-training by 3D object detection or LiDAR semantic segmentation tasks with fully-annotated Waymo datasets suffers from two gaps when fine-tuned on downstream tasks: (1) The gap between pre-training task and downstream task. In the Table below, it can be found that pre-training on semantic segmentation task harms the performance a lot. (2) The gap between different LiDARs used to collect datasets. When pre-training via detection task, the performance on downstream dataset also drops due to different sensors used in these two datasets. Thus **annotation itself does not directly bring improvements**.

Table. Pre-training with different tasks with annotations on Waymo Dataset and fine-tune the pre-trained backbones for 3D object detection on NuScenes dataset. 

| Pre-training Task  | P.D.A. | Downstream Task | mAP | NDS |
| --- | --- | --- | --- | --- |
| From Scratch | - | 5% NuScenes | 42.37 | 52.01 |
| Detection Task on Upstream | 100% | 5% NuScenes | 40.89 | 49.75 |
| Segmentation Task on Upstream | 100% | 5% NuScenes | 36.23 | 47.01 |
| SPOT | 5%  | 5% NuScenes | 43.56 | 53.04 |
| SPOT (SEMI) | 5%  + Unlabeled 15% | 5% NuScenes | 45.18 | 54.98 |

Thirdly, purely applying occupancy prediction for pre-training **is not enough.** Our proposed beam-sampling augmentation and class-balancing strategies also play important role in achieving such general 3D representations. As shown in the Table below, we conduct ablation study on different components of SPOT  and it can be found that purely applying occupancy prediction as pre-training task bring **much less improvement** than that of SPOT.

Table. Ablation study on pre-training strategies and fine-tuning performance across different datasets. 

| Occupancy Prediction | Loss Balancing   | Beam Re-sampling | Dataset Balancing | NuScenes (mAP / NDS) | ONCE(mAP) | KITTI(mAP) |
| --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  | 32.16 / 41.59  | 35.96  | 61.70 |
| √ |  |  |  | 36.55 / 46.98  | 36.00  | 63.70 |
| √ | √ |  |  | 37.90 / 47.82  | 37.30  | 64.70 |
| √ | √ | √ |  | 38.63 / 48.85  | 39.19  | 65.92 |
| √ | √ | √ | √ | 40.39 / 51.65 | 40.63  | 66.45 |