Paper: Pre-training LiDAR-based 3D Object Detectors through Colorization (ICLR-2024)

Reviewer comment (Experiments):
Authors use object detection as the downstream task, but I personally believe LiDAR-based segmentation is the best choice. Colorization-based pre-training mainly learns the semantics in my opinion, but object detection needs accurate locations and poses especially in the benchmark using IoU-based metrics such as KITTI and Waymo.

Author response:
**Downstream tasks.** 
Thanks for the comment. In our humble opinion, our colorization-based pre-training (i.e., GPC) does not just learn semantics but which nearby points belong to the same instance (e.g., cf. the last paragraph on page 4). In essence, the same semantic class can have different colors, even in the same scene (e.g., multiple cars of different colors in Fig. 4). Our method can accurately colorize each car instance (cf. Fig. 4 (d)), implying that it learns the notion of object instances, which we believe crucial for 3D object detection.

That said, during the rebuttal, we followed your suggestion to perform a preliminary study of GPC on LiDAR-point semantic segmentation. We use a UNet-like model as in PointContrast and perform experiments on NuScenes. While we did not have sufficient time to perform hyper-parameter tuning, we have already obtained a 6% gain on mIoU against training from scratch. We will include the comprehensive results in the final version.

Reviewer comment (Experiments):
The ultimate performance is not that good. The results on the Waymo Open Dataset are too weak. Other results on KITTI are also not convincing because PointRCNN and IA-SSD are not SOTA detectors nowadays.

Author response:
**Using SOTA detectors.**
We provide additional results of pre-training on Waymo and fine-tuning on KITTI (cf. Table 2), using PV-RCNN as the 3D detector. (We note that many SOTA results use PV-RCNN.) As shown in the following table, GPC consistently outperforms existing algorithms. Please also see our response **More comparisons in Table 1 and Table 2** to Reviewer Hnni for further discussions.

| Labels |      Method      | Overall mod. |  Car mod. | Ped. mod. | Cyc. mod. |
|:------:|:----------------:|:------------:|:---------:|:---------:|:---------:|
|   20%  | scratch          |     66.71    |   82.52   |   53.33   |   64.28   |
|        | ProposalContrast |     68.13    | **82.65** |   55.05   |   66.68   |
|        | PatchContrast    |     70.75    |   82.63   |   57.77   | **71.84** |
|        | GPC (Ours)       |   **71.37**  | **82.65** | **59.87** |   71.59   |
|   50%  | scratch          |     69.63    |   82.68   |   57.10   |   69.12   |
|        | ProposalContrast |     71.76    |   82.92   |   59.92   |   72.45   |
|        | PatchContrast    |     72.39    |   84.47   |   60.76   |   71.94   |
|        | GPC (Ours)       |   **72.75**  | **84.58** | **61.06** | **72.61** |
|  100%  | scratch          |     70.57    |   84.50   |   57.06   |   70.14   |
|        | GCC-3D           |     71.26    |     -     |     -     |     -     |
|        | STRL             |     71.46    |   84.70   |   57.80   |   71.88   |
|        | PointContrast    |     71.55    |   84.18   |   57.74   |   72.72   |
|        | ProposalContrast |     72.92    | **84.72** |   60.36   |   73.69   |
|        | ALSO             |     72.96    |   84.68   |   60.16   |   74.04   |
|        | PatchContrast    |     72.97    |   84.67   |   59.92   | **74.33** |
|        | GPC (Ours)       |   **73.58**  |   84.68   | **62.06** |   73.99   |

STRL: Huang  et al. "Spatio-temporal self-supervised representation learning for 3d point clouds." ICCV. 2021.

ALSO: Boulch et al. "Also: Automotive lidar self-supervision by occupancy estimation." CVPR. 2023.

GCC-3D: Liang et al. "Exploring geometry-aware contrast and clustering harmonization for self-supervised 3d object detection." ICCV. 2021.

PatchContrast: Shrout et al., "PatchContrast: Self-Supervised Pre-training for 3D Object Detection." arXiv preprint arXiv:2308.06985.

Reviewer comment (Writing):
Although the writing is clear, there are some unnecessary parts like the decorative math on page 4, which do not make the paper better.

Author response:
**Writing: decorative math.** 
We appreciate your feedback. We will try to squeeze Sect. 3 so we can include some rebuttal responses in the main paper.

That said, we respectfully think the math on page 4 is meaningful. While the concepts of color variation (e.g., the same object can have different color options) and hints may sound intuitive in hindsight, we think a more formal description of the problem and the solution is needed for the soundness of our paper. In particular, Sect. 3.1 justifies why the color variation cannot be resolved by designing a more complex neural network or training objective and why adding hints is an effective solution. Our ablation studies in Sect. 4.3, in particular, Table 8 and Fig. 4, further validate the importance of adding hints for pre-training with colorization.

Reviewer comment (Experiments):
According to my first concern in the weaknesses box, I wonder why not authors adopt segmentation as the downstream task? Or some other detection benchmarks relying on accurate semantics such as nuScenes? I believe the proposed method can obtain better performance on these tasks.

Author response:
**nuScenes dataset.**
We thank the reviewer for the suggestion. We have provided some preliminary results of segmentation in our first response, *Downstream tasks.* Meanwhile, we have tried our best to conduct 3D object detection experiments of pre-training on NuScenes and fine-tuning on 5% NuScenes, using CenterPoint. The results are summarized in the table below. The performance of GPC is close to the SOTA, TriCC, and better than previous algorithms. Please note that the performance shown here could be suboptimal due to the time constraint of the rebuttal period.

| Method        |    mAP   |    NDS   |
|---------------|:--------:|:--------:|
| scratch       |   38.0   |   44.3   |
| PointContrast |   39.8   |   45.1   |
| GCC-3D        |   41.1   |   46.8   |
| SLidR         |   43.3   |   52.4   |
| TriCC         | **44.6** | **54.4** |
| GPC (Ours)    |   44.1   |   53.3   |

TriCC: Pang et al. "Unsupervised 3D Point Cloud Representation Learning by Triangle Constrained Contrast for Autonomous Driving." CVPR. 2023.

SLidR: Sautier et al. "Image-to-lidar self-supervised distillation for autonomous driving data." CVPR. 2022.


***In light of our clarification and additional results, we would like to ask if the reviewer will reconsider the rating. Thank you.***

Reviewer comment (Experiments):
Authors should adopt SOTA detectors for experiments on KITTI, such as PVRCNN. The effectiveness should further be proven on Waymo Open Dataset with SOTA detectors such as PVRCNN, FSD, etc. The current performance is not competitive enough.

Author response:
**Using SOTA detectors.**
We provide additional results of pre-training on Waymo and fine-tuning on KITTI (cf. Table 2), using PV-RCNN as the 3D detector. (We note that many SOTA results use PV-RCNN.) As shown in the following table, GPC consistently outperforms existing algorithms. Please also see our response **More comparisons in Table 1 and Table 2** to Reviewer Hnni for further discussions.

| Labels |      Method      | Overall mod. |  Car mod. | Ped. mod. | Cyc. mod. |
|:------:|:----------------:|:------------:|:---------:|:---------:|:---------:|
|   20%  | scratch          |     66.71    |   82.52   |   53.33   |   64.28   |
|        | ProposalContrast |     68.13    | **82.65** |   55.05   |   66.68   |
|        | PatchContrast    |     70.75    |   82.63   |   57.77   | **71.84** |
|        | GPC (Ours)       |   **71.37**  | **82.65** | **59.87** |   71.59   |
|   50%  | scratch          |     69.63    |   82.68   |   57.10   |   69.12   |
|        | ProposalContrast |     71.76    |   82.92   |   59.92   |   72.45   |
|        | PatchContrast    |     72.39    |   84.47   |   60.76   |   71.94   |
|        | GPC (Ours)       |   **72.75**  | **84.58** | **61.06** | **72.61** |
|  100%  | scratch          |     70.57    |   84.50   |   57.06   |   70.14   |
|        | GCC-3D           |     71.26    |     -     |     -     |     -     |
|        | STRL             |     71.46    |   84.70   |   57.80   |   71.88   |
|        | PointContrast    |     71.55    |   84.18   |   57.74   |   72.72   |
|        | ProposalContrast |     72.92    | **84.72** |   60.36   |   73.69   |
|        | ALSO             |     72.96    |   84.68   |   60.16   |   74.04   |
|        | PatchContrast    |     72.97    |   84.67   |   59.92   | **74.33** |
|        | GPC (Ours)       |   **73.58**  |   84.68   | **62.06** |   73.99   |

STRL: Huang  et al. "Spatio-temporal self-supervised representation learning for 3d point clouds." ICCV. 2021.

ALSO: Boulch et al. "Also: Automotive lidar self-supervision by occupancy estimation." CVPR. 2023.

GCC-3D: Liang et al. "Exploring geometry-aware contrast and clustering harmonization for self-supervised 3d object detection." ICCV. 2021.

PatchContrast: Shrout et al., "PatchContrast: Self-Supervised Pre-training for 3D Object Detection." arXiv preprint arXiv:2308.06985.

Reviewer comment (Experiments):
The authors use point-level, especially for the ground category, for pretraining, but lack validation of its effectiveness in instance-level 3D object detection tasks.

Author response:
**Lack of validation in instance-level 3D object detection tasks.** 
We respectfully think there might be a misunderstanding. While we pre-train with point-level colorization, our main experimental results (cf. Table 1–9) are about instance-level 3D object detection. We will make sure we clarify this in the final version. In short, during pre-training, we leverage colors and the hint mechanism to equip the model backbone with the semantic cues that facilitate the subsequent fine-tuning for 3D object detection.

Reviewer comment (Experiments):
They employ a multimodal pretraining approach (camera + LiDAR), while the compared methods are all unimodal (LiDAR-only).

Author response:
**Multimodal pretraining approach.**
Thank you for the comment. In our original submission, we indeed compared with other multimodal pre-training in Table 3. Please also see the paragraph *Learning with Images* on page 8 of the main paper.

Here, we further provide the results of fine-tuning on 100% KITTI, using the PV-RCNN model. 
All the compared methods in the following table use multimodal pre-training, including the current SOTA: TriCC. GPC performs the best overall. 

|            |   easy   |   mod.   |   hard   |
|------------|:--------:|:--------:|:--------:|
| SLidR      |   82.9   |   71.9   |   68.0   |
| TriCC      |   84.1   |   73.3   |   69.4   |
| GPC (Ours) | **84.4** | **73.6** | **69.7** |

TriCC: Pang et al. "Unsupervised 3D Point Cloud Representation Learning by Triangle Constrained Contrast for Autonomous Driving." CVPR. 2023.

SLidR: Sautier et al. "Image-to-lidar self-supervised distillation for autonomous driving data." CVPR. 2022.

Reviewer comment (Experiments):
They did not conduct linear probing, so it's challenging to determine whether pretraining itself significantly improves performance or if enhancements were added during fine-tuning.

Author response:
**Linear probing.** 
Thanks for the comment. We note that while linear probing is usually used to evaluate pre-training on classification tasks, it may not be suitable for the downstream task of 3D object detection. For 3D object detection, the detection head added on top of the backbone contains multiple neural network layers and is trained for multiple tasks (e.g., regression and classification). In this paper, we follow the convention in this branch of study (e.g., PatchContrast, ProposalContrast, etc.) to perform full fine-tuning to compare with existing SOTA algorithms.

Reviewer comment (Reproducibility):
Many details related to the proposed method and experiments have not been provided, which could confuse the readers.

Author response:
**Details about the proposed method.**
Thank you for pointing out the problem. Due to the space limit, we keep some details in the supplementary. We will incorporate more details and discussion in the final version.

Reviewer comment (Evaluation):
The camera and point cloud perspectives are inconsistent, which means that only a portion of points can find corresponding color information. It's somewhat surprising that this pretraining can significantly boost fine-tuning performance, especially in the case of KITTI, which only has a front-facing camera. Maybe the authors could provide a detailed analysis of why using colorization for pretraining is so effective.

Author response:
**Inconsistency of camera and point cloud.**
Thanks for the question. We provided a detailed discussion regarding this question in the first response (i.e., *color assignment*) to Reviewer Hnni.

Specifically about the KITTI dataset, since the 3D object detection metric only considers objects that appear within the frontal camera's field of view (Ref. [1]), it is a convention to input only the corresponding frontal-view LiDAR points into the 3D object detector in both training and testing (Ref. [2]). 

Regarding why using colorization for pre-training is effective, we have provided several analyses (cf. Sect. 4.3, S2, and S4) and arguments (cf. Sect. 1 and 3) in the main paper and supplementary. We will be happy to conduct additional analyses in the final version. We note that pre-training with naive colorization is suboptimal (cf. Table 6 with the MSE loss and Table 8 without hints). Our key contribution is identifying the intrinsic difficulty (cf. Sect. 3.1 and 3.2) and proposing GPC as the appropriate way to use color for pre-training. 

[1] https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d

[2] https://github.com/open-mmlab/OpenPCDet/blob/master/tools/cfgs/dataset_configs/kitti_dataset.yaml#L17

Reviewer comment (Reproducibility):
The authors claim that GPC + 20% fine-tuning can surpass 100% of training from scratch. However, it seems they haven't mentioned how they selected this 20%. If it's a random 20%, then the current LaserMix approach can achieve nearly the same level of fully supervised performance.

Author response:
**Comparison to LaserMix at 20% fine-tuning.**
Thanks for providing the reference. We carefully read it (Ref. [3]) and found that the focus differs from ours. LaserMix focuses on LiDAR semantic segmentation, while we focus on 3D object detection. We could not find its performance of 3D object detection. The fine-tuning subsets used in our paper are randomly sampled. However, we provide the results of repeated experiments in Appendix S1, showing that the improvement is consistent regardless of which data are sampled for fine-tuning. Our claim about the 20% fine-tuning is just to explain the effectiveness of the proposed method and demonstrate the potential benefits of reducing labeling efforts. It is worth noting that, at the time of our paper submission, the SOTA algorithm, ProposalContrast, achieved only an overall moderate performance of 66.2% when fine-tuned on 20% labeled data (cf. Table 2). In contrast, GPC achieved 70.1%, better than 69.5% achieved by training from scratch using the entire dataset. The same trend is found in our new results using PV-RCNN as the 3D model (cf. the response *More comparisons in Table 1 and Table 2* to Reviewer Hnni): GPC performs fairly well when fine-tuning with 20% data. 

[3] Kong et al. "Lasermix for semi-supervised lidar semantic segmentation." CVPR. 2023.

Reviewer comment (Experiments):
The main weakness that I see is with the two baseline approaches, why did you focus on Point R-CNN and IA-SSD? There seem to be significantly better performing methods out there. These might not be point-based, but the paper also shows that the method transfers to voxel-based methods. So I'm mainly wondering what happens when we look at current state-of-the-art methods? Can we still gain something from colorization-based pretraining for those methods? Or is there some explicit reason why stronger methods cannot be considered?

Author response:
**Baseline approaches.** In the main paper, we aimed to present our colorization idea in a clear and accessible way, allowing readers to grasp the core concept easily. Point-based detectors (e.g., Point R-CNN and IA-SSD) were chosen for the purpose. That said, our GPC can definitely be applied to voxel-based detectors, after a voxel-to-point operation. (Such an operation has been used in LiDAR point segmentation when a voxel-based backbone is adopted.) To illustrate this, we have conducted 3D object detection experiments with PV-RCNN on Waymo -> KITTI. The results are summarized in the table below. GPC consistently outperforms the existing algorithms. We appreciate your suggestion and we will incorporate these new results into the final version.

| Labels |      Method      | Overall mod. |  Car mod. | Ped. mod. | Cyc. mod. |
|:------:|:----------------:|:------------:|:---------:|:---------:|:---------:|
|   20%  | scratch          |     66.71    |   82.52   |   53.33   |   64.28   |
|        | ProposalContrast |     68.13    | **82.65** |   55.05   |   66.68   |
|        | PatchContrast    |     70.75    |   82.63   |   57.77   | **71.84** |
|        | GPC (Ours)       |   **71.37**  | **82.65** | **59.87** |   71.59   |
|   50%  | scratch          |     69.63    |   82.68   |   57.10   |   69.12   |
|        | ProposalContrast |     71.76    |   82.92   |   59.92   |   72.45   |
|        | PatchContrast    |     72.39    |   84.47   |   60.76   |   71.94   |
|        | GPC (Ours)       |   **72.75**  | **84.58** | **61.06** | **72.61** |
|  100%  | scratch          |     70.57    |   84.50   |   57.06   |   70.14   |
|        | GCC-3D           |     71.26    |     -     |     -     |     -     |
|        | STRL             |     71.46    |   84.70   |   57.80   |   71.88   |
|        | PointContrast    |     71.55    |   84.18   |   57.74   |   72.72   |
|        | ProposalContrast |     72.92    | **84.72** |   60.36   |   73.69   |
|        | ALSO             |     72.96    |   84.68   |   60.16   |   74.04   |
|        | PatchContrast    |     72.97    |   84.67   |   59.92   | **74.33** |
|        | GPC (Ours)       |   **73.58**  |   84.68   | **62.06** |   73.99   |

STRL: Huang  et al. "Spatio-temporal self-supervised representation learning for 3d point clouds." ICCV. 2021.

ALSO: Boulch et al. "Also: Automotive lidar self-supervision by occupancy estimation." CVPR. 2023.

GCC-3D: Liang et al. "Exploring geometry-aware contrast and clustering harmonization for self-supervised 3d object detection." ICCV. 2021.

PatchContrast: Shrout et al., "PatchContrast: Self-Supervised Pre-training for 3D Object Detection." arXiv preprint arXiv:2308.06985.

Reviewer comment (Novelty):
Point2color: 3D Point Cloud Colorization Using a Conditional Generative Network and Differentiable Rendering for Airborne LiDAR by Shinohara et al. looks at point cloud colorization with a GAN setup. This could actually be a very interesting alternative to the seeding proposed here.

Author response:
**More related work about colorization.** We sincerely appreciate your valuable comments and references. We will certainly cite and discuss them, and include more papers on image colorization-based pre-training, image colorization with seeds, and general point cloud colorization. In the following, we provide a short summary and discussion of the literature. 

Image colorization has been an active research area in computer vision [1-13], predicting colorful versions of grayscale images. Typically, this task relies on human-related cues such as scribbles [9], exemplars [7], or texts [8], as well as external priors like pre-trained detectors [3], generative models [10], or superpixels [11, 18], to achieve realistic and diverse colorization. [14-17] have extended the task to 3D point clouds. Specifically, [13, 16] uses a GAN-style loss, learning a discriminator to differentiate real colorful images and colorized images and using it to train the colorization model. This provides an alternative to using hints to resolve the color variation problem. 

Instead of treating colorization as the ultimate task, [1, 12, 13] have explored colorization as a pre-training task for 2D detection and segmentation. We note that [1, 12] do not specifically address the color variation problem. 

In our paper, we introduce colorization as the pre-training task to equip the point cloud representation with semantic cues useful for the downstream 3D LiDAR-based detectors. We identify color variation as the intrinsic challenge and propose *inserting color hints into the output of the backbone* to ground the colorization taking place in the color decoder (cf. Figure 2). We note that such a design is crucial—we cannot add color hints as input to the backbone as the downstream task takes only the LiDAR points as input. In contrast, for colorization methods not for pre-training, the hints can be directly added to the input gray-level image [9]. [18] proposed a hybrid method to first predict the color of each superpixel pixel using a neural network, and propagated them to nearby pixels using a hand-crafted method (e.g., 2D-PCA). In contrast, our GPC learns to propagate colors in an end-to-end fashion by directly minimizing the colorization loss, which is the key to benefit downstream 3D detection. 

[1] Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image colorization." ECCV, 2016.

... [list continues same as W2 above] ...

Reviewer comment (Novelty):
Automated Colorization of a Grayscale Image With Seed Points Propagation by Wan et al. looks at seed-based image colorization. While different, it could be interesting to discuss such differences.

Author response:
**More related work about colorization.** We sincerely appreciate your valuable comments and references. We will certainly cite and discuss them, and include more papers on image colorization-based pre-training, image colorization with seeds, and general point cloud colorization. In the following, we provide a short summary and discussion of the literature. 

... [identical full text as W2 above] ...

Reviewer comment (Experiments):
The SL1 regression loss seems better than CE loss in quite some cases, what about a balanced SL1? Could that be better?

Author response:
**Loss ablation for pre-training.** We apologize for the confusion. Indeed, different losses require different hyperparameter tuning. Actually, we conducted comprehensive grid searches for pre-training learning rates and fine-tuning learning rates (i.e. 0.0001 to 0.01), as well as other optimization-related hyperparameters. What we reported in Table 6 of the main paper is the *best performance* of each loss. We will clarify the ablation and provide more discussion in the final version. Thank you for bringing this to our attention.

Regarding regression, it is known to be sensitive to outliers. In contrast to MSE, SL1 is more commonly used in many computer vision tasks such as colorization and regressing the bounding box locations or offsets. Your suggestion of a balanced SL1 loss sounds interesting. If we understand it correctly, you mean to balance the SL1 loss similarly to the BalancedSoftmax. Unfortunately, unlike classification where one can easily calculate the balance factors based on the number of samples per discrete class, calculating such a factor for continuous values is not trivial (e.g., some binning may be needed).

... [R3 full text continues] ...

Reviewer comment (Experiments):
How did you sample the points that get a ground truth? Is this completely random? Or do you do some more informed sampling, ensuring that at least 1 sample is present for each cluster (if available in an image)? I guess this is not super crucial, but it could potentially reduce the amount of needed points and thus boost scores further since the network has to predict more.

Author response:
**Point Sampling for hints.** Thank you for the interesting suggestion. In the main paper, the hints provided in the pre-training are completely random. We agree that, during pre-training, it may not be necessary to provide as many hints for points located on the ground, and instead, we should focus more on foreground points. One potential solution is to sample fewer points from the major color classes. For example, the number of sampled points per class is set proportional to the square root of the total number of points per class. We have implemented the idea and reported the results of fine-tuning on 5% KITTI in the table below. We found no significant difference with more balanced sampling. However, we agree that smarter sampling is indeed interesting for future research and exploration. 

|                        | Overall mod. | Car mod. | Ped. mod. | Cyc. mod. |
|------------------------|:------------:|:--------:|:---------:|:---------:|
| random sampling        |     62.7     |   65.1   |    57.4   |    65.7   |
| more balanced sampling |     62.5     |   64.9   |    56.5   |    66.2   |

Reviewer comment (Experiments):
It would be great to have an ablation study regarding the number of color bins used.

Author response:
**Ablation study regarding the number of color bins.** Thanks for the question. We conducted a study on fine-tuning with 5% KITTI (cf. top row of Table 1 in the main paper), using 32, 64, and 128 color bins in pre-training. The results are summarized in the following table. We found no significant difference in the downstream detection task with different color bins in pre-training. We hypothesize two reasons. First, our pre-training goal is to equip the backbone with the knowledge to identify which subset of points should possess the same color and be segmented together. In this sense, it could be less sensitive to the number of color bins as long as we have enough of them. (As in Fig. 3, images with 32 bins still show high contrast between objects and the background.) Second, the pre-trained backbone will still be optimized with the downstream task data. We will include the ablation in the final version.

| bin | easy | mod. | hard |
|-----|------|------|------|
| 32  | 76.0 | 62.6 | 56.8 |
| 64  | 76.8 | 62.2 | 56.1 |
| 128 | 77.7 | 62.7 | 56.7 |

Reviewer comment (Experiments):
The reweighted loss idea in Eq 5 sounds reasonable, but one concern is that it may make the pretrained model specific to a particular dataset, reducing its transferability since each dataset can have a different number of classes and class distribution. It would also be interesting to see how the pretrained model performs when it is trained on one dataset but fine-tuned on another (this experiment is not required in the rebuttal; providing insights into this is sufficient.)

Author response:
**The reweighted loss vs. transferability.** This is an interesting question. We indeed have results with pre-training on Waymo and fine-tuning on KITTI (cf. Table. 2 in the main paper). GPC consistently outperforms baselines and performs similarly to pre-training with KITTI, demonstrating the transferability. We will provide more results with PV-RCNN in the next response. 

We thank you for the insights about transferability and the distribution shifts between datasets. (This issue also exists in 2D recognition, e.g., ImageNet pre-training transferred to COCO detection.) We respectfully think the reweighting loss could mitigate the distribution shift to facilitate transferability. As discussed in the literature on class-imbalanced learning and long-tailed recognition, learning a model without re-weighting would result in a model attempting to predict the major classes in the dataset—essentially capturing the dataset-specific bias. Learning with the re-weighting loss (cf. Eq 5) would reduce this effect. When a huge distribution shift exists between datasets, one can use the quantization bins obtained from the target dataset for pre-training with the source data.

Reviewer comment (Experiments):
Please include more recent state-of-the-art works for comparison in Table 1 and Table 2. While I can gauge the effectiveness of the proposed idea through existing comparisons, a more extensive comparison with more recent works on the same benchmark is also necessary. The leaderboard of these two benchmarks has several entries showing better performance, so it would be great to discuss the advantages of the proposed method when compared to them.

Author response:
**More comparisons in Table 1 and Table 2.**
We provide more experimental results of pre-training on Waymo and fine-tuning on KITTI (cf. Table 2), using PV-RCNN as the 3D detector. (We note that many SOTA results use PV-RCNN.) As shown in the following table, GPC consistently outperforms existing algorithms. The existing SOTA algorithms primarily utilize contrastive learning (e.g., ProposalContrast and PatchContrast). They require specific designs (i.e., proposal encoding module, patch extraction module, etc.) to identify meaningful regions to contrast. GPC, in contrast, offers a fresh perspective on pre-training for 3D detectors. It is straightforward in its approach yet remarkably effective. GPC is particularly advantageous when the labeled data are scarce. Regardless of the 3D model (i.e., PointRCNN or PV-RCNN) and the pre-training dataset (i.e., Waymo or KITTI), GPC’s performance of fine-tuning on just 20% of labeled data is consistently better than training from scratch with the entire dataset (cf. Table 1 and Table 2 and the following table). This advantage holds the potential to reduce the annotation effort required. We will include the new results and more discussion in the final version. 

| Labels |      Method      | Overall mod. |  Car mod. | Ped. mod. | Cyc. mod. |
|:------:|:----------------:|:------------:|:---------:|:---------:|:---------:|
|   20%  | scratch          |     66.71    |   82.52   |   53.33   |   64.28   |
|        | ProposalContrast |     68.13    | **82.65** |   55.05   |   66.68   |
|        | PatchContrast    |     70.75    |   82.63   |   57.77   | **71.84** |
|        | GPC (Ours)       |   **71.37**  | **82.65** | **59.87** |   71.59   |
|   50%  | scratch          |     69.63    |   82.68   |   57.10   |   69.12   |
|        | ProposalContrast |     71.76    |   82.92   |   59.92   |   72.45   |
|        | PatchContrast    |     72.39    |   84.47   |   60.76   |   71.94   |
|        | GPC (Ours)       |   **72.75**  | **84.58** | **61.06** | **72.61** |
|  100%  | scratch          |     70.57    |   84.50   |   57.06   |   70.14   |
|        | GCC-3D           |     71.26    |     -     |     -     |     -     |
|        | STRL             |     71.46    |   84.70   |   57.80   |   71.88   |
|        | PointContrast    |     71.55    |   84.18   |   57.74   |   72.72   |
|        | ProposalContrast |     72.92    | **84.72** |   60.36   |   73.69   |
|        | ALSO             |     72.96    |   84.68   |   60.16   |   74.04   |
|        | PatchContrast    |     72.97    |   84.67   |   59.92   | **74.33** |
|        | GPC (Ours)       |   **73.58**  |   84.68   | **62.06** |   73.99   |

STRL: Huang  et al. "Spatio-temporal self-supervised representation learning for 3d point clouds." ICCV. 2021.

ALSO: Boulch et al. "Also: Automotive lidar self-supervision by occupancy estimation." CVPR. 2023.

GCC-3D: Liang et al. "Exploring geometry-aware contrast and clustering harmonization for self-supervised 3d object detection." ICCV. 2021.

PatchContrast: Shrout, Oren, et al. "PatchContrast: Self-Supervised Pre-training for 3D Object Detection." arXiv preprint arXiv:2308.06985. 2023.