Paper: CALICO: Self-Supervised Camera-LiDAR Contrastive Pre-training for BEV Perception (ICLR-2024)

Reviewer comment (Reproducibility):
How does it identify the semantic-less and semantic-rich points? How does it calculate the 4th dimension of the points? What is the range of 4th dimension of the points?

Author response:
In our approach, the differentiation between semantic-rich and semantic-less points is achieved through a novel unsupervised technique termed **semantic pooling**. This process is designed to identify points potentially associated with objects of interest in an unsupervised manner, such as cars, cyclists, and pedestrians, categorizing them as *semantic-rich*. As our semantic pooling is based on the modified DBSCAN clustering algorithm with heuristics, these pooled points are extracted into different clusters based on distance metrics. The representation of these clusters can be seen in different colors in Figures 6, 7, and 8 in Appendix C. Points that are not selected by our semantic pooling, typically associated with less relevant features like buildings, ground, or grass, are marked as *semantic-less*. 

Regarding the 4th dimension of the points, it serves as a cluster identifier for the semantic-rich points. Each point is assigned an integer value in this dimension that corresponds to the cluster it belongs to, as determined by our semantic pooling operation. For semantic-less points, this value is set to -1, distinguishing them from their semantic-rich counterparts. It's important to note that the 4th dimension is used as metadata during our pretraining process and does not directly feed into the model learning.

We have recognized the need for a more detailed explanation of these concepts and have included additional clarifications in the revised version of our paper.

Reviewer comment (Reproducibility):
Is T2 identity transform in Fig. 1? If not, how does it generate the images from the original images? Were any related augmentations applied to the images, when the lidar points were changed?

Author response:
The second transformation $T_2$  does not necessarily need to be the identity transformation. We would like to clarify that Figure 1 is only for illustrative purposes. There are three transformations in the actual implementation of our framework.

In the first stage of CALICO, namely point-region contrast (PRC), we employ two transformations, $T_1$ and $T_2$, to facilitate effective contrastive pretraining. 

In the second stage of CALICO, which is the region-aware distillation (RAD), we introduce another transformation on the xy-plane. This transformation, denoted as $T_3$, is similar to $T_1$ and $T_2$ and serves as a data augmentation method for the LiDAR modality. Concurrently, the view transformation module in the camera modality, LSS [a,b], incorporates $T_3$ when transforming the features from the perspective view to the bird’s eye view (BEV). This ensures that the feature maps from both LiDAR and camera modalities remain well-aligned in the distillation process. 

We hope our response addresses all the comments and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

[a] Philion, Jonah, and Sanja Fidler. "Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16. Springer International Publishing, 2020.

[b] BEVFusion Codebase https://github.com/mit-han-lab/bevfusion/blob/main/mmdet3d/models/vtransforms/base.py (disclaimer: this link contains a public codebase that does not leak any information from the authors.)

Reviewer comment (Experiments):
The proposed method adopts the BEVFusion architecture. Lidar backbone is PointPillars, and Image backbone is Swin-T. Evaluation of different Lidar/Image backbone models could have a high impact.

Author response:
We are glad that the reviewer found our method effective and comprehensive. We appreciate the opportunity to address the points you have raised.

> The proposed method adopts the BEVFusion architecture. Lidar backbone is PointPillars, and Image backbone is Swin-T. Evaluation of different Lidar/Image backbone models could have a high impact.

We adopted the BEVFusion codebase for its state-of-the-art performance and its versatility as a general sensor fusion framework at the time of our implementation. In Section 3.5 of our paper, we have conducted an ablation study using the VoxelNet backbone and Transfusion head to specifically demonstrate the effectiveness of our point-region contrast (PRC) approach in the LiDAR modality. The results, as illustrated in Figure 4, clearly show that our PRC method consistently outperforms other baseline approaches in terms of performance.
Additionally, in response to your valuable suggestion, we have integrated ResNet as an alternative backbone for the camera modality (as configured in the BEVFusion codebase). Specifically, we use our PRC pretrained LiDAR backbone as the teacher model and leverage the 10% finetuning data setting in this experiment.
This further step is taken to demonstrate the adaptability and generalizability of our framework across different backbone architectures.

| ResNet as the Camera Backbone with 10% Finetuning Data  | mAP  | NDS  |
|---------------------------------------------------------|------|------|
| **PRC**+Rand. Init. (C)                                         | 47.9 | 52.0 |
| SimIPU                                                  | 46.6 | 51.4 |
| **PRC**+BEVDistill                                          | 48.5 | 52.5 |
| **CALICO**                                                  | **49.0** | **53.1** |

The results in the above table illustrate a consistent trend in performance between Swint-T and ResNet as the image backbone. More importantly, our CALICO using the ResNet backbone consistently achieves the best performance compared to other baseline methods.

Reviewer comment (Novelty):
The paper is entirely empirical, and is as such a purely application based work. One may or may not take this as a weakness, of course.

Author response:
We are glad that the reviewer found our study valuable to this field and that our method is effective and generalizable. We appreciate the opportunity to address the points you have raised.

> The paper is entirely empirical, and is as such a purely application based work. One may or may not take this as a weakness, of course.

We agree with the reviewer's observation regarding the empirical nature of our study. While it is true that our paper focuses on application-driven research, we believe this approach is not only valid but also crucial in the context of pretraining methods. As the reviewer notes, the empirical focus should not inherently be seen as a weakness.

In the field of pretraining, whether contrastive (e.g., MoCo [a]) or generative (e.g., MAE [b]), the practical application of methods often precedes and informs theoretical understanding. This is particularly true in rapidly advancing areas where empirical results can guide and refine theoretical models. Our work contributes to this tradition by providing valuable insights through real-world scenarios in autonomous driving perception.

To comply with the suggestions from the reviewer, we have included a new discussion in Appendix D.1 of our revised manuscript on potential avenues for theoretical exploration in pretraining methods, acknowledging the current limitations while highlighting the importance of empirical contributions like ours in laying the groundwork for future theoretical advancements.

Reviewer comment (Experiments):
On the same lines as above, the paper is heavy on tables, but I feel that qualitative analysis of where the improvement comes from is light. Some analysis through carefully designed experiments that show improvement with and without various fittings would shed insight. It appears that semantic pooling (feature rich vs feature less regions) plays a part from the figure 5, but I would like more examples of failure cases.

Author response:
We appreciate the reviewer's insightful observation regarding the balance between quantitative and qualitative analysis in our paper. We agree that the abundance of tables could potentially overshadow the qualitative insights. The choice to emphasize tabular data was driven by our intent to provide a comprehensive and detailed comparison across different metrics.

In response to the reviewer's suggestion, we have included a more focused qualitative analysis in Appendix B.1. This analysis presents key insights derived from our results. Notably, we observe that the benefits of pretraining diminish with the increase in fine-tuning data. This highlights the nuanced trade-off between pretraining and fine-tuning in model performance, which is as expected. Furthermore, when comparing our proposed PRC method (augmented with the RAPC component) to the PLRC approach, it becomes evident that our method demonstrates more significant improvements, especially when finetuning on 50% of the data. Moreover, our ablation study in Table 5 also showcases that the RAPC component in our design serves as a good regularization term to balance the performance of our framework among different levels of availabilities in finetuning data.

We hope that this enhanced qualitative discussion complements the extensive quantitative data presented and provides a more holistic understanding of our research contributions.

Reviewer comment (Experiments):
The clustering methods look rather empirical. More experiments on how they work in different cases would be useful.

Author response:
In our exploration of the unsupervised nature of our semantic pooling operation in the CALICO framework, we encountered certain limitations. For instance, the semantic pooling sometimes erroneously clusters tree points as semantic-rich. Nonetheless, these instances did not significantly impact overall performance. The primary objective during the pretraining phase is to learn robust and prominent representations within the model’s backbone, a process that remains largely independent from downstream tasks. Misclassifying a small number of random objects as semantic-rich does not detrimentally affect the contrastive pretraining for the model backbone because the objective is to learn the **correspondence** between regions. Furthermore, the core concept of semantic pooling is pivotal, transcending the specifics of its implementation. Our current implementation utilized a basic approach to achieve region partitioning in LiDAR BEV space relying on independent LiDAR frames. Enhancements to semantic pooling could include integrating temporal data to refine point aggregation or to better aggregate points and identify moving points. However, such an approach necessitates temporally continuous data, introducing an additional layer of assumption and complexity. CALICO is designed to be versatile with respect to the choice of clustering algorithms, as long as the region partitioning is reasonable, as the core of CALICO is its following contrastive learning design based on the partitioned regions. 

We have added an ablation study in Appendix B to quantitatively demonstrate the effectiveness of our semantic pooling compared to the random partitioning leveraged in original PLRC. We have included visualizations and a discussion of this in our revised manuscript, as shown in Figure 7 and 8  in Appendix C. In particular, Figure 7 visualizes a LiDAR frame with some failure clusters and Figure 8 showcases that our semantic pooling could successfully cluster some small but interesting objects.

Reviewer comment (Experiments):
Calibration error analysis: I think it would help to learn if the system can demonstrate robustness against calibration error, a common occurrence in autonomous driving setups.

Author response:
We are grateful for the reviewer's insightful suggestion to assess our system's resilience to calibration errors, a challenge in autonomous driving systems. Initially, it's worth clarifying that the primary objective of our framework is to enhance the efficiency and overall performance of self-driving perception tasks, with the robustness analysis serving as a supplementary aspect.

Our current evaluation framework operates under the premise of accurate alignment between LiDAR and camera data. This alignment is pivotal for the efficacy of our pretraining methodology, particularly due to the necessity of data synchronization for successful knowledge distillation in RAD. In scenarios where we intentionally alter the calibration matrix between the two modalities, our method demonstrates similar performance to other baseline approaches.

On one hand, we acknowledge that in real-world scenarios, misalignments due to calibration errors can occur, potentially impacting the performance of systems like ours. Although our current study did not specifically focus on this aspect, we recognize its importance in the broader context of autonomous driving. On the other hand, evaluating calibration errors is challenging, as misalignments result in differing localization ground truths for both modalities (i.e., the ground truth bounding boxes also do not align with each other). Therefore, determining a method and metric to quantify the evaluation is controversial.

In response to the reviewer's valuable feedback, we have incorporated the above discussion in our revised manuscript about the potential implications of calibration errors on systems pre-trained using CALICO. This addition not only acknowledges the issue but also underlines it as a vital area for future research, aiming to further refine the robustness and reliability of autonomous driving technologies.

Reviewer comment (Presentation):
Superfluous lines from possible prior submission (Appendix C, under 'Visualization') — "As we mentioned in the rebuttal,camera features trained from scratch are not salient to contribute to robustness improvement."

Author response:
We thank the reviewer for pointing out the superfluous lines in our manuscript and we have removed them in our revised manuscript.