Paper: FusionViT: Hierarchical 3D Object Detection via Lidar-Camera Vision Transformer Fusion (ICLR-2024)

Reviewer comment (Novelty):
It seems to be an exaggeration to claim that this work is "the first study to investigate the possible pure-ViT based 3D object detection framework". Transformer based architectures for 3D tasks seem to have been explored extensively in the literature, see [1] for a survey.

Author response:
As [1] Section 4.1.2 indicates, current transformer-based 3D Object Detection Frameworks are primarily restricted to DETR-based methods, that is, the frameworks with both encoder and decoder (such as 3DETR, CT3D, CAT-Det mentioned in [1]). Designing an encoder-based framework in 3D Object Detection is still a fairly unexplored area.

Reviewer comment (Novelty):
The literature survey in this work is unfortunately not effective. While it lists some of the recent and classical fields, it fails to clearly define the relevance of the current work against the prior approaches. It will help position this work better if such an explicit analysis is presented.

Author response:
We clarify that due to space limitations, we cut down several literature surveys of this work. Nevertheless, we tried to outline the connection, pros and cons among the previous related works, as well as their relevance to the current work.

Reviewer comment (Experiments):
The paper does not seem to provide much justifications on the various design choices. There are a few well-known 3D object detection paradigms in the literature, voxel based, pillar based and projection based. The paper focuses on the voxel based paradigm. It does not seem to be particularly favorable for a pure ViT architecture. Compared to pillar based approaches it is likely slower and harder to train due to the larger complexity in the attention layers. Compared to a projection based approach it is likely harder to align with the image features (being an 3D detection framework that fuses 2D and 3D). Given this large design space and obvious concerns, I think the paper should provide more rationale and ablation studies to justify the design choices.

Author response:
As for comparing with other 3D object detection paradigms, we conducted serval experiments on them. For example, we compared our model with PointPillars and FasterPillars (Pillar-based methods), CenterPoint and Part-{A^2}-free (Point-based methods). For some assumptions such as `Compared to pillar-based approaches it is likely slower and harder to train due to the larger complexity in the attention layers`, and `Compared to a projection-based approach it is likely harder to align with the image features`, my opinion is that we cannot do such general categories' comparison (we could never prove that $a-$based framework must have a good performance (faster, more accurate, etc.) than $b-$based framework, since these framework categories are quite probably. There is no mathematical formula to restrict one method belonging to a group of frameworks. In other words, these frameworks are just categories by Point Cloud input feature preprocessing ways. After all, besides feature preprocessing, it is also important to design an efficient backbone, neck, detection head, and loss function. These efficient models could exist in any kind of data preprocessing category. Therefore, it is not reasonable to do justifications on the various design choices. One could only report methods performance from different such categories.

Reviewer comment (Experiments):
While ViT architectures are shown to be very capable for classification/embedding tasks, it does have a few significant shortcomings in practice. For example, due to lack of the inductive bias it typically requires a large dataset for training to achieve competitive performance. Also, the quadratic complexity in attention layers makes it much harder to be used in dense prediction tasks. These are particularly relevant for 3D object detection using LiDAR as an input modality. It would be reasonable to expect a paper that set out to explore a "pure-ViT based 3D object detection framework" should provide deep analysis on those issues and propose effective mitigations of those shortcomings. It would also be reasonable to expect the same paper to demonstrate the superiority of ViT based architecture versus other sensible architecture choices, such as Swin based methods, despite the possible shortcomings. However, neither is presented in this work.

Author response:
While to the best of our knowledge it is the first study to investigate the possible pure-ViT-based 3D object detection framework, we accept the suggestion to conduct more exploration on ViT-related issues, such as large datasets for training and the quadratic complexity. We may consider conducting several mathematical analyses and ablation discussions and proposing more strategies to resolve these ViT common issues in future work. We could also report the FLOPs and inference time for a more comprehensive performance comparison.

Reviewer comment (Theory):
The paper is motivated by the absence of “pure-ViT” based multi-modal 3D detection models. However, the paper doesn’t explain why such a ViT-only approach is expected to be beneficial for the task.

Author response:
We clarify that, as explained in detail in the Fourth paragraph of Section 1, designing a pure-ViT framework in 3D Object Detection is still a fairly unexplored area. To fill in this gap, we conducted extensive design discussions (Section 3) and experiments (Section 4.3) on it. In the end, we concluded what we found and proposed the presented FusionViT framework.

Reviewer comment (Novelty):
The overall architecture is not quite novel in that most multi-modal fusion approaches, e.g., DeepFusion, Transfusion, DeepInteraction and any of the BEV fusion based approaches use single modality representation learning followed by multi-modal fusion. It’s unclear what the novelty of the architecture shown in Fig. 1 is.

Author response:
We clarify that while multi-modal fusion is a general approach that utilizes more than one kind of input for CV tasks (detection), its following structures that embed and process these input features could be much different and require lots of innovation. For instance, DeepFusion emphasizes its two novel techniques: InverseAug and LearnableAlign. Transfusion highlights its soft-association mechanism and image-guided query initialization module. DeepInteraction advocates its modality interaction strategy. Our framework on the other hand promotes our pure-ViT hierarchical architecture. 
We also note that while Transfusion and DeepInteraction use BEV as Lidar input, our method and DeepFusion directly use Point Cloud as the Lidar input.

Reviewer comment (Experiments):
There’s comparison missing with DeepInteraction (NeurIPS 2022) around the same time as other papers which performs camera-lidar fusion for 3D object detection.

Author response:
We agree that the ablation study could be sounder if we add more robustness experiments for MixViT (add `Without LidarViT and MixViT` and `Without LidarViT and MixViT` experiments in Table 4) and for lidar-camera spatio-temporal misalignment. Adding additional comparing experiments for DeepInteraction could also make our paper more convincing. We will add these in the revised version.

Reviewer comment (Experiments):
Performance on NuScenes is also missing in the paper.

Author response:
KITTI, Waymo Open, and NuScenes are the three most popular datasets for 3D object detection in traffic scenarios. KITTI was first proposed in 2012, it has about 100GB of size which has been well suited for academic research. NuSense and Waymo Open are two industry-level datasets that contain more than 1TB of size for each. They are almost similar in data size, tasks, popularity, and resolution. Due to limited time and devices, we chose to conduct experiments on KITTI and Waymo Open to report the performance under a broader range we could do.

Reviewer comment (Writing):
There are several typos in the paper, for example: "Swim" Transformer written in several places; Section 3.5, what is "Multi-Level Perceptions"?

Author response:
We will correct the typos to have a nicer presentation to readers. Thanks for pointing out!

Reviewer comment (Experiments):
How does the approach perform under single modality failure and spatio-temporal misalignment?

Author response:
We agree that the ablation study could be sounder if we add more robustness experiments for MixViT (add `Without LidarViT and MixViT` and `Without LidarViT and MixViT` experiments in Table 4) and for lidar-camera spatio-temporal misalignment. Adding additional comparing experiments for DeepInteraction could also make our paper more convincing. We will add these in the revised version.

Reviewer comment (Experiments):
Experiment section, besides main results on KITTI and Waymo, not sufficient.

Author response:
For the Experiment section, you mentioned it is not sufficient, could you explain it in more detail? We compared our model with several baselines. For the baseline choosing, we selected both classical and SOTA models, typically for Object Detection tasks in camera-only, lidar-only, and camera-lidar fusion input. In addition, we followed the experiment settings and reported the full evaluation metrics from both KITTI and WOD.

Reviewer comment (Evaluation):
Model runtime is key for object detection, and with these heavy transformer models, it would good to see some numbers on this.

Author response:
As for Ablations, we chose not to report `Model Runtime`, `The Loss Choices`, and other potential choices since they either do not have a great variation or not being the main research object in this experiment. `Model Runtime` is the first situation. While we will report the general `Model Runtime` in our experiment, it varies nearly negligible under different model structures.

Reviewer comment (Experiments):
The authors also introduce a "corner loss" in Section 3.5, which I would have liked to see in the ablations.

Author response:
As for Ablations, we chose not to report `Model Runtime`, `The Loss Choices`, and other potential choices since they either do not have a great variation or not being the main research object in this experiment. `The Loss Choices` is the second situation. The main contribution of this paper is to introduce a novel vision transformer-based 3D object detection model with pure-ViT-based hierarchical architecture. While it could be a future work, we focused on the model structure design, rather than the specific loss function that could match our models best.

Reviewer comment (Writing):
Writing needs improvement. The writing in Sections 3.2, 3.3 is clear enough to understand, but the mathematical notation is overloaded and actually makes it harder to understand.

Author response:
Thank you for your constructive comments and suggestions, and they are exceedingly helpful for us to improve our paper. We will improve the writing structure in Section 3 to have a nicer presentation to readers.

Reviewer comment (Presentation):
Table 1: from the writing, I assume the first row is performing 2D detection (comparing DETR, Swin, CameraViT). The second two rows are 3D detection with lidar and lidar-camera fusion, respectively. Why is the first group being compared to the second two?

Author response:
1. For the experiments, as mentioned, we selected both classical and SOTA models, typically for Object Detection tasks in camera-only, lidar-only, and camera-lidar fusion input. We put these three groups of experiments together, to show the necessity of using 3D object Detection as well as using fusion. By comparing groups `1 and 2`, we should that 3D Object Detection frameworks generally perform better than pure 2D Detection. From groups `1 and 3` or `2 and 3` we show fusion strategies generally outperform single-modal strategies.

Reviewer comment (Reproducibility):
Is there any difference in the camera/fusion architecture when dealing with single-view (KITTI) and multi-view (Waymo) camera images?

Author response:
2. We first clarify that while both KITTI and Waymo have multi-view camera images, we use just a single view for each dataset. For KITTI we used `camera2` and for Waymo we used `front`. Therefore, we do not need to encode the camera pose information.

Reviewer comment (Reproducibility):
How/is the camera pose information being used in the positional embeddings?

Author response:
2. We first clarify that while both KITTI and Waymo have multi-view camera images, we use just a single view for each dataset. For KITTI we used `camera2` and for Waymo we used `front`. Therefore, we do not need to encode the camera pose information.

Reviewer comment (Reproducibility):
Section 3.6: the authors state they train the model in 3 stages due to "some potential issues of large memory consumption", but still run the model end to end for training the MixVit? Are the subsequence layers frozen in this stage?

Author response:
3. The subsequence layers of MixVit are not frozen in this stage. For the pertaining, we pre-train CameraViT and LidarViT on the training set. In the fine-tuning, only parameters in CameraViT and LidarViT are frozen.