From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: This paper receives 2x marginally above the acceptance threshold, 1x accept, good paper, and 1x marginally below the acceptance threshold. 

The three reviewers who give accept think that the paper is well written, the bidirectional fusion module is novel. It enables joint optimization for both image and lidar backbones using multiple training tasks.The overall dual backbone representation is also novel and provides a natural way of feature fusion. The experimental results are strong. Detailed ablation studies show the effectiveness of each design choice. The authors get top-ranking results on the KITTI leaderboard for the cycle class. 

Some concerns are results are shown on KITTI dataset which was built a long time ago, more recent datasets such as nuScenes/Waymo should be used.  Some of the claims about related work might be inappropriate and could be improved. The authors showed further results on the Waymo dataset, and have updated their paper on the related works. 

The metareviewer carefully considers the comments of the reviewer who gives marginally below the acceptance threshold. The main concerns are: It is not fair enough to compare a model with only detection ground-truth and a model with multiple surpervision. Have authors tried achieve camera to lidar propagation with BEV fashion? Missing comparison experiment. The metareviewer thinks that these doubts are well answered by the authors with additional experimental results and clarifications in their response. 
justification_for_why_not_higher_score: This paper has a 1x marginally below the acceptance threshold. 
justification_for_why_not_lower_score: This paper receives 2x marginally above the acceptance threshold and 1x accept, good paper.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper1174/Authors
title: Any more questions? Let us discuss them.
comment: Dear Reviewers,   
     
Thanks for your time and efforts in reviewing our ICLR submission, as well as the valuable comments.  We hope that our responses and the revised manuscript have well addressed your concerns.  Do you have any more questions and comments?  We are glad to have further discussions with you.

Best Regards,

From: ICLR.cc/2023/Conference/Paper1174/Authors
title: Summary of Responses
comment: We sincerely thank the reviewers for their detailed and insightful comments, as well as the favorable recommendations. We also thank the area chair for your time and efforts in handling our paper. We are glad that the reviewers found that
1. Our idea is simple, effective, reasonable, and novel.
2. The proposed 2D normalized local coordinate estimation task for improving 3D backbone is intuitive.
3. Strong experimental results/solid performance demonstrates the advantage of the proposed method.
4. The manuscript is well-written and easy to read.
5. Detailed ablation studies show the effectiveness of each element in the network design.

Following the valuable comments and suggestions, we carefully revised the manuscript  (we highlighted the major modifications in blue). Specifically, 
1. We have added extra experiments on the large-scale Waymo dataset in Appendix A.4 (suggested by Reviewers N1xe, 2AJq, and vG37).
2. We have added extra experiment results in Table 8 to demonstrate the effectiveness of our method when the 2D and 3D backbones are asymmetric (suggested by Reviewer N1xe).
3. We split the original Table 11 into Table 13 and Table 14, and supplemented the official definition of occlusion level in the caption (suggested by Reviewer N1xe).
4. We have improved statements about the limitation of pseudo-LiDAR-based methods to avoid misleading readers ( suggested by Reviewer 2AJq).
5. We have corrected the notation of $q_{p_i}^{NLC}$ and $ \hat{q}_{p_i}^{NLC} $ in the explanation of Equation 10 (suggested by Reviewer N1xe).
6. As pointed out by Reviewer 2AJq, the camera-based methods have made extraordinary progress recently. We have corrected some outdated claims and incorporated several latest works.
7. We have added discussion about recent works including TransFusion [1] and BEVFusion [2] on cross-modal 3D object detection in related works.
8. We have supplemented more details to explain the difference between 3D center estimation from 3D bounding box estimation to avoid misunderstanding.

Again, thank you all of you for your time and efforts to our paper. We also refer the reviewers to the detailed responses to each comment. We look forward to discussing with you over the next few days.

[1] Bai et al., Transfusion: Robust lidar-camera fusion for 3d object detection with transformers, CVPR'22.\
[2] Liang et al., BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework, NeurIPS'22.

From: ICLR.cc/2023/Conference/Paper1174/Authors
title: Responses to Reviewer N1xe (2/2)
comment: **Comment**: *Table 1 should include a comparison with Wang et. al. 2021, since that is the closest work to this.*

**Response:** Thanks for your valuable suggestion. As the performance of PointAugmenting [2] on the KITTI test set was not reported in the paper,  we further evaluated our method on the Waymo Open dataset to perform comparisons with PointAugmenting. As the Waymo dataset **does not** provide the ground-truth 2D segmentation maps, we **removed** the 2D segmentation auxiliary task of our framework during training. The results are listed in the following table, where "-" denotes that results were not reported in the paper of PointAugmenting, and it can be seen that our method even without the 2D segmentation auxiliary task achieves substantial improvements over PointAugmenting under all scenarios except "Ped_L2", demonstrating the advantage of our framework.

|   Method     | Vec_L1 @mAP | Vec_L1 @mAPH | Vec_L1 @mAP | Vec_L1 @mAPH | Ped_L1 @mAP | Ped_L1 @mAPH | Ped_L2 @mAP | Ped_L2 @mAPH | Cyc_L1 @mAP | Cyc_L1 @mAPH | Cyc_L2 @mAP | Cyc_L2 @mAPH |
| :-------------- | :---:| :----:| :---:| :----:| :---:| :----:| :---:| :----:| :---:| :----:| :---:| :----:|
| PointAugmenting | 67.41      | -            | 62.7       | -            | 75.42      | -            | 70.55      | -            | 76.29      | -            | 74.41      | -            |
| Ours            | 78.36      | 77.91       | 69.45      | 69.04       | 76.32      | 71.67       | 65.93      | 61.81       | 79.64      | 78.55       | 76.36      | 75.26       |
| Improvement | +10.95      | -            | +6.75       | -            | +0.9      | -            | -4.62      | -            | +3.35      | -            | +1.95      | -            |


**Comment:**  *Table 11 is a bit hard to read. Are you combining experiments with changing distance and changing occlusion? How do you define levels of occlusion?*

**Response:** Sorry for the confusion caused. The two groups of results in the original Table 11 of the manuscript refer to the performance breakdown over different occlusion levels and distances. In the revised version, we have split the original Table 11 into **Table 13** and **Table 14**. According to the definition on the KITTI official website, occlusion levels 0, 1, and 2 correspond to fully visible samples, partly occluded samples, and samples difficult to see, respectively. 

**Comment:**  *Presumably the NLC map estimation aux task is only done with cars and bikes and not pedestrians, but this is not mentioned anywhere.*

**Response:** The NLC map estimation task is conducted for all three classes.

**Comment:** *In the abstract: “..we further construct and interactive bidirectional…” What do you mean by interactive?*

**Response:** We use it to express "deep interaction between image features and point cloud features". In the revised version, we removed "interactive" to avoid confusion.

**Comment:** *Figure 6 caption: “…multi layer perception.” Should be corrected to multi layer perception. -Equation 10: q_pi and qcap_pi in equation 10 are referred to as pi and pi_cap in the explanation. Make this consistent.*

**Response:** Thank you again for your detailed thorough review. We have corrected the pointed typos in the revised version.

[1] Chen et al., Sasa: Semantics-augmented set abstraction for point-based 3d object detection, AAAI'22.\
[2] Wang et al., Pointaugmenting: Cross-modal augmentation for 3d object detection, CVPR'21.\
[3] He et al., Structure aware single-stage 3d object detection from point cloud, CVPR'20.

From: ICLR.cc/2023/Conference/Paper1174/Authors
title: Responses to Reviewer N1xe (1/2)
comment: We thank the reviewer for your detailed and thoughtful reviews. We also appreciate your favorable recommendation and recognition of our work. In what follows, we will address your concerns clearly and comprehensively.


**Comment:** *The symmetry in the two backbones - which is required to fuse features from one into the other might be a bit artificial and a bit of a drawback as efficient and State of the Art LIDAR pipelines look a bit different to image pipelines.*

**Response:** Indeed, the symmetric structures adopted in our framework (as depicted in Fig. 6 of the paper) can be somewhat artificial, since the encoder-decoder network in the LiDAR branch is similar to the image backbone. **Nevertheless**, it is worth clarifying that our approach **is not limited to** symmetrical structures and can be generalized to different point-based backbones. Here, we **replaced** the 3D branch of the original framework with an efficient single-stage detector—SASA [1], using a backbone **only** with the encoder in the LiDAR branch, which is asymmetric with the encoder-decoder structure of the image backbone. Accordingly, the  proposed bidirectional propagation is only performed  between the 3D backbone and the encoder of the 2D image backbone. The experimental results are shown in the following table, where it can be seen that the proposed method still works well even when the two backbones are asymmetric. Thanks for pointing out this issue, and we have added these results to the appendix of the revised version.

| Method  | Car@Easy | Car@Mod | Car@Hard | Ped.@Easy | Ped.@Mod | Ped.@Hard | Cyc.@Easy | Cyc.@Mod | Cyc.@Hard | mAP   |
| :---: | :-: | :-: | :-: | :--: | :-: | :--: | :--: | :-: | :--: | ----- |
| SASA [1]      | **92.17**    | 84.90   | 82.57    | 66.75     | 61.40    | 56.00     | 89.91     | 74.05    | 69.41     | 75.24 |
|  Ours (SASA) | 92.11    | **85.67**   | **82.99**    | **70.52**     | **63.38**    | **58.16**     | **91.58**     | **74.81**    | **70.22**     | **76.61** |


**Comment:** *How does one account for / deal with the fact that the same image feature (or same neighboring features) might get attached to a multiple LIDAR points (features) as points at different distances along a ray might all be projected to the same image point? There is nothing to deal with this, and this might actually be reducing the expressiveness of the LIDAR features. Maybe a BEV representation (used in Wang et al. 2021) is a better choice of representation?*

**Response:** In fact, according to the sensing mechanism of LiDAR sensors, there can **only exists a single** 3D point along a ray. Thus, the concern "*...as points at different distances along a ray might all be projected to the same image point...*" posed by the reviewer is **invalid**. Here, we assume that the reviewer is actually referring to the phenomenon that "points along **different** rays might correspond to the same image pixel scope", which does exist in real scenarios. **However**, we need to remind that, when projecting 3D points onto the 2D image plane, **there is no way that different 3D points correspond to exactly the same continuous 2D coordinate**. More importantly, note that in our implementation we adopted **bilinear interpolation** (as introduced in the paper) to propagate features from neighboring pixels to the target point. Therefore, even if different points fall into the same pixel grid scope, they will retrieve different pixel feature interpolation results because of their different **continuous** distances to the neighboring pixels.


**Comment:** *LIDAR head auxiliary task #2: 3D center estimation: how is this an auxiliary task? Doesn’t the 3D bounding box estimation already do this (3D center estimation)?*

**Response:** As we stated in the third line of Section 3.2 of the manuscript, we just follow SA-SSD [3] to introduce center estimation to learn structure-aware point-wise features, which is not claimed as our contribution. Due to space limitation, we did not explain the definition of 3D center estimation in the manuscript. In the appendix of the revised version, we added more details about this auxiliary task to make the paper self-contained.

The RPN stage of the pipeline obtains the proposals based on the *subsampled points* after the encoder by estimating the offsets of pre-defined anchors relative to the ground-truth boxes. While the 3D center estimation task aims to learn the relative position of *each foreground point* to the object center, i.e., object structure-aware features. Thus, the 3D center estimation differs from the 3D bounding box estimation.

From: ICLR.cc/2023/Conference/Paper1174/Authors
title: Responses to Reviewer 9Ksn
comment: We thank the reviewer for their time and efforts in reviewing our paper. We also appreciate your favorable recommendation and recognition of our work. In the following, we will address your concerns.

**Comment**: *It would be great if the author could also benchmark the proposed approach on other datasets, e.g. NuScenes to show the generalization ability of the proposed approach.*

**Response:** Thanks for your valuable suggestion. In the original manuscript, we did not conduct experiments on the two popular large-scale datasets  Waymo Open and nuScene because these two datasets **do not** provide the ground-truth 2D segmentation maps. Due to the limited time, during the revision period, we further conducted experiments on the Waymo Open dataset by **removing** the 2D segmentation auxiliary task of our framework. The results are listed in the following table, where we compared our method with state-of-the-art methods Transfusion (CVPR'22) [1] and PointAugmenting (CVPR'21) [2] and  "-" denotes that results were not reported in their papers. 
Unlike  TransFusion [1] and PointAugmenting [2] that freeze the camera backbone  pre-trained on other datasets, we train the entire pipeline in an end-to-end manner.  From the following table,
It can be seen that our method still achieves higher accuracy under **all scenarios** except ”Ped L2” **even without** the 2D segmentation auxiliary task, which can effectively demonstrate its advantage. We have included these results in the revised version.

|   Method     | Vec_L1 @mAP | Vec_L1 @mAPH | Vec_L1 @mAP | Vec_L1 @mAPH | Ped_L1 @mAP | Ped_L1 @mAPH | Ped_L2 @mAP | Ped_L2 @mAPH | Cyc_L1 @mAP | Cyc_L1 @mAPH | Cyc_L2 @mAP | Cyc_L2 @mAPH |
| :--------| :----:| :-----:| :----:| :-----:| :----:| :-----:| :----:| :-----:| :----:| :-----:| :----:| :-----:|
| PointAugmenting | 67.41      | -            | 62.7       | -            | 75.42      | -            | 70.55      | -            | 76.29      | -            | 74.41      | -            |
| TransFusion     | -           | -            | -           | 65.14       | -           | -            | -           | 64.00       | -           | -            | -           | 67.40        |
| Ours            | 78.36      | 77.91       | 69.45      | 69.04       | 76.32      | 71.67       | 65.93      | 61.81       | 79.64      | 78.55       | 76.36      | 75.26       |


[1] Bai et al., Transfusion: Robust lidar-camera fusion for 3d object detection with transformers, CVPR'22.\
[2] Wang et al., Pointaugmenting: Cross-modal augmentation for 3d object detection, CVPR'21.

From: ICLR.cc/2023/Conference/Paper1174/Authors
title: Responses to Reviewer 2AJq (2/2)
comment: **Comment**: *I think there is still some prior work conducting 3D detection with the point-to-pixel fusion between LiDAR and camera. I believe that I have seen a few in the past, but at this moment only remember one (EPNet++: Cascade Bidirectional
Fusion for Multi-Modal 3D Object Detection). It would be nice if the authors can add some comments to summarize the differences/similarity between these works.*

**Response:** Thanks for pointing out this related paper. We revised Section 2 of our manuscript by including more recent related works. It is also worth mentioning that the authors were **indeed** not aware of EPNet++ [4] in **Arxiv**  when conducting this project. Here we summarize the differences between our work and EPNet++ [4] in three aspects:

- EPNet++ proposes an LI-Fusion layer to enable more interaction between the two modalities and finally obtains more comprehensive features. In addition to enhancing 3D point cloud features by appending 2D image features (i.e., pixel-to-point fusion), we explore point-to-pixel propagation from the perspective of **directly improving the representation capability of the 3D LiDAR backbone network**. Specifically, by designing feasible 2D auxiliary tasks, the **gradients** backpropagated from the training objectives of the image branch can boost the representation ability of the point cloud backbone. We experimentally and exactly demonstrated our claim in **Table 2** of our manuscript. To the best of our knowledge, it has never been identified previously that 2D auxiliary tasks can be used to improve 3D backbone.
- We also verify that the bidirectional propagation benefits not only the 3D object detection task but also the 2D semantic segmentation task (see **Table 12** of the manuscript). The decent results show the potential of 2D-3D joint learning between 3D object detection and more 2D scene understanding tasks.
- We design a **concise yet very effective** bidirectional propagation strategy without bells and whistles, which achieves significantly better performance than EPNet++ [4], according to its reported results on the KITTI test set. The source code of EPNet++ is not publicly available now. 

Again, we thank reviewer 2AJq for  providing helpful and insightful suggestions. We hope that the above responses adequately address your concerns.

[1] Bai et al., Transfusion: Robust lidar-camera fusion for 3d object detection with transformers, CVPR'22.\
[2] Wang et al., Pointaugmenting: Cross-modal augmentation for 3d object detection, CVPR'21.\
[3] Qian et al., End-to-End Pseudo-LiDAR for Image-Based 3D Object Detection, CVPR'20.\
[4] Liu et al., EPNet++: Cascade Bi-directional Fusion for Multi-Modal 3D Object Detection, Arxiv'21.

From: ICLR.cc/2023/Conference/Paper1174/Authors
title: Responses to Reviewer 2AJq (1/2)
comment: We thank the reviewer for the detailed review and insightful comments. We also appreciate your favorable recommendation and recognition of our work. In what follows, we will address your remaining minor concerns comprehensively.

**Comment:** *It would make the paper’s results significantly more trustable if the results can be demonstrated on these more recent large datasets.*

**Response:** Thanks for the valuable suggestion, as well as your understanding of the missing results on large-scale datasets. In the original manuscript, we did not conduct experiments on the popular large-scale datasets Waymo Open and nuScene partly because these two datasets **do not** provide the ground-truth 2D segmentation maps. During the revision period, we further conducted experiments on the Waymo Open dataset by **removing** the 2D segmentation auxiliary task of our framework. The results are listed in the following table, where we compared our method with state-of-the-art methods Transfusion (CVPR'22) [1] and PointAugmenting (CVPR'21) [2], and  "-" denotes that the corresponding results were not reported in their papers.
It can be seen that our method still achieves **higher** accuracy under all scenarios except "Ped_L2" **even without** the 2D segmentation auxiliary task, demonstrating the advantage of our method. We have included this experiment in the revised manuscript.


|   Method     | Vec_L1 @mAP | Vec_L1 @mAPH | Vec_L1 @mAP | Vec_L1 @mAPH | Ped_L1 @mAP | Ped_L1 @mAPH | Ped_L2 @mAP | Ped_L2 @mAPH | Cyc_L1 @mAP | Cyc_L1 @mAPH | Cyc_L2 @mAP | Cyc_L2 @mAPH |
| :-------------- | :---------- | :----------- | :---------- | :----------- | :---------- | :----------- | :---------- | :----------- | :---------- | :----------- | :---------- | :----------- |
| PointAugmenting | 67.41      | -            | 62.7       | -            | 75.42      | -            | 70.55      | -            | 76.29      | -            | 74.41      | -            |
| TransFusion     | -           | -            | -           | 65.14       | -           | -            | -           | 64.00       | -           | -            | -           | 67.40        |
| Ours            | 78.36      | 77.91       | 69.45      | 69.04       | 76.32      | 71.67       | 65.93      | 61.81       | 79.64      | 78.55       | 76.36      | 75.26       |



**Comment:** *Some of the claims about related work might be inappropriate and could be improved. Looking at the new SOTA methods on the nuScenes/Waymo leaderboard, there are many successful image-only methods (e.g., BEVStereo, BEVdepth, BEVFormer) that achieve strong performance, though not beating lidar-based methods but are getting closer.*

**Response:** Thanks for your valuable comments. We agree with the reviewer that image-based 3D object detectors have achieved significant improvements in recent years and play important roles in practical applications. In the revised manuscript, we have updated the descriptions about this research line for 3D object detection. 

**Comment**: *The statement about the pseudo-lidar-based method—"it is non-trivial to optimize the depth estimator and LiDAR detection simultaneously in pseudo-lidar based methods”，might be problematic since there are already works in that direction (e.g., one of the first work, End2end pseudo-lidar, CVPR 2020).*

**Response:** It is claimed in End2end pseudo-lidar [3] that it is non-trivial to backward the gradient signals of the detection loss to the depth map. Even so, we refine the statement in the revision to avoid misleading readers.

**Comment:** *It seems that adding 3D segmentation auxiliary task seems not to bring any improvement to the detection task. Are these two tasks really beneficial to each other in this multi-task learning setting?*

**Response:** It seems that the 3D segmentation auxiliary task does not make an effect when comparing **Table 3 (a)** and **Table 3 (b)**. **Nevertheless**, when comparing **Table 3 (c)** and **Table 3 (d)** of the manuscript, it can be seen that the 3D segmentation auxiliary task brings **0.48%** mAP improvement, and especially the mAP improvement on class "Pedestrian" achieves **1.17%**. In other words, the 3D segmentation task may play a more critical role when used together with other tasks. Thus, we think the 3D segmentation head should be reserved.
Empirically speaking, the learned semantic-aware features can help foreground points gain awareness of what category they belong to. And since foreground points are correctly classified, the structure information of objects can be clearer. For the inference speed, the auxiliary tasks **do not** matter because only backbones are used for 3D object detection and the head of 3D semantic segmentation will be detached during inference.

From: ICLR.cc/2023/Conference/Paper1174/Authors
title: Responses to Reviewer vG37 (3/3)
comment: We further compared our method with cross-modal fusion in the BEV space. We compared with ContFuse [7] on the KITTI val set, and PointAugmenting [2] on the Waymo dataset, listed in the following two tables.  In general, although recent works have demonstrated the effectiveness and advantage of cross-modal fusion on BEV space, the decent results of our method could also show the promise of point-level fusion to some extent. 

| Method | Easy@3D  | Mod@3D | Hard@3D  | Easy@BEV | Mod@BEV | Hard@BEV |
| -------- | :-: | :-: | :-: | :-: | :-: | :-: |
| ContFuse [7] | 83.68 | 68.78    | 61.67 | **94.07** | 85.35    | 75.88 |
| Ours     | **89.13** | **82.97**    | **80.05** | 92.96 | **91.36**    | **86.80**  |


|   Method     | Vec_L1 @mAP | Vec_L1 @mAPH | Vec_L1 @mAP | Vec_L1 @mAPH | Ped_L1 @mAP | Ped_L1 @mAPH | Ped_L2 @mAP | Ped_L2 @mAPH | Cyc_L1 @mAP | Cyc_L1 @mAPH | Cyc_L2 @mAP | Cyc_L2 @mAPH |
| :-------------- | :--:| :---:| :--:| :---:| :--:| :---:| :--:| :---:| :--:| :---:| :--:| :---:|
| PointAugmenting [2]| 67.41      | -            | 62.7       | -            | 75.42      | -            | 70.55      | -            | 76.29      | -            | 74.41      | -            |
| Ours            | 78.36      | 77.91       | 69.45      | 69.04       | 76.32      | 71.67       | 65.93      | 61.81       | 79.64      | 78.55       | 76.36      | 75.26       |
| Improvement | +10.95      | -            | +6.75       | -            | +0.9      | -            | -4.62      | -            | +3.35      | -            | +1.95      | -            |




**Comment:** *Missing comparison experiment. Authors should discuss these recent works (Transfusion, BEVFusion e.g.) and conduct experiment on nuScenes dataset to further verify the effectiveness of proposed method.*

**Response:** Thanks for your valuable suggestion. In Section 2 of the revised manuscript, we discussed more recent works of cross-modal 3D object detection, including the mentioned Transfusion [11] and BEVFsuion [9]. We also want to point out that the mentioned paper BEVFusion is **contemporaneous** with our work as it was published in **NeurIPS 2022**.

In the original manuscript, we did not conduct experiments on the Waymo Open and nuScene datasets  because these two datasets **do not** provide the ground-truth 2D segmentation maps. During the revision period, we further conducted experiments on the Waymo Open dataset by **removing** the 2D segmentation auxiliary task of our framework. The results are listed in the following table, where we compared our method with Transfusion [11] (CVPR'22) and PointAugmenting [2] (CVPR'21) and  "-" denotes that results are not reported in their papers. As BEVFusion [9] was evaluated only on the nuScenes dataset, we did not compare with it here.  It can be seen that our method also achieves **higher** accuracy under all scenarios except "Ped_L2", demonstrating the advantage of our method convincingly.

|   Method     | Vec_L1 @mAP | Vec_L1 @mAPH | Vec_L1 @mAP | Vec_L1 @mAPH | Ped_L1 @mAP | Ped_L1 @mAPH | Ped_L2 @mAP | Ped_L2 @mAPH | Cyc_L1 @mAP | Cyc_L1 @mAPH | Cyc_L2 @mAP | Cyc_L2 @mAPH |
| :-------------- | :---:| :----:| :---:| :----:| :---:| :----:| :---:| :----:| :---:| :----:| :---:| :----:|
| PointAugmenting | 67.41      | -            | 62.7       | -            | 75.42      | -            | 70.55      | -            | 76.29      | -            | 74.41      | -            |
| TransFusion     | -           | -            | -           | 65.14       | -           | -            | -           | 64.00       | -           | -            | -           | 67.40        |
| Ours            | 78.36      | 77.91       | 69.45      | 69.04       | 76.32      | 71.67       | 65.93      | 61.81       | 79.64      | 78.55       | 76.36      | 75.26       |



[1] Vora et al., Pointpainting: Sequential fusion for 3d object detection, CVPR'20.\
[2] Wang et al., Pointaugmenting: Cross-modal augmentation for 3d object detection, CVPR'21.\
[3] Wang et al., Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving, CVPR'19.\
[4] Wu et al., Sparse fuse dense: Towards high quality 3d detection with depth completion, CVPR'22.\
[5] Shi et al., Pointrcnn: 3d object proposal generation and detection from point cloud, CVPR'19.\
[6] Yin et al., Center-based 3d object detection and tracking, CVPR'21.\
[7] Liang et al., Deep continuous fusion for multisensor 3d object detection, ECCV'18.\
[8] Philion et al., Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D,  ECCV'20.\
[9] Liang et al., BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework, NeurIPS'22.\
[10] Zhang et al., Cat-det: Contrastively augmented transformer for multimodal 3d object detection, CVPR'22.\
[11] Bai et al., Transfusion: Robust lidar-camera fusion for 3d object detection with transformers, CVPR'22.\

From: ICLR.cc/2023/Conference/Paper1174/Authors
title: Responses to Reviewer vG37  (2/3) 
comment: **Comment:** *It is important to verify the effectiveness of auxiliary tasks on other cross-modal methods. Thus, an important missing experiment is to add 2D seg and NLC losses to other cross-modal models.*
 
**Response:** First, we want to point out that the auxiliary tasks may not work on all cross-modal methods shown in Fig. 1 of the manuscript. Specifically, the auxiliary tasks do not aim to enhance the 2D detectors for the result-level fusion; the proposal-level fusion method may benefit less from the auxiliary tasks since it does not fully use the fine-grained correspondence between points and pixels. We have to emphasize that the proposed NLC map estimation is exactly suitable for the **point-based** fusion strategy since we can obtain the point-wise local spatial-aware information based on the accurate location of points and point-pixel correspondence. Besides, as the crucial fine-grained location information of points is inevitably lost in voxel/BEV-based methods, the proposed auxiliary task may also not be suitable for the pipeline of BEV-based fusion.

Based on the above analysis, we validated the generality of the  proposed NLC map estimation on a point-based cross-modal method. Specifically, following PointPainting [1], we decorated the input point clouds of PointRCNN [5] with image features extracted from an image-based network trained with 2D auxiliary tasks. The experimental results on KITTI val set are listed in the following table, where row (a) denotes the single-modal baseline PointRCNN [5], row (b)  the point clouds are decorated with image  features extracted from an image-based network only trained with the 2D segmentation task, and row (c) the point clouds are decorated with image features extracted from the image-based network trained with both 2D segmentation and NLC map estimation. By comparing Exp. (c) and Exp. (b), we can observe our proposed NLC map estimation brings substantial improvement (+1.21% mAP)  demonstrating its generality on point-level fusion-driven cross-modal methods. 

| Exp. | PointRCNN | 2D Seg | 2D NLC | Car   | Ped.  | Cyc.  | mAP      |
| :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |
| (a)  | &#10004;  |        |        | 83.30  | 55.32 | 74.91 | 71.17 |
| (b)  | &#10004;  | &#10004; |        | 83.55 | 56.77 | 75.76 | 72.03    |
| (c)  | &#10004;  | &#10004; | &#10004; | 84.16 | 57.93 | 77.64 | 73.24 |



**Comment:** *Could the camera-to-lidar propagation be extended to BEV-based methods?* 

**Response:** First, we want to point out that We **do not** claim the camera-to-lidar (pixel-to-point) propagation as our main contribution, as it has been well explored in previous works. In general, the cross-modal fusion on BEV space can be roughly divided into three categories:

- ContFuse [7] proposes the continuous fusion layer to retrieve corresponding image features of the nearest 3D points for each grid in BEV feature map. 
- PointAugmenting [2] decorates point clouds with middle CNN features extracted from 2D detection models before voxelization, and adopts separate additional 3D sparse convolution streams for camera features and LiDAR features before merging them.
- LSS [8] and BEVFusion [9] unify camera and LiDAR features in a shared BEV space by predicting the discrete depth distribution of each pixel and rescaling the associated features along the ray by their corresponding depth probabilities.

We have to emphasize that both the two kinds of propagation manners, i.e., point-to-pixel and pixel-to-point, involved in our bidirectional propagation depend on **point-pixel correspondence**. While the point-based backbone retains fine-grained location of points, there is inevitable location information loss in voxel/BEV-based methods. Empirically, the center of a highly occluded voxel may correspond to different image features than the points in that voxel. Based on the coarse-grained correspondence between voxel and pixels, we may retrieve inappropriate image features and obtain fewer performance gains with auxiliary tasks. Therefore, we think it is **inappropriate** to replace the methods especially designed for BEV-fashion with our method for BEV-level fusion.

From: ICLR.cc/2023/Conference/Paper1174/Authors
title: Responses to Reviewer vG37  (1/3) 
comment: We thank the reviewer for your time and efforts in reviewing our paper, as well as the recognition of the merits of our work. In the following, we will comprehensively address your concerns.

**Comment:** *Based on Table 3, the main performance gain of image branch comes from the auxiliary tasks*.

**Response:** Note both **Table 2 (e)**  and **Table 3 (g)** of the manuscript refer to **our full model**. We want to emphasize that, compared to **Table 3 (e)** of the manuscript (76.38% mAP), the improvement of **Table 3 (g)** (78.64% mAP) comes from **not only** the auxiliary tasks **but also** the proposed bidirectional propagation. Such a claim is validated by the result shown in **Table 2 (d)** of the manuscript, where instead of bidirectional propagation, we adopted the commonly used pixel-to-point fusion and kept all the other settings the same as Table 3 (g) (or Table 2 (e)), leading to 77.47% mAP.  In other words, the proposed bidirectional propagation significantly helps to realize the potential of the 2D auxiliary tasks.


**Comment:** *It is not fair enough to compare a model with only detection ground-truth and a model with multiple surpervision.*

**Response:**
- First, it is worth noting that the supervisions for the auxiliary tasks 2D NLC map estimation, 3D center estimation, and 3D semantic segmentation are **free-of-charge**, i.e., they are derived **only** from the ground-truth bounding boxes, and **NO** other kinds of ground-truth information is needed. Only the 2D semantic segmentation involves extra ground-truth. Here, as demonstrated in the following table, even **without** the 2D semantic segmentation task, our method still achieves **higher** accuracy than the state-of-the-art method CAT-Det [10] under **almost all**  scenarios on the KITTI val set. 

|  Method    | Car@Easy | Car@Mod | Car@Hard | Ped.@Easy | Ped.@Mod | Ped.@Hard | Cyc.@Easy | Cyc.@Mod | Cyc.@Hard | mAP    |
|--|:--:|:-:|:--:|:---:|:--:|:---:|:---:|:--:|:---:|:-:|
| CAT-Det          | **90.12** | 81.46 | 79.15  | **74.08**   | 66.35    | 58.92     | 87.64     | 72.82    | 68.2      |75.42  |
| Ours (w/o 2D Seg) | 89.39  | **84.97** | **79.16**  | 70.16   | **66.51**  | **61.12**    | **95.09**   | **76.24**  | **74.51**   | **77.47**  |


- Second, we also want to point out that it is **common** to use extra ground-truth in cross-modal 3D object detection. For example, Pointpainting [1] uses the semantic segmentation model trained with extra datasets. PointAugmenting [2] trains the 2D detection model on extra ground-truth of object detection. Pseudo-LiDAR [3] and SFD [4] use the monocular depth estimator trained with additional ground-truth. 


- Last but not least, we want to point out that introducing 2D semantic segmentation in the pipeline not only improves the performance of 3D detection but also demonstrates the potential of our bidirectional propagation for joint training of 3D object detection and more 2D scene understanding tasks because it also improves the performance of 2D semantic segmentation, as demonstrated in **Table 12** of the manuscript.

From: ICLR.cc/2023/Conference/Paper1174/Reviewer_N1xe
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes fusion of image and point cloud features for 3D object detection. In contrast to previous work (like Point-augmenting; Wang et. al. 2021), features are not just borrowed from the image plane to enrich the point features, but also vice-versa. Additional auxiliary tasks - NLC-map estimation for the image backbone and 3D semantic segmentation for the LIDAR backbone improve the performance of object detection. Detailed ablation studies show the effect of each element in the network design.
strength_and_weaknesses: Strengths:
The paper is well written, and is a simple idea that works well in practice. Detailed ablation studies show the effectiveness of each design choice. The authors get top-ranking results on the KITTI leaderboard for the cycle class.

Weaknesses:
-The symmetry in the two backbones - which is required to fuse features from one into the other might be a bit artificial and a bit of a drawback as efficient and State of the Art LIDAR pipelines look a bit different to image pipelines.
-How does one account for / deal with the fact that the same image feature (or same neighboring features) might get attached to a multiple LIDAR points (features) as points at different distances along a ray might all be projected to the same image point? There is nothing to deal with this, and this might actually be reducing the expressiveness of the LIDAR features. Maybe a BEV representation (used in Wang et. al. 2021) is a better choice of representation?
-LIDAR head auxiliary task #2: 3D center estimation: how is this an auxiliary task? Doesn’t the 3D bounding box estimation already do this?
-Table 1 should include a comparison with Wang et. al. 2021, since that is the closest work to this.

Minor corrections/clarifications:
-Presumably the NLC map estimation aux task is only done with cars and bikes and not pedestrians - but this is not mentioned anywhere.
-In the abstract: “..we further construct and interactive bidirectional…” What do you mean by interactive?
-More details about the 3D semantic segmentation task will be nice.
-Figure 6 caption: “…multi layer perception.” Should be corrected to multi layer perceptron.
-Table 3: Please include labels a-g in the caption.
-Equation 10: q_pi and qcap_pi in equation 10 are referred to as pi and pi_cap in the explanation. Make this consistent.
-Table 11 is a bit hard to read. Are you combining experiments with changing distance and changing occlusion? How do you define levels of occlusion?

clarity,_quality,_novelty_and_reproducibility: The paper is easy to read and has enough experimental results to suggest that the approach, though very incremental, works well. Code has been provided, but I have not tried to reproduce it.

summary_of_the_review: This paper provides an incremental improvement to image and point cloud fusion for 3D object detection by doing so in both directions and at multiple levels of an encoder-decoder network. Auxiliary tasks further improve performance. The authors have achieved leading results in the cycle class on the KITTI leaderboard.

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: No ethics concerns as the paper uses widely popular existing datasets.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1174/Reviewer_9Ksn
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposed a novel fusion layer, which allows feature-map fusion across image and lidar backbones. Fusion is enabled bidirectional, which allows jointly training for multiple auxiliary tasks as regularization. This work also introduces a novel normalized local coordinate (NLC) map estimation, which predicts the perpixel prediction of the featuremaps encoded with rotation and geometry information of each object. Combining with other auxiliary tasks like 2D & 3D segmentation, the proposed approach is able achieve superior performance. 
strength_and_weaknesses: 1. To my knowledge, the bidirectional fusion module is novel. It enables joint optimization for both image and lidar backbones using multiple training tasks. 
2. The overall dual backbone representation is also novel and provides a natural way of feature fusion. 
3. The experimental results are strong.

It would be great if the author could also benchmark the proposed approach on other datasets, e.g. NuScenes to show the generalization ability of the proposed approach.  
clarity,_quality,_novelty_and_reproducibility: The paper is well written and has included enough details to reproduce the implementation. The authors also attached the training code.
summary_of_the_review: Good paper. Recommend for acceptance 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper1174/Reviewer_2AJq
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper proposed a LiDAR+RGB fusion-based 3D detection method. Compared to prior work, the main claims are twofold: 1) the use of bidirectional fusion between 3D point cloud features and 2D image features so that the 3D backbone can be improved with gradients coming from supervision on the 2D branch; 2) the use of another 2D auxiliary task, i.e., normalized relative coordinate estimation for each object on the 2D image. Results on the KITTI test server shows solid performance compared to recent methods. 
strength_and_weaknesses: Paper Strengths

1. The design of the proposed fusion method is reasonable, i.e., by building on top of prior work with pixel-to-point fusion, this work also investigates point-to-pixel fusion and combines them to build bidirectional fusion. Experiments do support that both directions improve the performance and the 3D backbone can be improved with additional gradient flow

2. The performance of the proposed method on the KITTI test set is solid, which can make this work useful to the community if the code can be released (I haven’t checked personally for reproduction but the authors state that the code is in the supplementary)

3. The other proposed minor contribution – normalized local coordinate estimation task – seems intuitive to me. Since estimating relative coordinates requires some kind of 3D understanding, it can benefit the 3D features coming from the 3D point cloud backbone and potentially also improve the 3D backbone during training as the paper claims. Some ablation also supports this claim. 

Paper Weaknesses

I do not see a critical concern that significantly impacts my score, but below are some minor concerns.

1. I am happy with the performance the paper achieves on the KITTI test set. However, as the KITTI dataset was built a long time ago and is quite small (and can potentially lead to overfitting issues) compared to other recent large-scale datasets, people tend to focus more on the nuScenes/Waymo datasets recently. So it would make the paper’s results significantly more trustable if the results can be demonstrated on these more recent large datasets, and use KITTI results to supplement. I understand that working with a new dataset can be practically time-consuming, but presumably, this could be something the authors have already been working on after the submission (since it is quite obvious). 

2. Some of the claims about related work might be inappropriate and could be improved. For example, in related work and under “monocular 3D object detection”, the paper states that “images have not been demonstrated as an appropriate modality to estimate 3D boxes”. I think this claim might raise concerns from many readers and is quite outdated (might be true 2 years ago). For example, Tesla has been using images to do auto-piloting nowadays and shows reasonable performance. Also, looking at the new STOA methods on the nuScenes/Waymo leaderboard, there are many successful image-only methods (e.g., BEVStereo, BEVdepth, BEVFormer) that achieve strong performance, though not beating lidar-based methods but are getting closer. Similarly, the statement about the pseudo-lidar-based method – “it is non-trivial to optimize the depth estimator and LiDAR detection simultaneously in pseudo-lidar based methods” – might be problematic since there are already works in that direction (e.g., one of the first work, End2end pseudo-lidar, CVPR 2020). I would suggest that the authors may reconsider improving these statements about prior work

3. One minor question I have regarding the ablation study in Table 3 is: comparing row (a) and row (b) solely, it seems that adding 3D segmentation auxiliary task seems not to bring any improvement to the detection task. Are these two tasks really beneficial to each other in this multi-task learning setting? Any more concrete evidence in addition to the results in row (d)? I am asking this because from the results, it seems not worth adding this auxiliary task considering its cost during training/inference. It would be nice if the authors could comment on this.

4. Although pixel-to-point fusion is one of the mainstream directions in the field as the authors mentioned, I think there is still some prior work conducting 3D detection with the point-to-pixel fusion between LiDAR and camera. I believe that I have seen a few in the past, but at this moment only remember one (EPNet++: Cascade Bi-directional Fusion for Multi-Modal 3D Object Detection). It would be nice if the authors can do a literature search in this direction and add some comments to summarize the differences/similarity between these works. 

clarity,_quality,_novelty_and_reproducibility: Quality and clarity are fine to me. Originality is related to my last point about the above weaknesses. Reproducibility is possible based on the paper but I haven't checked the code for performance reproduction
summary_of_the_review: Good explanation of the method, experiments supporting the main claims, and some incremental novelty with a solid performance on the KITTI dataset, but some problems about related work can be improved and clarified
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: n/a
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1174/Reviewer_vG37
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper tackles the problem of multiple modal (Lidar + Camera) based 3D object detection. In this work, auhtors explored to use both point-to-pixel and pixel-to-point  feature propagation schemes to achieve bidirectional feature propagation. Great experiment results are achieved in KITTI dataset.

strength_and_weaknesses: Strength:

* The idea of point-to-pixel projection is simple but effective.
* In this work, the effect of auxiliary tasks are well explored. In Table 3, we can find 2D tasks (2D Seg and NLC) really helped the 3D task.

Weekness
* Based on Table 3, the main performance gain of image branch comes from the auxiliary tasks. Simply adopting image branch brings few promotion. It is not fair enough to compare a model with only detection ground-truth and a model with multiple surpervision. Meahwhile, it is important to verify the effectiveness of auxiliary tasks on other methods. Thus, an important missing experiment is to add 2D seg and NLC losses to other cross-modal models.
* In this work, camera to lidar (pixel to point) propagation is conduct on  point-level. Recent works demonstrate that conduct cross-modal fusion on BEV space is more effective. Have authors tried achieve camera to lidar propagation with BEV fashion?
* Missing comparison experiment: there are plenty of cross-modal 3D detection methods this year, most of them conduct experiments on nuScenes dataset. Authors should discuss these recent works (Transfusion, BEVFusion e.g.) and conduct experiment on nuScenes dataset to further verify the effectiveness of proposed method. 
clarity,_quality,_novelty_and_reproducibility: 

The writing of this paper is clear. There is no Reproducibility issue as codes provided. The novelty of this work is somehow limited.

summary_of_the_review: In conclusion, this work propose a new point-to-pixel propogation scheme to achieve cross-modal detection. However, it seens the main performance promotion comes from auxiliary tasks. Meanwhile, the comparsion with recent works is not enough. Thus I give "marginally below the acceptance threshold" for now.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold