Paper: Towards Generalizable Multi-Camera 3D Object Detection via Perspective Debiasing (ICLR-2024)

Reviewer comment (Evaluation):
It is better to analyze that the 2D feature is more suitable to deal with the DA or UDA problems, ie., some statistical analysis or specific documentary evidence.

Author response:
# **1. Suitability of 2D detector**

Here, we re-elaborate the suitability of the 2D detector from four aspects, including statistical indexes, theory, ablation study, and related work:


**Statistical result.** Comparing the performance of 2D and 3D predictions can be challenging due to the lack of suitable metrics. To address this issue, we adopt a straightforward approach by projecting the 3D prediction result center onto the 2D front camera and evaluating the Focal loss on the front camera. We report the average Focal loss in the source domain (nuScenes) and the target domain (Lyft) for both 2D and 3D predictions, as shown below:


| Method| Source:nuScenes | Target:Lyft | 
| :---:  | :---:  |:---:  |  
| 2D | 0.6557 | 0.7492 |
| 3D | 0.6623 | 1.1412 |


The results demonstrate that 2D detection results are significantly more stable than 3D results in both the source and target domains, with the 2D detection results being notably higher than the 3D results in the target domain.


**Experimental proof.** In Table 2, we conduct an ablation experiment on our approach by explicitly adding 2D extra supervision (training on the source domain) and rerendering, which leads to significant improvements.



**The mathematical derivation.** We provide the mathematical derivation that the 3D prediction error caused by domain shift can be reflected in a single perspective (2D image plane), and the final error on the single perspective is defined as Eq. (2). The detailed proof is presented in Appendix C. This derivation demonstrates that correcting errors in different 2D planes can reduce the errors in the final 3D prediction results, providing a theoretical basis for our method.



**Proof of related work:** Extensive research has been undertaken to exploit 2D techniques in the context of 3D applications. One such approach involves incorporating precise 2D outcomes as supplementary information into the network, thereby augmenting its capabilities [1, 2]. However, these methods tend to compromise the original network results and increase inference delays, whereas our proposed method preserves the original network structure. Another method entails utilizing 2D-3D consistency to enhance network detection capabilities [3, 4]. Although this approach bears some resemblance to our proposed method, we will elucidate the differences between our algorithm and these existing methods in part 4.
 


[1] Wang Z, Huang Z, Fu J, et al. Object as Query: Lifting Any 2D Object Detector to 3D Detection. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 3791-3800.

[2] Yang C, Chen Y, Tian H, et al. BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 17830-17839.

[3] Yang J, Wang T, Ge Z, et al. Towards 3D Object Detection with 2D Supervision. arXiv preprint arXiv:2211.08287, 2022.

[4] Lian Q, Xu Y, Yao W, et al. Semi-supervised monocular 3d object detection by multi-view consistency. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 715-731.

Reviewer comment (Experiments):
In Table 1, are the settings of BEVDepth and PC-BEV aligned?

Author response:
# **2. The fair comparison with BEVDepth**


**The restatement of fairness in Table 1.** All parameters of the comparison in Table 2 are the same, including image resolution, learning rate and training rounds. The improvement of our algorithm is only the addition of additional auxiliary networks and auxiliary constraints.


**The comparison with a stronger baseline.** To align the result with the original paper, we improve our baseline by changing the backbone to R101-DCN and the input resolution to 768 $\times$ 1408. Additionally, both our proposed method and DG-BEV modify the basic backbone and image resolution. These approaches are tested on a cross-dataset (from nuScenes to Lyft), and the results for the source domain (nuScenes) and the target domain (Lyft) are presented in the table below:


| Method |Protocol| nuScenes (mAP / NDS) | Lyft (mAP / NDS) | 
| :---:  | :---:  |:---:  | :---:  | 
| BEVDepth| DG  |0.414 / 0.538 | 0.120 / 0.304 |
| DG-BEV | DG  |0.409 / 0.531 | 0.279 / 0.431 | 
| Ours | DG  | 0.421 / 0.541 | 0.324 / 0.484 |
| Ours |  UDA  | 0.417 / 0.539 | 0.335 / 0.497 |

Here, DG represents domain generalization, and UDA represents unsupervised domain adaptation. As shown in the table, the results in the source domain (nuScenes) have significantly improved, but the improvement in the target domain (Lyft) is less pronounced. This indicates that increasing the model size and image resolution has a limited effect on the model's cross-domain performance. It is worth noting that our proposed method still significantly improves the results in the target domain.

Reviewer comment (Experiments):
Many papers have shown that adding a 2D detection prediction task for the MC3D-Det series detectors will significantly boost the performance. I'm worried about how much of your current rise is coming from the 2D prediction of the detector. So you should scrupulously add an ablation to show the effect of 2D prediction (even other sub-modules) for source domain only.

Author response:
# **3. Ablation study**

In Table 2, we conducted an ablation experiment. In particular, in our comparison, the addition of 2D supervision improved performance, and our rerendering (source
domain debiasing) also improves performance.

Reviewer comment (Novelty):
You should discuss the relative works of consistent learning and extra 2D prediction, i.e., [a,b] etc.  
[a] Probabilistic and Geometric Depth: Detecting Objects in Perspective.  
[b] Towards 3D Object Detection with 2D Supervision.

Author response:
# **4. The comparison with 2D-3D consistent method**

The key difference between our render-based method and existing 2D-3D consistency approaches lies in the following aspects:


**Geometric feature correction with less semantic destruction.** Our rendering approach enables accurate projection of 2D images to the exact location of the BEV, thereby allowing the network to modify the geometric position of features with minimal semantic destruction. In contrast, 2D-3D consistency constraints have a significant impact on the entire network and often result in the destruction of semantic information. To compare our approach with existing 2D-3D consistency methods, we selected two representative approaches, namely Hybrid-Det [1] and MVC-Det[2]. Specifically, we report the unsupervised domain adaptation result (mAP/NDS) of these methods in nuScenes (source) and Lyft (target) datasets:


| Method| nuScenes (source) | Lyft (target) | 
| :---:  | :---:  |:---:  |  
| DG-BEV |0.330  / 0.397 | 0.284 / 0.435 |
| MVC-Det | 0.314 / 0.374 | 0.288 / 0.437 |
| Hybrid-Det | 0.301 / 0.361 | 0.292 / 0.441 |
| Ours | 0.331 / 0.396 | 0.316 / 0.476 |


As shown in the table above, our algorithm achieves excellent performance. Although MVC-Det and Hybrid-Det exhibit improved performance in the target domain, their results in the source domain have deteriorated significantly. This is because other 2D-3D consistency methods enable them to fine-tune the entire network with semantic destruction. In contrast, our algorithm modifies the geometric space position of the feature, which leads to relatively stable results in the source domain.


**The unified method for any detection head.** MC3D-Det offers multiple detection heads, including a combination of classification and regression (Centerpoint) and end-to-end (DETR) approaches. However, designing a uniform differentiable mechanism that satisfies 2D-3D consistency is challenging due to the diverse nature of these detection heads. For instance, in Centerpoint, the final prediction is obtained by combining the outputs of the classification head and an offset regression head. It is difficult to effectively supervise the classification head in this case. In contrast, our approach presents a unified solution through a rendering approach.


**Significant improvement on both source and target domains.** Our approach is highly effective in improving the performance of both the source and target domains. In the source domain, our approach utilizes rendering to generate new views with different camera parameters, enabling the network to learn perspective-independent features.  In the target domain, our approach employs a more robust 2D detector to correct the 3D results by rendering. The other 2D-3D consistency methods do not have much effect in the source domain because 3D supervision is inherently more accurate and informative. However, our approach stands out by enabling the supervision of new perspectives through rendering.


[1] Yang J, Wang T, Ge Z, et al. Towards 3D Object Detection with 2D Supervision. arXiv preprint arXiv:2211.08287, 2022.
[2] Lian Q, Xu Y, Yao W, et al. Semi-supervised monocular 3d object detection by multi-view consistency. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 715-731.

Reviewer comment (Novelty):
This article corrects the model's bias through the consistency of 2D detection and 3D detection. I am quite curious whether the 2D render is necessary. Is it possible to project the 3D box into a 2D plane and supervision only applied to the 2D bounding box. There are many papers 3D consistency supervision on monocular 3D detection. From this perspective, the novelty of the model is insufficient.

Author response:
The key difference between our render-based method and existing 2D-3D consistency approaches lies in the following aspects:


**Geometric feature correction with less semantic destruction.** Our rendering approach enables accurate projection of 2D images to the exact location of the BEV, thereby allowing the network to modify the geometric position of features with minimal semantic destruction. In contrast, 2D-3D consistency constraints have a significant impact on the entire network and often result in the destruction of semantic information. To compare our approach with existing 2D-3D consistency methods, we selected two representative approaches, namely Hybrid-Det [1] and MVC-Det[2]. Specifically, we report the unsupervised domain adaptation result (mAP/NDS) of these methods in nuScenes (source) and Lyft (target) datasets:


| Method| nuScenes (source) | Lyft (target) | 
| :---:  | :---:  |:---:  | 
| DG-BEV |0.330  / 0.397 | 0.284 / 0.435 |
| MVC-Det | 0.314 / 0.374 | 0.288 / 0.437 |
| Hybrid-Det | 0.301 / 0.361 | 0.292 / 0.441 |
| Ours | 0.331 / 0.396 | 0.316 / 0.476 |


As shown in the table above, our algorithm achieves excellent performance. Although MVC-Det and Hybrid-Det exhibit improved performance in the target domain, their results in the source domain have deteriorated significantly. This is because other 2D-3D consistency methods enable them to fine-tune the entire network with semantic destruction. In contrast, our algorithm modifies the geometric space position of the feature, which leads to relatively stable results in the source domain.


**The unified method for any detection head.** MC3D-Det offers multiple detection heads, including a combination of classification and regression (Centerpoint) and end-to-end (DETR) approaches. However, designing a uniform differentiable mechanism that satisfies 2D-3D consistency is challenging due to the diverse nature of these detection heads. For instance, in Centerpoint, the final prediction is obtained by combining the outputs of the classification head and an offset regression head. It is difficult to effectively supervise the classification head in this case. In contrast, our approach presents a unified solution through a rendering approach.


**Significant improvement on both source and target domains.** Our approach is highly effective in improving the performance of both the source and target domains. In the source domain, our approach utilizes rendering to generate new views with different camera parameters, enabling the network to learn perspective-independent features.  In the target domain, our approach employs a more robust 2D detector to correct the 3D results by rendering. The other 2D-3D consistency methods do not have much effect in the source domain because 3D supervision is inherently more accurate and informative. However, our approach stands out by enabling the supervision of new perspectives through rendering.


[1] Yang J, Wang T, Ge Z, et al. Towards 3D Object Detection with 2D Supervision. arXiv preprint arXiv:2211.08287, 2022.

[2] Lian Q, Xu Y, Yao W, et al. Semi-supervised monocular 3d object detection by multi-view consistency. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 715-731.

Reviewer comment (Evaluation):
How to evaluate the quality of the 2D branch of the render, at first I thought the model render the rgb image, but after carefully reading the paper, I found it was mainly about the heatmap. However, from Figure 3 (c), it does not show the quality of the rendered heatmap very well. So, how do we validate the motivation well.

Author response:
Figure 3 shows that a more robust 2D detector can be used to correct spurious BEV geometry by correcting the rendered 2D heatmaps. In fact, Fig. 3 (d) is the heatmaps of our method, while Fig. 3 (c) is the heatmaps affected by domain shift without using our method. Specifically, Fig. 3 (c) shows that the rendering heatmaps of the target domain deteriorate due to domain shift, indicating that the chaotic BEV geometry. After correction by our algorithm, Figure 3 (d) shows more accurate rendering heatmaps on the target domain. In this way, our algorithm can use more robust 2D detection results to correct 3D chaotic geometric distributions. Our ultimate goal is to improve the detection performance, so we do not have quantitative statistical analysis on the heatmaps.

Reviewer comment (Experiments):
The proposed method has limitations as it has not been validated on sparse-query methods. In the past year, sparse-query methods like Sparse4Dv2, SparseBEV have shown great performance and speed advantages. Without an explicit BEV, would the domain shift problem still be as significant?

Author response:
We also conducted experiments on several methods without BEV representation (DETR3D [1], PETR [2], SparseBEV [3]) on cross-domain results (from nuScenes to Lyft), as shown in the table below.

| Method | w/o ours | w ours | 
| :---:   |:---:  | :---:  |
|         | **mAP/NDS**| **NDS/NDS**|  
| DETR3D$^*$      | 0.008/ 0.044 |0.028/ 0.076 |
| PETR$^*$      | 0.012/ 0.051 |0.032/ 0.091 |
| SparseBEV$^*$ | 0.016/ 0.059  |0.038/ 0.0097 |
| BEVFormer| 0.084 / 0.246 | 0.208 / 0.355 |
| BEVDet     | 0.104 / 0.275 | 0.296 / 0.446 |
| FB-BEV    | 0.113/ 0.294 | 0.301 / 0.454|
| BEVDepth  | 0.114 / 0.296 | 0.304/ 0.458 |

Methods with an asterisk ($^*$) do not have BEV representation, while all others do. For these methods without BEV representation, we leverage the LSS mechanism in BEVDet [4] to build additional BEV presentation so that our algorithm can be used to improve these methods. Notably, our algorithm is model-agnostic, in other words, our algorithm can be applied to any method with image features and BEV features as shown in Fig. 2. According to the table, we draw a number of conclusions:


(1) Methods without BEV presentation perform very poorly across domains compared to other algorithms. This can be attributed to their tendency to overfit camera extrinsic parameters in a learning way. Conversely, methods with BEV representation solely employ camera extrinsic parameters to project 2D image features into 3D space through a physical modeling form. This approach is highly resilient to variations in camera extrinsic parameters, thereby increasing its robustness and reliability.  In conclusion, the BEV representation can effectively establish the connection of different perspectives through physical modeling, as opposed to learning. This way enables the model to have superior cross-domain generalization.


(2) Our algorithm has the potential to enhance the performance of DETR3D, PETR, and SparesBEV algorithms by rendering the new viewpoints from an auxiliary BEV presentation. This is because the process of rerendering new perspectives compels the network to acquire a generalizable BEV representation. The generalizable BEV representation, in turn, encourages the network to learn more robust visual features that mitigate the impact of overfitting camera parameters.


[1] Wang Y, Guizilini V C, Zhang T, et al. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. Conference on Robot Learning. PMLR, 2022: 180-191.

[2] Liu Y, Wang T, Zhang X, et al. Petr: Position embedding transformation for multi-view 3d object detection. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 531-548.

[3] Liu H, Teng Y, Lu T, et al. Sparsebev: High-performance sparse 3d object detection from multi-camera videos. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 18580-18590.

[4] Huang J, Huang G, Zhu Z, et al. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11790, 2021.

Reviewer comment (Experiments):
More experiments about sparse-query methods are needed to prove the effectiveness of the method.

Author response:
We also conducted experiments on several methods without BEV representation (DETR3D [1], PETR [2], SparseBEV [3]) on cross-domain results (from nuScenes to Lyft), as shown in the table below.

| Method | w/o ours | w ours | 
| :---:   |:---:  | :---:  |
|         | **mAP/NDS**| **NDS/NDS**|  
| DETR3D$^*$      | 0.008/ 0.044 |0.028/ 0.076 |
| PETR$^*$      | 0.012/ 0.051 |0.032/ 0.091 |
| SparseBEV$^*$ | 0.016/ 0.059  |0.038/ 0.0097 |
| BEVFormer| 0.084 / 0.246 | 0.208 / 0.355 |
| BEVDet     | 0.104 / 0.275 | 0.296 / 0.446 |
| FB-BEV    | 0.113/ 0.294 | 0.301 / 0.454|
| BEVDepth  | 0.114 / 0.296 | 0.304/ 0.458 |

Methods with an asterisk ($^*$) do not have BEV representation, while all others do. For these methods without BEV representation, we leverage the LSS mechanism in BEVDet [4] to build additional BEV presentation so that our algorithm can be used to improve these methods. Notably, our algorithm is model-agnostic, in other words, our algorithm can be applied to any method with image features and BEV features as shown in Fig. 2. According to the table, we draw a number of conclusions:


(1) Methods without BEV presentation perform very poorly across domains compared to other algorithms. This can be attributed to their tendency to overfit camera extrinsic parameters in a learning way. Conversely, methods with BEV representation solely employ camera extrinsic parameters to project 2D image features into 3D space through a physical modeling form. This approach is highly resilient to variations in camera extrinsic parameters, thereby increasing its robustness and reliability.  In conclusion, the BEV representation can effectively establish the connection of different perspectives through physical modeling, as opposed to learning. This way enables the model to have superior cross-domain generalization.


(2) Our algorithm has the potential to enhance the performance of DETR3D, PETR, and SparesBEV algorithms by rendering the new viewpoints from an auxiliary BEV presentation. This is because the process of rerendering new perspectives compels the network to acquire a generalizable BEV representation. The generalizable BEV representation, in turn, encourages the network to learn more robust visual features that mitigate the impact of overfitting camera parameters.


[1] Wang Y, Guizilini V C, Zhang T, et al. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. Conference on Robot Learning. PMLR, 2022: 180-191.

[2] Liu Y, Wang T, Zhang X, et al. Petr: Position embedding transformation for multi-view 3d object detection. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 531-548.

[3] Liu H, Teng Y, Lu T, et al. Sparsebev: High-performance sparse 3d object detection from multi-camera videos. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 18580-18590.

[4] Huang J, Huang G, Zhu Z, et al. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11790, 2021.

Reviewer comment (Presentation):
Figure 1 presents some ambiguity. It is not immediately clear how this figure is intended to convey the robustness of the perspective view to domain shifts.

Author response:
Figure 1 does not depict our specific approach or demonstrate how to convey the robustness of the perspective view to domain shifts. Its purpose is merely to illustrate our motivation: we reveal a considerable discrepancy between the 3D prediction results and the ground-truth, which is the domain gap arising from overfitting with limited viewpoints, camera parameters, and similar environments. However, the target domain lacks additional supervision to rectify this error. Fortunately, we have identified that 2D detectors exhibit remarkable cross-domain performance. Consequently, we utilize more robust 2D results to constrain the new rendered perspective and rectify the spurious BEV features in the target domain. Furthermore, we propose to augment the network's capacity to learn more robust BEV features by rendering 2D images from different perspectives in the source domain. In summary, we establish the relationship between BEV features and 2D images through a neural rendering method, which significantly enhances the generalization ability of the detection results.

Reviewer comment (Experiments):
An inquiry for further clarification: How would methods that do not employ perspective view transformations, such as DETR3D, fare in performance across varying domains?

Author response:
We also conducted experiments on several methods without BEV representation (DETR3D [1], PETR [2], SparseBEV [3]) on cross-domain results (from nuScenes to Lyft) as shown in the table below.

| Method | w/o ours | w ours | 
| :---:   |:---:  | :---:  |
|         | **mAP/NDS**| **NDS/NDS**|  
| DETR3D$^*$      | 0.008/ 0.044 |0.028/ 0.076 |
| PETR$^*$      | 0.012/ 0.051 |0.032/ 0.091 |
| SparseBEV$^*$ | 0.016/ 0.059  |0.038/ 0.0097 |
| BEVFormer| 0.084 / 0.246 | 0.208 / 0.355 |
| BEVDet     | 0.104 / 0.275 | 0.296 / 0.446 |
| FB-BEV    | 0.113/ 0.294 | 0.301 / 0.454|
| BEVDepth  | 0.114 / 0.296 | 0.304/ 0.458 |

Methods with an asterisk ($^*$) do not have BEV representation, while all others do. For these methods without BEV representation, we leverage the LSS mechanism in BEVDet [4] to build additional BEV presentation so that our algorithm can be used to improve these methods. Notably, our algorithm is model-agnostic, in other words, our algorithm can be applied to any method with image features and BEV features as shown in Fig. 2. According to the table, we draw a number of conclusions:


(1) Methods without BEV presentation perform very poorly across domains compared to other algorithms. This can be attributed to their tendency to overfit camera extrinsic parameters in a learning way. Conversely, methods with BEV representation solely employ camera extrinsic parameters to project 2D image features into 3D space through a physical modeling form. This approach is highly resilient to variations in camera extrinsic parameters, thereby increasing its robustness and reliability.  In conclusion, the BEV representation can effectively establish the connection of different perspectives through physical modeling, as opposed to learning. This way enables the model to have superior cross-domain generalization.


(2) Our algorithm has the potential to enhance the performance of DETR3D, PETR, and SparesBEV algorithms by rendering the new viewpoints from an auxiliary BEV presentation. This is because the process of rerendering new perspectives compels the network to acquire a generalizable BEV representation. The generalizable BEV representation, in turn, encourages the network to learn more robust visual features that mitigate the impact of overfitting camera parameters.


[1] Wang Y, Guizilini V C, Zhang T, et al. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. Conference on Robot Learning. PMLR, 2022: 180-191.

[2] Liu Y, Wang T, Zhang X, et al. Petr: Position embedding transformation for multi-view 3d object detection. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 531-548.

[3] Liu H, Teng Y, Lu T, et al. Sparsebev: High-performance sparse 3d object detection from multi-camera videos. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 18580-18590.

[4] Huang J, Huang G, Zhu Z, et al. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11790, 2021.

Reviewer comment (Writing):
The paper has clarity issues that make it difficult to understand. It is not clear what the perspective bias comes from in the first place, and how re-rendering different viewpoints addresses the issues of poor generalization. It is also not clear how the perspective debiasing works.

Author response:
I am very sorry for your confusion about my paper, which may be caused by your lack of knowledge about our field. This is my further elaboration of our paper, hoping to help you understand our paper.

# **1. What is the source of perspective bias in MC3D-Det?**

The perspective bias in MC3D-Det comes from overfitting with limited viewpoints, camera parameters, and similar environments. The perspective bias represents the error of the final 3D result projected in a single viewing (2D image plane).


# **2. How does re-rendering address the issues of poor generalization?**


We restate our method from three aspects: the specific approach, the in-depth reason, and the mathematical derivation.


**The specific approach**: we establish an implicit foreground volume (IFV) to associate the camera with the bird's eye view (BEV) plane and re-render different view maps from BEV features. IFV allows for rendering view maps from BEV features with different camera parameters. By supervising the rendering of maps from different viewing angles, BEV features can be more accurately placed in the exact location. For the target domain, we can use 2D detectors trained in the source domain to monitor the rendered view so that the BEV features fall to the exact location.


**The in-depth reason**: (1) For the source domain, supervising new view maps with different camera parameters can help the network learn perspective-invariant features. Because only the features of the 2D image are placed in the exact 3D position, we can re-render the reasonable result in a new viewpoint. (2) For the target domain, the more robust 2D detector can correct 3D results by reprojecting on a single image. This is because domain offset allows the network to place the 2D image in the wrong 3D position, which can be corrected by rendering consistency.


**The mathematical derivation**: The 3D prediction error caused by domain shift can be reflected in a single perspective, and the final error on the single perspective is defined as Eq. (2). Detailed proof can be found in Appendix C. In other words, correcting errors in different 2D planes can reduce the errors in the final 3D prediction results, which provides a theoretical basis for our method.


# **3. How does perspective debiasing work?**

We expound the principle of perspective debiasing on both the source domain and target domain:

For the source domain, supervising new view maps with different camera parameters can help the network learn perspective-invariant features. Because only the features of the 2D image are placed in the exact 3D position, we can re-render the reasonable result in a new viewpoint. By generating diverse view maps from bird's eye view (BEV) features using implicit foreground volumes (IFV) to relate the camera and BEV planes, the MC3D-Det framework can render view maps with varied camera parameters, which helps the network learn perspective- and environment-independent features. This enhances the model's robustness to different perspectives and domain conditions, improving its generalization ability on the source domain.

For the target domain, the more robust 2D detector can correct 3D results by reprojecting on a single image. This is because domain offset allows the network to place the 2D image in the wrong 3D position, which can be corrected by rendering consistency. By projecting the object's box from the ego coordinate to the 2D camera plane, generating category heatmaps and object dimensions, and using focal loss and L1 loss to supervise the class information and object dimensions on the source domain, the MC3D-Det framework can rectify the perspective bias of the view maps and learn the object heights and dimensions without requiring 3D data on the target domain. Additionally, a 2D detector is trained for the image feature using 3D boxes, and the depth of the network prediction is forced to learn normalized virtual depth to improve performance. This enables the network to learn the depth information without requiring explicit 3D data on the target domain.

Overall, perspective debiasing addresses the issues of limited viewpoints, camera parameters, and similar environments, enhancing the model's generalization ability on both the source and target domains. By re-rendering diverse view maps and rectifying perspective bias, the model can learn perspective- and environment-independent features, improving its ability to generalize to new and unseen scenarios.

Reviewer comment (Presentation):
Figure 3, the only figure in the paper that explains the perspective debiasing, is uninformative. It’s not clear how the re-rendering has improved the heatmaps.

Author response:
# **4. How does the re-rendering improve the heatmaps?**

Re-rendering is actually about fixing the messy geometry of the BEV space, in other words, putting 2D features in the right place. When 2D object features are placed in the wrong position, the rendered 2D heatmaps will be misrepresented. By monitoring the rendered heatmaps, it is to correct the incorrect geometric distribution in the BEV feature space. 

We visualized all kinds of heatmaps on the target domain as shown in Fig.3: (a) ground-truth, (b) 2D detector, (c) rendered from IVF, and (d) revised by 2D detector.  The heatmaps in Fig.3 (a) show the real annotation projected onto the 2D plane. The heatmaps in Fig.3 (b) are from 2D detectors trained in the source domain. The heatmaps in Fig.3 (c) are derived from the rendering of BEV features without $L_{con}$. The heatmaps in Fig.3 (D) are derived from the rendering of BEV features with $L_{con}$.

Reviewer comment (Reproducibility):
Which components of the model need to be retrained, and which ones are lifted off-the-shelf from existing BEV methods? Are they finetuned?

Author response:
# **6. Training details**

We retrain these detectors following their default settings and add the proposed constraints as training-only auxiliary networks and losses. Therefore, all components are retrained, but it is not a finetuning process. As shown in Figure 2, our algorithm only imposes additional constraints on image features and BEV features through some auxiliary networks. So, compared to the original algorithm, we just add additional loss and auxiliary networks during training. In Figure 2, the main pipeline is the existing MC3D-Det algorithm, and the perspective debiasing above is our proposed regular constraint. Not a fine-tuning process, but a complete training process with the auxiliary network and additional loss.

Reviewer comment (Reproducibility):
Quick minor comment on similar lines on DG-BEV. The reproduced results for DG-BEV in Table. 1 are much worse than the published results of DG-BEV (Table. 1). For Lyft -> nuScenes, for domain generalization, DG-BEV (published) == MC3D-Det > DG-BEV (reproduced). However, the code for DG-BEV is not available, so I would side with the authors here.

Author response:
# **1. The stronger baseline**

We improve our baseline by changing the backbone to R101-DCN and the input resolution to 768 $\times$ 1408. Additionally, both our proposed method and DG-BEV modify the basic backbone and image resolution. These approaches are tested on a cross-dataset (from nuScenes to Lyft), and the results for the source domain (nuScenes) and target domain (Lyft) are presented in the table below:


| Method |Protocol| nuScenes (mAP / NDS) | Lyft (mAP / NDS) | 
| :---:  | :---:  |:---:  | :---:  | 
| BEVDepth| DG  |0.414 / 0.538 | 0.120 / 0.304 |
| DG-BEV | DG  |0.409 / 0.531 | 0.279 / 0.431 | 
| Ours | DG  | 0.421 / 0.541 | 0.324 / 0.484 |
| Ours |  UDA  | 0.417 / 0.539 | 0.335 / 0.497 |

Here, DG represents domain generalization, and UDA represents unsupervised domain adaptation. As shown in the table, the results in the source domain (nuScenes) have significantly improved, but the improvement in the target domain (Lyft) is less pronounced. This indicates that increasing the model size and image resolution has limited effect on the model's cross-domain performance. It is worth noting that our proposed method still significantly improves the results in the target domain.

Reviewer comment (Evaluation):
Computational overhead compared to baselines: Proposed method adds an overhead to the baseline method. It would be helpful to quantify the additional parameters and GLOPs used compared to the methods evaluated in Table. 3.

Author response:
# **2. Parameter overhead**
Our algorithm does not add any inference overhead when reasoning. As shown in Figure 2, our algorithm adds auxiliary constraints on image features and BEV features during training. Although many additional auxiliary networks are used in this process, none of these networks participate in the reasoning process. In other words, the original structure of the network remains unchanged during inference. Moreover, our algorithm is model-agnostic, which can be applied to various MC3D-Det algorithms, such as BEVFormer and FB-BEV, as provided in Table 3.

Reviewer comment (Experiments):
The focus on the car category: Please mention the categories used for evaluation in Table. 1. Correct me if I am wrong, but all the evaluations only consider the vehicle and car category (Lyft <-> nuScenes) - a rigid object with easy-to-learn 3D geometry prior, allowing for a consistent rendering with a perturbed camera. Do the results also hold for categories like the pedestrian, cyclists or less represented rigid classes like truck, construction vehicle, bus, and trailer? Using Waymo -> nuScenes protocol here would be more informative.

Author response:
# **3. The results of other categories**

We conduct domain generalization tests (from nuScenes to lyft) on more categories, and the results are presented in the following table:

| Method | car | truck | bus | bicycle | pedestrian |
| --- | --- | --- | --- | --- | --- |
| BEVDepth | 0.103 | 0.000 | 0.000 | 0.000 | 0.004 |
| DG-BEV | 0.263 | 0.000 | 0.012 | 0.027 | 0.021 |
| Ours | 0.297 | 0.016 | 0.052 | 0.047 | 0.077 |

The table reveals that the other categories exhibited a substandard performance across all methods. Notably, our algorithm exhibited enhanced performance in the aforementioned categories. This improvement can be attributed to the different viewpoint heatmaps for diverse categories, as opposed to the use of original images. Such an approach has demonstrated the algorithm's resilience to non-rigid bodies. The lackluster performance of other categories can be attributed to the substantial variance in image representation across distinct datasets.

Reviewer comment (Experiments):
Camera perturbations used: The qualitative results do not show the rendered views from the perturbed cameras. Since perspective overfitting is an issue, we should augment the view to avoid biasing, but still, we want to stay within the camera extrinsics distributions of the target domain. How is this balance achieved? What is the magnitude of translation and rotation perturbations used for perspective debiasing in MC3D-Det? How does the performance change when the perturbations increase from the anchor positions?

Author response:
# **4. Information on Camera perturbations**

We provide a systematic discussion on the perturbations of camera parameters, which play a vital role in semantic rendering. The extrinsic parameters include perturbations in rotation (pitch, yaw, roll) and translation, while the intrinsic parameters mainly concern the focal length.

**The camera extrinsic parameters**

The perturbed range of pitch, yaw, roll, are discussed in the supplementary appendix Fig.9. As shown in Fig.9, a small increase in the pitch and roll range is optimal, as most of the cameras on the car are parallel to the road with only slight fluctuations. However, yaw perturbations require a larger augmentation angle to facilitate the network in learning perspective-invariant features.

Furthermore, we evaluate the perturbed range of translation (X, Y, Z), and the results are presented in the table above. 

| Perturbed X range (m)  | mAP | NDS |
| --- | --- | --- |
| [0.0, 0.0] | 0.290 | 0.438 | 
| [-1.0, 1.0] | 0.295 | 0.446 | 
| [-2.0, 2.0] | 0.301 | 0.453 | 
| [-4.0, 4.0] | 0.294 | 0.443 |


| Perturbed Y range (m)  | mAP | NDS |
| --- | --- | --- |
| [0.0, 0.0] | 0.290 | 0.438 | 
| [-1.0, 1.0] | 0.296 | 0.444 | 
| [-2.0, 2.0] | 0.304 | 0.457 | 
| [-4.0, 4.0] | 0.300 | 0.449 |


| Perturbed Z range (m)  | mAP | NDS |
| --- | --- | --- |
| [0.0, 0.0] | 0.290 | 0.438 | 
| [-1.0, 1.0] | 0.292 | 0.441 | 
| [-2.0, 2.0] | 0.294 | 0.443 | 
| [-4.0, 4.0] | 0.284 | 0.422 |


The ablation in the table reveals that random translation of the observation position to render a new perspective can effectively enhance the model's performance. Notably, the difference between the camera extrinsic parameters of different datasets primarily pertains to the camera position and camera yaw angle. Among them, the camera position relative to the ego coordinate of car does not exceed 2m. Based on the findings presented in Figure 9 and the table above, it can be inferred that our proposed algorithm can significantly enhance the robustness of the model. In other words, our perturbation range already includes the camera extrinsic range of the target domain.


**The camera intrinsic parameters**

The camera intrinsic parameters (i.e., focal length) are affected by the resize amplitude of the input image, so we conducted an ablation experiment on this:


| Perturbed resize range | mAP | NDS |
| --- | --- | --- |
| [-0.03, 0.06] | 0.274 | 0.399 | 
| [-0.06, 0.11] | 0.304 | 0.458 | 
| [-0.08, 0.18] | 0.299 | 0.447 | 
| [-0.14, 0.30] | 0.267 | 0.381 |


Our experimental results reveal that significant perturbations in these camera intrinsic parameters can cause objects located at the edge of the image to be missed, ultimately leading to poor results. Therefore, we adopt the default resize range ([-0.06, 0.11]) to mitigate the aforementioned issue. Obviously, such a small resize range don't meet the alignment of intrinsic camera parameters (focal length 553 on nuScence and 407 on Lyft). In short, the camera intrinsic parameters cannot be aligned, but can also effectively improve the performance of the target domain.