Paper: MagicDrive: Street View Generation with Diverse 3D Geometry Control (ICLR-2024)

Reviewer comment (Writing):
The paper’s claim of “geometry control” appears to be somewhat overstated. Geometry encompasses more than just pose and Box3D; it also includes shape, topology, and the ability to modify any object within the scene.

Author response:
Thank you for highlighting this concern. To clarify, we have added a footnote on page 2 that specifically defines the scope of "geometry control" as considered in this study. We hope this addition provides a clearer understanding of the term.

Reviewer comment (Novelty):
The novelty of this work is unclear to me, as I am not very familiar with this topic. Upon a quick review of BEVGen and BEVControl, it appears that the main difference lies in the new modeling of 3D bounding boxes (in this work, the authors decouple the 3D boxes and road maps and model them separately), the use of Stable Diffusion, and cross-view attention. However, none of these elements seem to be significantly innovative.

Author response:
Street view generation presents unique challenges, distinct from other text/pixel-level conditional generations. This complexity arises not only from the increased number of conditions but also from the unique format and perspective requirements, including the crucial aspect of multi-view consistency. 

1. **Difference in handling conditions**: Compared to BEVGen, MagicDrive employs a distinct framework and produces significantly higher-quality data. BEVControl, on the other hand, projects 3D annotations to image view for ControlNet guidance, which loses crucial occlusion relationships and multi-view connections. In contrast, MagicDrive leverages raw 3D bounding boxes and BEV maps for encoding, leading to enhanced controllability, as demonstrated in Figure 5. Our innovative encoding method maintains its novelty and effectiveness even when compared to later developments like DrivingDiffusion [1].
2. **Enhanced Multi-View Consistency**: Street views have a limited number of cameras and different camera configurations (e.g., exposure). Therefore, the novelty lies in proposing a simple module that effectively solves the problem. Despite BEVGen and BEVControl employing more complex methods, their results are relatively modest compared to MagicDrive (as shown in Figure 5).

Given that BEVControl was released in August, our work should be viewed as concurrent. Additionally, the effectiveness of our proposed methods is further substantiated by the ablations in Section 6 and Appendix C.

Reviewer comment (Experiments):
What if apply the data augmentation to the SOTA camera 3D detection models, can you achieve the new SOTA on nuscenes?

Author response:
Our paper primarily focuses on the controllable generation of high-quality street views. The training on downstream tasks is presented as a proof of concept, demonstrating the potential of synthetic data to enhance 3D perception models. For this purpose, we selected a strong and influential baseline model, BEVFusion, which remains a formidable contender in camera 3D detection when considering single-frame input.

We acknowledge that achieving new SOTA nuScenes often involves extensive engineering efforts, such as utilizing unannotated data [2-3] and applying test-time augmentation. Therefore, our approach is to compare with a reliable baseline, rather than striving for new SOTA results. This comparison is intended to underscore the effectiveness of our method in a realistic and pragmatic context.

Reviewer comment (Experiments):
the synthesized views are impressive, the experiments are conducted in 700 street-view scenes for training and 150 for validation, which is a much smaller scale than the real-world senario. Wonder how to possiblly make it generalizable for real world domain. Does this be helpful to improve the detection & other understanding tasks when the data is large.

Author response:
### Weakness 1: limited scale of the dataset used in experiments and the generalizability of MagicDive to real-world scenarios. 
NuScenes, a well-established real-world dataset for autonomous driving, forms the basis of our training, allowing MagicDrive to demonstrate strong generalization across various scenarios, including unseen objects' positions and maps. This adaptability is evident from our tests involving manual modifications of object boxes (rotation, deletion, and moving), as detailed in Figure 6. While the current dataset scale is smaller than real-world scenarios, our results already show promising generalization capabilities. Scaling up the dataset size would likely present a broader spectrum of real-world scenarios, enhancing the model's performance even further. It is expected that such an expansion would significantly enhance MagicDrive's utility in downstream perception tasks.

Reviewer comment (Experiments):
In addition, not only for the dark scene, many generated instances such as human can be distorted with diffusion models. Wonder how that affects the detection accuracy for each subclass. The author provides overall accuracy in 3D object detection, may also analysis the details how to mix the synthesized images and real-images for training the model.

Author response:
### Weakness 2: effects of distortion in generated images on the detection accuracy for each subclass and details on how synthetic images are integrated with real images.

Thank you for your insightful comments. In response, we have expanded Appendix E to include a subclass-specific analysis of detection accuracy. This analysis demonstrates that the integration of synthetic data, even with instances of distortion, positively impacts the detection performance of each subclass. For instance, pedestrian detection shows a notable improvement of +2.36 mAP. This suggests that despite some distortions, the high-quality rendering of objects, particularly those close to the ego car, substantially aids in enhancing perception model accuracy.

In our experiments, we generate an equal number of samples as the training set for data augmentation in BEV segmentation. For 3D object detection, we randomly delete 50% of objects in each scene for generation, as detailed in Sec. 5.2. Exploring various strategies for integrating real and synthetic data is indeed a worthy endeavor. While our current focus is on advancing high-quality street view generation, we acknowledge the importance of this area and leave its detailed exploration to future research.

Reviewer comment (Experiments):
Diffusion models are costly, Could the framework be extended or modified to handle real-time or near real-time massive generation requirements, which are crucial for applications in autonomous driving systems? This also related to handle dynamic entities in the scene such as moving vehicles or pedestrians, especially when synthesizing views over a period of time?

Author response:
1. **Regarding Real-time Generation.** Our current model is optimized more for offline applications, such as simulation and data augmentation, with a focus on realism and controllability rather than low latency. However, we are aware of the importance of real-time processing in autonomous driving. The generation time in our current setup is approximately 3 seconds for 6 views using a single V100 card. Although this is not yet real-time, advancements in inference acceleration (e.g., advanced sampling techniques, model distillation methods like the Latent Consistency Model [1], or engineering solutions like low-bit inference [2] and flash attention [3]) are promising areas for reducing this latency. These potential improvements are areas we plan to explore in future research to make our model more applicable to real-time scenarios.
2. **Handling Dynamic Entities.** To address the dynamic nature of autonomous driving scenes, we are exploring the integration of video generation techniques, such as tune-a-video [4], into MagicDrive. This would allow the model to account for moving entities over time. Preliminary results from these explorations show promise, and we have included a demonstration video in Appendix F and on our [anonymous website](https://magic-drive.github.io/). This indicates that adapting MagicDrive to dynamically changing environments is feasible.

Reviewer comment (Experiments):
How closely does the synthetic data correlate with real-world data and what measures were taken to ensure the accuracy and reliability of this synthetic data?

Author response:
### Question 2: measurement of generation accuracy and reliability

We employ two evaluation approaches, as detailed in Table 1. Firstly, traditional generation metrics (such as FID) assess the congruence between real and synthetic data distributions. Secondly, we utilize pre-trained perception models (BEVFusion for 3D object detection and CVT for BEV segmentation), originally trained on real data. The closer our results align with real data, the better. Additionally, the "C+L" improvement noted in Table 2 indicates a high degree of consistency between generated objects and LiDAR points, further evidencing our model's superior controllability.

Reviewer comment (Novelty):
The main concern is for their marginal technical contribution. The proposed method is ControlNet applied into street view generation setting with multi-modality condition. The novelty probably lies in how to organize the condition into controlNet setting, which might not sufficient for acceptance.

Author response:
Street view generation presents unique challenges, distinct from other conditional methods such as ControlNet. This complexity arises not only from the increased number of conditions but also from the unique format and perspective requirements, including the crucial aspect of multi-view consistency. Our technical contributions, detailed in Sec. 1, are summarized as follows:

1. **Utilization of Raw 3D Data in MagicDrive**: Unlike other works, MagicDrive is the first to employ raw BEV maps and objects' bounding boxes & camera pose with 3D coordinates for street view generation. This approach differs significantly from ControlNet for two main reasons:
    - **Interactive Conditions**: The conditions are interdependent. For instance, the relevance of 3D bounding boxes is contingent on the camera pose, and the terrain height in BEV maps is inferred from the objects' positional heights.
    - **Varying Coordinates/Perspectives**: We handle conditions presented in different coordinates and perspectives, such as BEV maps in BEV perspective and 3D objects and camera pose in LiDAR coordinates. All of them are distinct from the pixel-level conditions in ControlNet.
    
    Actually, we only adapt ControlNet's method for map condition injection, and propose our own encoders for other conditions. Note that MagicDrive is the first to show that ControlNet is capable of encoding conditions from different perspectives. Besides, our tailored encoding of the box and camera, as evidenced in Sec. 6 and Table 4, is crucial for performance and multi-view consistency.

2. **Innovative Cross-View Attention Module**: To address the limited overlap and varying characteristics of each camera view (e.g., intrinsic differences, and exposure variation as shown in Figure 5), we developed a simple yet effective cross-view attention module. This module ensures consistency across views, a feature not addressed by ControlNet. As demonstrated in Figure 5, MagicDrive outperforms BEVGen and BEVControl in achieving this consistency, offering a more effective solution.

Reviewer comment (Evaluation):
MagicDrive does not ensure consistency across adjacent frames after checking their website demo.

Author response:
It could be relatively easy to implement temporal consistency for video generation with MagicDrive. For instance, we applied the concept from 'tune-a-video' [1] to modify a low-resolution version of our MagicDrive, and the results are presented in both Appendix F and on our [anonymous website](https://magic-drive.github.io/). Since this extension is straightforward, it is not highlighted as a major contribution in our paper.

Reviewer comment (Reproducibility):
Do you have different CFG weights for different conditions? If so, I am curious on how you make that work.

Author response:
No. We consider whether CFG for map condition exists with others or not, and different weights (scales) for CFG, as shown in Figure 7. If exist, they share the same CFG weight.

Reviewer comment (Experiments):
The paper presents a ControlNet stable diffusion pipeline coupled with a multi-view conditional UNet, but beyond the benefit of encoding bounding boxes, it is unclear whether the chosen cross-view attention module for consistency is the best choice compared to other methods such as inpainting, panorama input, or feature volumes.

Author response:
1. **Differences between ControlNet and MagicDrive.**

   **Firstly, our task diverges significantly from ControlNet's scope**. The complexity of street view generation mandates considering multiple, interdependent conditions (e.g., BEV map, 3D object parameters, and camera pose). For example, given bounding boxes with 3D coordinates, only if the camera pose is presented can determine which box to appear in the view; given maps in BEV, the height of terrain should be reasoning from the height of objects' positions. This multi-faceted approach transcends ControlNet's simpler image-space controls by necessitating intricate interactions and perspective-aware condition handling. 

   **Secondly, our approach distinctively extends ControlNet by integrating a diverse set of conditions.** Our conditions differ substantially from standard images. We demonstrate in Sec. 6 and Table 4 (see w/o $E_{box}$ example) that our tailored encoding mechanisms are crucial for achieving high-quality results, especially in maintaining multi-view consistency.

2. **Challenges and non-trivial design in achieving multi-view consistency.** Implementing cross-view consistency in street view generation presents unique challenges. Unlike indoor settings, each street view has limited overlap and varies significantly in camera characteristics (e.g., intrinsic properties, exposure), as exemplified in Figure 5. Our MagicDrive notably outperforms existing methods like BEVGen and BEVControl in achieving more effective consistency, as demonstrated in the comparative analysis in Figure 5.

3. **Reasons for choosing cross-view attention over alternatives.** Other consistency methods, such as inpainting or panorama inputs, typically assume significant overlaps and uniform view characteristics, which are not feasible in street view scenarios. Our cross-view attention module, being straightforward yet highly effective, addresses these limitations by accommodating the disparate nature of street views, thereby surpassing the performance of other baseline methods.

Reviewer comment (Evaluation):
The comparisons in the paper are limited and use different baseline numbers, with BEVFormer used for some comparisons and CVT for others; for BEVFormer, the reported numbers are significantly lower than those in the original paper, and it is doubtful that this discrepancy is solely due to resolution differences, as no numbers match.

Author response:
1. **Reasons for using BEVFusion\* and CVT as baselines.** In the evaluation, we consider CVT for BEV segmentation because it is used in both BEVGen and BEVControl. We consider BEVFusion because it is one of the most influential works in 3D BEV object detection, which is also referenced as one of the baselines in DrivingDiffusion [1].

\* We think BEVFormer in the comments is a typo of BEVFusion.

2. **Why the reported numbers of BEVFusion are significantly lower compared to the original paper?** We have updated the number in Table 2 and Table 6 by enabling `test_mode` as described in [issue \#411](https://github.com/mit-han-lab/bevfusion/issues/411), which we did not notice when submitting our paper. Taking Table 6 "CAM-Only 2x" as an example, BEVFusion reports 35.56 mAP, while our reproduced mAP is 35.49. The mAP with data from MagicDrive is 35.74, improving the mAP by 0.25.

Reviewer comment (Experiments):
The method does not appear mature enough, as neither of the two baselines (BEVGen/BEVControl) have, to the best of the reviewer’s knowledge, been accepted at a major conference; furthermore, MagicDrive overlooks practical considerations such as temporally-consistent frames.

Author response:
1. **Problem settings and practical applications.** Street-view data generation is a promising way to improve the perception capabilities of autonomous vehicles. The key challenges of this task include achieving controllability and ensuring multi-view consistency. Our approach addresses these problems effectively. Additionally, we utilize nuScenes, a widely recognized real-world dataset for autonomous driving, to demonstrate the realism of our generated street views. The practicality of our method is further substantiated by evaluating its performance on two different perception tasks, both in training and testing environments. This underscores the relevance and applicability of our settings to real-world scenarios.

2. **Reasons for choosing BEVGen and BEVControl as baselines.** At the time of our submission, these baselines represented the forefront of using controllable generation for street view synthesis, addressing the challenge of multi-view consistency. Our comparison demonstrates that MagicDrive significantly surpasses these baselines in generation quality. Moreover, our experiments using the CVT show substantial improvements over BEVGen, highlighting the advancement our approach brings to the field. We do not apply DrivingDiffusion as a baseline since it was released on arXiv after the ICLR submission deadline.

3. **Extending MagicDrive to achieve temporal consistency.** Ensuring temporal consistency in the generation of MagicDrive is a relatively straightforward process. To illustrate, we have implemented a technique inspired by the method from tune-a-video[2], applying it to fine-tune a low-resolution version of MagicDrive. The preliminary results of this enhancement are detailed in Appendix F, and a demonstrative video is available on our [anonymous website](https://magic-drive.github.io/). These demonstrate that MagicDrive is not only scalable but also serves as a robust foundational model for controllable street-view generation.

Reviewer comment (Experiments):
Why are the BEVFusion numbers much lower, and why was BEVFusion not used for the BEV segmentation as well?

Author response:
1. **Why are the BEVFusion numbers much lower?** Sorry for the confusion caused. We have updated the results in Table 2 and Table 6 by enabling `test_mode` as described in [issue \#411](https://github.com/mit-han-lab/bevfusion/issues/411) in the GitHub of BEVFunsion, which we did not notice when submitting our paper. The updated results are close to those reported in the paper of BEVFusion. Still, MagicDrive improves the performance of 3D object detection.

2. **Why not use BEVFusion for BEV segmentation?** In fact, we did employ BEVFusion for BEV segmentation. However, due to the absence of an established baseline for comparison, we initially refrained from presenting these results in our submission. To address this, we have now included the BEVFusion-tested results for BEV segmentation on the nuScenes validation set in Appendix F．

Reviewer comment (Experiments):
If the aim is just to generate novel views, why not add additional elements to the bounding box images and use controlNet image encoding, as in DrivingDiffusion?

Author response:
**First, for 3D object encoding**, as illustrated in Figure 2 of Sec. 1, it is crucial for accurate 3D control. After projection from 3D to 2D, the occlusion relationship and connection between views, for example, is unclear. Please see the visualization in Figure 13 for further understanding. It will be much clearer if 3D positions are available. **Second, for raw BEV maps**, as demonstrated in the last paragraph of Sec. 4.1, projecting a map from BEV to FPV without height information is an ill-posed problem. Therefore, there is no guarantee that this projection will not affect fidelity.

Reviewer comment (Experiments):
If the data augmentation strategy works so well, why not start with a state-of-the-art method such as EA-BEV and see what improvements could be made from there?

Author response:
Our choice of BEVFusion as the baseline is motivated by its significant influence and robust performance in the field. The primary focus of our method is to explore the impact of generating high-quality street-view data, rather than to establish new benchmarks in downstream perception tasks. The improvements we have demonstrated in these tasks serve as a proof of concept for the effectiveness of our approach. In terms of generation capabilities, our method surpasses existing baselines. Furthermore, while EA-BEV utilizes 1600x900 resolution images, replicating this in our training and generation processes is challenging due to limitations in resources and time.

Reviewer comment (Experiments):
The method is heavily reliant on nuScenes; how could generalization be improved?

Author response:
Note that nuScenes is a typical autonomous driving dataset featuring multiple diverse driving views. Therefore, we chose it as our testbed. Our method is based on stable diffusion, pre-trained with billions of images, demonstrating strong generalization ability in variant applications. Given a larger and more diverse dataset, we anticipate that MagicDrive would exhibit enhanced generalization in generating novel street views.