Paper: GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation (ICLR-2024)

Reviewer comment (Experiments):
Related to 1), Stable Diffusion is not a good option for comparison of layout-based inpainting in Figure 5 as Stable Diffusion does not consume bounding box explicitly. I suggest to replace it with ControlNet and GLIGEN for a fair comparison.

Author response:
**W2. Additional baseline methods for the inpainting comparison in Figure 5.**

We provide a qualitative comparison between GeoDiffusion and Stable Diffusion in Figure 5 primarily to demonstrate how exactly the geometric-awareness enhances the inpainting ability. As shown in Figure 5, GeoDiffusion can better understand the requested inpainting locations especially under multi-object circumstances, and therefore, better deal with the inpainting requests.

Following your suggestions, we further provide comparison with ControlNet and GLIGEN in Figure 5, where GeoDiffusion still demonstrates more superior inpainting results.

Reviewer comment (Experiments):
Related to 1), the importance of camera views is not clearly due to the missing baselines. Furthermore, there are no qualitative illustration of the effect of camera views except a single example in Figure 1.

Author response:
**W3. More discussion on the camera view control generation.**

**(1) Importance of camera views** has been verified by the ablation study conducted in Table 10 and Appendix B. Specifically, we train a GeoDiffusion without camera view tokens with all the other settings unchanged. As reported in Table 10 (rows 1 and 2), the variant without camera view tokens is surpassed by the default GeoDiffusion with respect to both the image quality (*i.e.*, FID) and layout consistency (*i.e.*, mAP), revealing the importance of camera views.

**(2) More qualitative comparison of camera views** is provided in Figure 10 following your suggestions.
As demonstrated, GeoDiffusion can effectively convert the camera views simply by changing the camera view tokens in text prompts, while maintaining consistent with the given semantic layouts.

Reviewer comment (Evaluation):
In Table 4, the proposed GeoDiffusion outperforms other methods except in AP50. ControlNet significantly outperform all the other method at AP50 by a very large margin. I am quite curious what leads to this? I think it worths some discussion in the paper.

Author response:
**Q1. Discussion on the high AP50 but low mAP and AP75 performance of ControlNet in Table 4.**

This is primarily because ControlNet can only take image-level condition inputs instead of instance-level ones as GeoDiffusion.

**(1) Implementation of ControlNet.** To facilitate the layout-to-image generation, ControlNet is implemented by converting the semantic layouts to $C$ binary semantic masks, with each mask for a single semantic class, following the ControlNet paper. Therefore, the model can no longer distinguish different instances of the same class. When two instances of the same class are close with each other, the model would tend to generate a single but large instance to cover the areas of both instances, which is a typical reason to obtain a high AP50 but low mAP and AP75.

**(2) Qualitative verification.** A qualitative example is provided in Figure 5. As demonstrated, ControlNet completes the inpaiting requests by considering the left two people as a single instance, and therefore, only complements the right part of the human body.

Reviewer comment (Experiments):
Given the fact that recent diffusion-based methods such as ControlNet, ReCo, and GLIGEN can also do Layout-to-Image tasks, the authors should include them in Table 2 (Comparison of generation fidelity on NuImages) and Table 3 (Comparison of generation trainability on NuImages). Current comparisons in Tables 2 and 3 are not up-to-date.

Author response:
As discussed in Section 4.3, we compared GeoDiffusion with recent diffusion-based methods, including ControlNet, ReCo, and GLIGEN, on the COCO-Stuff dataset using publicly available checkpoints. In these comparisons, GeoDiffusion has demonstrated significant superiority.

Following your suggestions, we have also reported the performance of ControlNet, ReCo, and GLIGEN on the NuImages dataset in Tables 2 and 3, where GeoDiffusion continues to show consistent improvement. 
For detailed experimental results, please refer to **the common response to Reviewers WbtJ and VRFf**.

Reviewer comment (Novelty):
The proposed camera-dependent conditioning prompt is very similar to the view-dependent prompting in DreamFusion. The authors need to ablate more on why their method is novel compared to the one proposed in DreamFusion.

Author response:
**(1) Controllability.** In GeoDiffusion, camera view tokens are explicitly utilized as input conditions to generate images from different camera views. This approach differs fundamentally from DreamFusion, where view-dependent prompting serves primarily as prior knowledge to enhance the supervision signal in training the NeRF model. DreamFusion still relies on explicit camera parameters to generate images from various views. In contrast, our method leverages direct input conditions, offering more intuitive controllability over the image generation process.

**(2) Purpose.** Our primary goal with camera views in GeoDiffusion is not to introduce a novel view encoding method but to demonstrate that text prompts can serve as a unified representation for various geometric conditions, facilitating independent manipulation without interdependencies. As illustrated in Figure 10, the simple conversion of camera view tokens in GeoDiffusion can directly generate images from different perspectives while maintaining semantic integrity. This showcases our method's ability to handle geometric conditions more flexibly than DreamFusion.

Reviewer comment (Experiments):
Lack of experiments of long-tail (rare) object detection datasets such as LVIS and domain adaptation such as GTAV to Cityscapes or Cityscapes to Foggy Cityscapes. In these settings, the importance of synthetic data is more significant than in the domain where real images and box annotations are largely available.

Author response:
**(1) Long-tailed generation on NuImages (semantic classes).** The NuImages dataset is inherently long-tailed, with rare classes such as trailers and construction vehicles comprising only 1.5% of training annotations (Page 7, second paragraph). Tables 2 and 3 demonstrate GeoDiffusion's efficacy in addressing the long-tailed object detection challenges.

**(2) Long-tailed generation on NuImages (weathers and time of day).** GeoDiffusion can generate images under different weather conditions and times of day through tailored text prompts (*e.g.*, **"A {weather} {time} image of {camera} camera with {bboxes}"**). Figure 9 showcases this ability, though limitations exist, such as the absence of foggy images in the NuImages dataset.

**(3) Application to the LVIS Dataset.** Thank you for your suggestions. The LVIS dataset, representing an extremely long-tailed scenario, is still a nightmare even for state-of-the-art models [1]. This suggests that adapting GeoDiffusion to LVIS requires a specifically designed optimization recipe, which we plan to explore in the future work.

---

**W5-2. Support of GeoDiffusion for domain adaptation.** 

GeoDiffusion shows adaptability to various domains like daytime, rainy, and night scenes, as shown in Figure 9. We aim to extend this capability to more diverse domain adaptation scenarios in our future work.

Reviewer comment (Presentation):
It would be better to move the ablation study from supp (table 9, 10) to the main paper.

Author response:
Thank you for your suggestion to include Tables 9 and 10 in the main paper. Due to page limitations, we cannot move these tables directly. However, we have made clear references in Section 4.4 to these tables in the supplementary material, ensuring easy accessibility for readers interested in our detailed ablation studies.

Reviewer comment (Experiments):
Can the model generate rare cases in the dataset? For example, can the model generate diverse night-time, foggy, raining scenes in NuImages? Another interesting question is can the model generate more samples of rare classes on a long-tailed dataset (LVIS).

Author response:
**(1) Long-tailed generation on NuImages (semantic classes).** The NuImages dataset is inherently long-tailed, with rare classes such as trailers and construction vehicles comprising only 1.5% of training annotations (Page 7, second paragraph). Tables 2 and 3 demonstrate GeoDiffusion's efficacy in addressing the long-tailed object detection challenges.

**(2) Long-tailed generation on NuImages (weathers and time of day).** GeoDiffusion can generate images under different weather conditions and times of day through tailored text prompts (*e.g.*, **"A {weather} {time} image of {camera} camera with {bboxes}"**). Figure 9 showcases this ability, though limitations exist, such as the absence of foggy images in the NuImages dataset.

**(3) Application to the LVIS Dataset.** Thank you for your suggestions. The LVIS dataset, representing an extremely long-tailed scenario, is still a nightmare even for state-of-the-art models [1]. This suggests that adapting GeoDiffusion to LVIS requires a specifically designed optimization recipe, which we plan to explore in the future work.

Reviewer comment (Experiments):
In Table 2 (Comparison of generation fidelity on NuImages), there is a version of GeoDiffusion with an input resolution of 800x456, how is this obtainable while Stable Diffusion input is 512x512?

Author response:
The UNet of the Stable Diffusion mainly consists of convolutional blocks and Transformer blocks, both of which are invariant to the input resolution. Therefore, we can flexibly convert the image resolution during fine-tuning.

Reviewer comment (Experiments):
How do the location bins resolutions affect generation in testing? For instance, if the location bins are trained at 256x256, can the model generate a prompt with a 512x512 grid in test time?

Author response:
To clarify, the grid size of location bins should be consistent between the training and testing time, otherwise the model can not correctly understand the given bounding boxes, and therefore, fail to generate objects in the specified locations.

Reviewer comment (Evaluation):
Inconsistent performance in some metrics. For example, FID with 512 setting is better in Table 2. The mAP metric for Ped and Cone is slightly lower in Table 3. Any discussions on this?

Author response:
**(1) FID score for the 512 $\times$ 512 model is better than that of the 800 $\times$ 456 variant in Table 2.** This can be attributed to the alignment between the pre-trained resolution of the Stable Diffusion model and the fine-tuning resolution of the 512 $\times$ 512 model. The Stable Diffusion model, in our study, is originally pre-trained at a resolution of 512 $\times$ 512. When we fine-tune this model at the same resolution (512 $\times$ 512), it results in a more effective transfer of learned features and parameters. This alignment of resolutions minimizes the discrepancies that might arise from adapting to different resolutions, and thereby enhancing the model's performance in terms of the FID score. However, when fine-tuning at a different resolution (as in the case of the 800 $\times$ 456 variant), the model faces the challenge of adapting to a new resolution, which can lead to a slight decrease in FID score.

**(2) Lower mAP for pedestrian and traffic cone than the real-image-only baseline in Table 3.** In Section 4.2.1, we discuss the challenges that GeoDiffusion faces, particularly with the high variance and small objects, such as pedestrians and traffic cones, where the performance of our model is slightly lower compared to the Oracle real-image-only baseline. This indicates that the quality of the generated data for these two categories may not be sufficient to significantly enhance the performance of object detectors. Similar difficulties are encountered by other generative models as well. As shown in Table 3, except for the real-image baseline, our method outperforms other generative competitors, including LostGAN, LAMA, Taming, ReCo, GLIGEN, and ControNet. We are the first to demonstrate that generated data can serve as effective data augmentation to improve the overall performance of object detectors. We will continue to focus on improving the unsatisfactory performance in generating high variance and small objects in our future work.

Reviewer comment (Evaluation):
As claimed in Section 4.2.1, the proposed method achieves 4X acceleration in the training procedure, how is it measured?

Author response:
Sorry for the confusion caused. The measure of training acceleration is calculated based on the number of training epochs required to achieve a comparable level of performance. In Table 2, we demonstrate that our GeoDiffusion model, trained for only 64 epochs, significantly outperforms models like LostGAN, LAMA, and Taming, which were trained for 256 epochs. This indicates that GeoDiffusion achieves the same or better performance four times faster in terms of epochs, hence the 4X acceleration in training. To ensure clarity, we have revised the manuscript to explain this comparison more explicitly.

Reviewer comment (Novelty):
The pipeline is quite straightforward with diffusion process in an encoder-decoder architecture. The results are impressive. What's the key ingredient? Why previous methods with more complicated design fail to achieve the performance?

Author response:
**(1) Utilization of foundational pre-trained models.** Unlike GAN-based methods, a significant advantage of GeoDiffusion lies in leveraging large-scale pre-trained text-to-image diffusion models, such as Stable Diffusion. These models enable the generation of highly realistic and diverse detection data, which is crucial for effective data augmentation.

**(2) Transferability of the text encoder.** In contrast to existing methods like GLIGEN and ControlNet, which require designing specific bounding box encoding modules and training parameters from scratch, GeoDiffusion capitalizes on the transferability of the text encoder. This approach allows for more efficient adaptation and reduces the need for annotated training data. This is particularly beneficial for long-tailed classes with scarce data annotations. For instance, GeoDiffusion shows remarkable improvement in rare classes such as trailers and construction vehicles, achieving +11.9 and +15.0 mAP, respectively, compared to GLIGEN in the Top-2 rare classes on NuImages, as demonstrated in Table 2. The efficacy of the text encoder's transferability for geometric conditions is further supported by our ablation studies in Table 9 (rows 1 and 2).

**(3) Usage of the foreground prior re-weighting.** This strategy significantly enhances generation performance by focusing on the modeling of foreground objects. The positive impact of this approach is evident in the comparison between the results in Table 9 (rows 2 and 5).