Paper: Object-level Data Augmentation for Visual 3D Object Detection in Autonomous Driving (ICLR-2024)

Reviewer comment (Experiments):
Is there any safeguard to guarantee the augmented data is located in reasonable regions?

Author response:
**Q1. How to guarantee the edited object be located in reasonable regions?**

We design a set of detailed rules for batch object editing, which could ensure that the edited objects appear in reasonable positions:

- **Adding objects**:
  - In driving scenarios, objects are mostly located on the ground plane. Therefore, we tend to place the added object on the ground. To achieve this, we approximate the ground plane function by analyzing scene data, including point cloud and object bounding boxes.
  - We randomly select a point within a certain range on the ground plane, place the object at that position, and rotate the object by a random angle parallel to the ground plane.
  - Subsequently, we perform collision detection between the added object's bounding box and other objects or the background scene. If there is a physical overlap with other objects, we will attempt another placement.
- **Moving objects**:
  - When moving static objects, we randomly choose a direction parallel to the ground plane and move the object a certain distance.
  - When moving dynamic objects, because of tricky issues such as velocity and direction, we employ a simple strategy: For each edit, we randomly select a number x. When handling a certain moving object at time t, we move it to its position at time t-x.
  - After an object is moved, we conduct collision calculations. If there is a physical overlap, the object undergoes re-editing.
- **Deleting objects**:
  - We randomly select an object in the scene for deletion. Most of the time the deletion operation does not result in the violation of real-world physical laws.

Finally, we perform manual inspections of the edited scenes to eliminate implausible elements.

This part will be included in the final version of the paper.

Reviewer comment (Experiments):
How does the algorithm control the appearance style of the added object? Is there a way to generate an unseen appearance from the training data set?

Author response:
**Q2. How to control the appearance style of the edited object? Is there a way to generate an unseen appearance from the training data set?**

The proposed augmentation method is based on the reconstruction of scenes and objects. Consequently, their appearance style is constrained to those present in the training dataset. The proposed method cannot generate object appearances with an unseen style. 

Recently, we have come across some excellent works about synthesizing new appearance styles of objects. It is an interesting and potential idea to train a unified generator for the objects in the same class, which could learn their common features and control the generated appearance style. In future work, we will explore this idea.

Reviewer comment (Experiments):
The PSNR of the synthesis image is <30 dB, which means the human eye distinguishes the synthesis image from the original image. It raises a question in my mind: does the quality of the synthesis data affect downstream tasks? An experiment that compares the results trained within the original image and the synthesis data is appreciated.

Author response:
**Q3. Comparison of the training results within the original image and the synthesis data.**

We design an experiment to discuss this issue. Considering the limited resources, we use 20% of the nuScenes data (a total of 140 scenes) as a mini-dataset for training. Additionally, we select 28 scenes from the mini-dataset and process them using different synthesis methods, which are then added to the mini-dataset. The synthesis methods we employed are as follows:

- **Rerender:** We rerender the original images by employing the same reconstruct & render method of the proposed method, without any camera setting and scene layout changes.
- **Original:** We make no modifications to the original images and add them directly into the mini dataset. This method serves as a baseline.

We employ these different mini datasets to train PETRv2, and the detailed training settings align with those outlined in Section 4.3 of the original paper. The detailed performance results are as follows.

| Dataset settings                                     | Synthesis Method | Synthesis data PSNR | Detection    mAP | Detection  NDS |
| ---------------------------------------------------- | ---------------- | ------------------- | ---------------- | -------------- |
| 140 original scene data + 28 **original** scene data | /                | /                   | 33.06            | 39.89          |
| 140 original scenes + 28 **synthesis** scene data    | Rerender         | 26.3*               | 32.58            | 40.37          |

[*] Note: We randomly sample a certain number of generated images, compare them to the original images, and calculate this PSNR value.

While there is indeed a gap in the quality between synthetic and original data, experimental results indicate that these differences do not significantly impact the model's detection performance. This further indicates that the proposed data generation method is applicable to downstream tasks.

**Reply to R3.** 

We conduct several additional experiments to address your question. 

For a quick evaluation, we use 5k keyframes of the nuScenes data as a mini-dataset for training, and replace different percentages of the mini-dataset with synthetic data. We employ these different mini datasets to train PETRv2, and the detailed training settings align with those outlined in Section 4.3 of the original paper. The performance results are as follows.

| Methods                                | mAP       | NDS       | mATE  | mASE  | mAOE  | mAVE   | mAAE  |
| -------------------------------------- | --------- | --------- | ----- | ----- | ----- | ------ | ----- |
| 100% original data                     | **26.75** | **31.83** | 91.40 | 29.87 | 82.51 | 88.23  | 23.43 |
| 50% original data + 50% synthesis data | 26.40     | 29.83     | 90.81 | 29.82 | 83.46 | 102.17 | 29.65 |
| 100% synthesis data                    | 25.40     | 29.35     | 93.29 | 30.46 | 83.45 | 99.42  | 26.86 |

We find that the detection performance of the model gradually declines with the increase of the proportion of synthetic data. This shows that augmented synthetic data can play an auxiliary role in model training, but can not replace the original data.

Considering the limited time resources, we can only provide results of very limited ratio settings. We will present more detailed experiments and analyses in the final version.

Reviewer comment (Evaluation):
Besides, the computation cost will be inevitably increased. The cost of generating the new training samples should be reported.

Author response:
Please note that our data synthesis process is **offline**. We generate the augmented data and store them before the detection model training. During the training process, we directly use the stored synthetic data. Therefore, the proposed augmentation process does not affect the training efficiency of the detection model.

We also provide the detailed computation costs of generating the new training samples as below:

- In the reconstruction stage：
  - **Time cost:** Our descriptors and render are trained on 4 RTX 3080Ti (12G) GPUS, which takes about **5 hours** for a scene with about 240 images (with a resolution of 1600x900). 
  - **Memory cost:** For a single scene, trained point-cloud descriptors take about **50MB~200MB** of storage, depending on the size of the point cloud. The trained UNet takes **116MB** as the renderer.

- In the rendering stage, the trained renderer takes about **20 seconds** to render a single image (with a resolution of 1600x900) from the point map on a single RTX 3080Ti (12G) GPU.

Reviewer comment (Experiments):
The authors only report the results on the nuScenes validation set, and the test set should also be presented.

Author response:
We conduct testing on the test set, and submit the results to the nuScenes detection challenge, obtaining the following performance metrics:

| Method             | mAP↑      | NDS↑      | mATE↓     | mASE↓     | mAOE↓     | mAVE↓     | mAAE↓     |
| ------------------ | --------- | --------- | --------- | --------- | --------- | --------- | --------- |
| **PETRv2***        | 39.68     | 49.39     | 69.56     | 26.27     | 47.49     | 48.87     | 12.36     |
| **PETRv2* + ours** | **42.00** | **51.27** | **67.05** | **25.97** | **47.29** | **44.81** | **12.15** |

The results of the test set further demonstrate that our method has brought significant performance improvements to the baseline.

*Note： The best device we can acquire is RTX 3080Ti (12G), which is hard to support large image resolution and strong backbones. Therefore, we use a training setting consistent with the paper's Section 4.3: a lightweight backbone and lower-resolution inputs. Additionally, we only utilize the training set for training and augmentation. As a result, the data in the table differs significantly from models at the forefront of the challenge leaderboard, which employ heavy backbones, high-resolution image inputs, and train on the entire dataset (train+val).

Reviewer comment (Novelty):
ROI-10D [1] also provides an instance-level data augmentation method for image-based 3D detection. Discussion and comparison should be included.

Author response:
**Q4. Discussion and comparison with ROI-10D.**

RoI-10D focuses on monocular 3D detection, which also provides certain methods to synthesize object-level edited images. In comparison to our method, the differences are outlined as follows:

- ROI-10D relies on CAD vehicle models to extract the vehicle meshes. Consequently, the range of augmented object categories and shapes is restricted. Our method, however, allows for the editing of all target objects present in the dataset, from small traffic cones to huge trailers.
- ROI-10D employs a single-frame image as input to reconstruct vehicles into 3D textured meshes. In contrast, our method utilizes the images and point cloud sequence across multiple frames, enabling the learning of more precise and detailed object shapes and texture information.
- ROI-10D's object editing methods are limited. For example, it cannot handle the removal of existing objects in the scene. Our method supports more flexible editing operations, including free rotation, movement, and deletion.

Regarding quantitative performance comparison, RoI-10D does not provide open-source code. Given the limited time window for the rebuttal phase, we regret that we are unable to reproduce it and give a quantitative assessment. We will complete the relevant experiments and present the results in the final version of the paper.

Reviewer comment (Experiments):
Comparison with data augmentation SOTAs. Although the authors compared the proposed method with the vanilla Cut & Paste in terms of the performance. However, since Cut & Paste doesn't use any point cloud raw data, so to have a fair comparison, it would be interesting if the authors could show the performance comparison with LiDAR-based data augmentation methods, i.e., Tongetal.(2023).

Author response:
Firstly, LiDAR-based augmentation methods are primarily employed for LiDAR 3D perception. Augmentation in the LiDAR point cloud is hard to synchronize to the image level. Our approach utilizes LiDAR as an intermediary to achieve augmentation of the image. In essence, our method serves as a bridge between LiDAR-based and image augmentation.

Secondly, Tong et al. (2023) mentioned does not necessarily rely on LiDAR data. In this method, LiDAR serves as a supplementary depth information source for NeRF procession. As Tong et al. (2023) do not public their source code and disclose detailed editing settings, within the limited rebuttal time windows, we apologize that we cannot conduct a quantitative comparison of augmentation algorithm performance. We will complete the relevant experiments and present the results in the final version of the paper.

Reviewer comment (Experiments):
It is unclear to me how large the augmented dataset is compared with the original one.

Author response:
The original nuScenes training dataset comprised **700 scenes**. We randomly selected **20%** of these scenes for augmentation. In the final training dataset, the ratio between original data and generated data was maintained at **5:1**.

Reviewer comment (Evaluation):
It would be interesting if the authors could show the detailed detection performance of 10 classes on the nuScenes. From my understanding, the proposed method is kind of sensitive to different objects with different sizes. Since the small-scale objects, like pedestrians and cyclists might only take a couple of pixels.

Author response:
We provide the detailed detection performance data of each object in nuScenes of PETRv2 in the following table.

| Methods      | mAP       | car       | truck     | bus       | trailer   | construction_vehicle | pedestrian | motorcycle | bicycle   | traffic_cone | barrier   |
| ------------ | --------- | --------- | --------- | --------- | --------- | -------------------- | ---------- | ---------- | --------- | ------------ | --------- |
| **Baseline** | 40.65     | 58.23     | 37.04     | 42.08     | **23.43** | 12.06                | 48.23      | **40.69**  | **39.85** | 56.10        | 48.81     |
| **+ ours**   | **41.45** | **59.33** | **38.32** | **43.26** | 23.10     | **12.82**            | **49.70**  | 40.01      | 39.26     | **58.50**    | **50.21** |

We can observe significant improvements in detection data for small objects (e.g., traffic_cone and barrier with 2.40 and 1.40 mAP increase respectively), medium-sized objects (e.g., car with a 1.10 mAP increase), and larger objects (e.g., truck and construction_vehicle with 1.28 and 0.76 mAP increase respectively).

It is worth noting that although the mAP increases for most object categories, there is a slight decline in the performance of motorcycles, bicycles, and trailers. Upon further analysis, we attribute this decline to the poor reconstruction performance of point cloud-based methods for objects with holes, such as the wheels of bicycles. Additionally, we identify challenges in the reconstruction of trailers, given their large and complex structures with diverse goods carried. These lead to insufficient quality in the generated data, causing a slight decline in performance data. In subsequent work, we will address and follow up on resolving these related issues.