Paper: 3D Dense Captioning beyond Nouns: A Middleware for Autonomous Driving (ICLR-2024)

Reviewer comment (Novelty):
The reviewer is not fully convinced by the importance and soundness of the `comprehensiveness` of the proposed task and dataset.

Author response:
Q1: (1) **does not go beyond previous definitions in terms of object-level comprehensiveness**.

We do use the pre-labelled 3D boxes from nuScenes, but to clarify, nuScenes annotate many boxes with the category of 'others' or 'pushable_pullable'. Our nuDESIGN dataset extends these annotations to detailed open-set entities like 'goose' or 'stroller'. So we argue that the object-level comprehensiveness is promoted.

Reviewer comment (Novelty):
The annotation process starts with pre-labeled 3D/2D bounding boxes, and the proposed task does not go beyond previous definitions in terms of object-level comprehensiveness. The reviewer questions why the authors did not directly attach more metadata to the original bounding box information.

Author response:
Q1: (1) **does not go beyond previous definitions in terms of object-level comprehensiveness**.

We do use the pre-labelled 3D boxes from nuScenes, but to clarify, nuScenes annotate many boxes with the category of 'others' or 'pushable_pullable'. Our nuDESIGN dataset extends these annotations to detailed open-set entities like 'goose' or 'stroller'. So we argue that the object-level comprehensiveness is promoted. 

(2) **attach more** **meta** **data**. 

Directly attaching meta data, expressed in real numbers describing geometric entities like [-2.60m/s, 1.64m/s] or [-23.78m, 0.05m], to the caption would greatly challenge LLM since they are not natural language. We conduct experiments with this new version of captioning, where we attach meta data to the caption. The downstream planning modules follow the same protocol. This representation underperforms DESIGN as shown below. This experiment demonstrates that the performance gain brought by DESIGN is non-trivial and serves as another evidence of DESIGN's effectiveness.

|                    | Input | C 1s     | C 2s     | C 3s     | Avg.     |
| ------------------ | ----- | -------- | -------- | -------- | -------- |
| Uniad              | 2D    | 0.05     | 0.17     | 0.71     | 0.31     |
| DESIGN w/ metadata | 2D    | 0.07     | 0.15     | 0.61     | 0.28     |
| DESIGN             | 2D    | **0.03** | **0.12** | **0.55** | **0.23** |

Reviewer comment (Novelty):
The authors claim their description goes beyond nouns compared to existing 3D bounding boxes. However, the current nuScenes dataset already provides bounding boxes with position, heading angle, velocity, and category. Besides converting these annotations into language, the proposed dataset mainly adds descriptions of objects’ visual appearance, as seen in Fig. 2(a,b), where most words describe objects' colors, which the reviewer considers not so helpful for driving. Furthermore, the dataset drops heading angle and accurate velocity information from classical box data, which can be critical for downstream tasks. Overall, the new middleware (language) does not contribute more useful information for driving.

Author response:
Q2: (1) **most words are about objects' colors** .

Indeed, we also include other important information like material, gloss, status (like 'clean') and most importantly, open-set entities. See Fig.8.

(2) **drops the heading angle and accurate velocity**.

To clarify, the DESIGN representation contains the moving status of 3D bounding boxes. For example, this is a ground truth caption in Fig.12: A stroller in the front left of ego car about 11 meters away is moving slowly to ego car in the crosswalk. As for whether accurate descriptions or rough natural language descriptions are better, the experiment above demonstrates that the 'metadata' representation underperforms DESIGN.

Reviewer comment (Experiments):
The paper overclaims a 'scalable automatic labeling pipeline.' The annotation process requires human evaluation in the loop, which is neither fully automatic nor easy to scale. The influence of human checks should be explained, including whether a fine-tuned expert captioner could achieve full automatic annotation when scaling to more diverse data.

Author response:
Q3: **over-claim of 'scalable automatic labeling pipeline**. 

Indeed, our labeling pipeline is not fully automatic but an auto+manual one.  A fully automatic labeling pipeline would generate pseudo labels and is not what we want. We would like to clarify that our annotation method is easy to expand. With this pipeline, a worker only needs to decide what is right and what is wrong, instead of writing down every sentence. If the caption is considered right, it is included into the ground truth. If the caption is considered wrong, we just send the instance into the next round. Our annotation process takes three rounds, which are 0-20%, 20%-50%, and 50-100% of the data volume. We have counted the number of manual corrections in the three rounds, which was about 42.1%, 23.3% and 9.7% respectively.

Reviewer comment (Evaluation):
Comparisons to previous methods do not bring more insights, and it is possible that the observed gains come from better detectors. The reviewer also asks what the corresponding captions are in 2D dense captioning.

Author response:
Q4: (1) **More baselines**.

|                     | Input | C@0.25 | B-4@0.25 | M@0.25 | R@0.25 | C@0.5 | B-4@0.5 | M@0.5 | R@0.5 |
| ------------------- | ----- | ------ | -------- | ------ | ------ | ----- | ------- | ----- | ----- |
| Vote2Cap-DETR+MLE   | C+L   | 187.6  | 57.1     | 40.5   | 74.5   | 189.9 | 57.3    | 40.9  | 74.8  |
| Vote2cap -DETR+SCST | C+L   | 198.3  | 58.0     | 41.9   | 75.6   | 200.1 | 58.4    | 42.1  | 76.1  |
| Ours                | C+L   | 218.0  | 60.2     | 43.8   | 76.2   | 220.3 | 61.5    | 50.9  | 80.1  |

We compare with the newest SOTA baseline Vote2Cap-DETR in the figure below. There are two settings: the first one exploits a standard cross entropy loss (denoted as maximum likelihood estimation, MLE) and the second one exploits a self-critical sequence training (denoted as SCST) that optimizes towards a better CIDEr metric. We can see that our model outperforms Vote2cap-DETR .

(2) **2D dense captioning**.

We show the results of 2D dense captioning on Tab.20.

Reviewer comment (Experiments):
Although the method serves as a baseline for the proposed task, the reviewer believes it would be beneficial to include some ablation studies to provide more insights, such as ablations on adapters or training strategies.

Author response:
Q5: **More ablation results**.

(1) Training strategies：

It is difficult to directly optimize the entire network, thus we utilize the widely used training strategy that divides the optimization process into several stages. We can see from the experiments below that the 3-stage training yields improvement significantly.

|                                                          | C@0.25 | B-4@0.25 | M@0.25 | R@0.25 | C@0.5 | B-4@0.5 | M@0.5 | R@0.5 |
| -------------------------------------------------------- | ------ | -------- | ------ | ------ | ----- | ------- | ----- | ----- |
| One stage:Train all                                | 94.2   | 44.2     | 35.9   | 70.6   | 94.5  | 44.4    | 36.0  | 70.8  |
| Two stages:Train Detector+Train Decoder            | 152.4  | 50.9     | 38.2   | 72.9   | 153.8 | 51.0    | 38.4  | 73.1  |
| Three stages:Train Detector+Train Decoder+Train all | 158.7  | 51.2     | 38.4   | 73.1   | 160.2 | 51.4    | 38.5  | 73.3  |

(2) Different language decoders:

We conduct experiments with different language decoders and we observe that LLAMA achieves the best performance.

|        | C@0.25 | B-4@0.25 | M@0.25 | R@0.25 | C@0.5  | B-4@0.5 | M@0.5 | R@0.5 |
| ------ | ------ | -------- | ------ | ------ | ------ | ------- | ----- | ----- |
| S&T[1] | 116.3  | 45.2     | 38.0   | 70.9   | 116.5  | 45.4    | 38.3  | 71.0  |
| GPT2   | 137.0  | 47.7     | 37.8   | 71.1   | 5137.2 | 47.8    | 38.1  | 71.2  |
| LLAMA  | 158.7  | 51.2     | 38.4   | 73.1   | 160.2  | 51.4    | 38.5  | 73.3  |

[1] Show and Tell: A Neural Image Caption Generator

Reviewer comment (Presentation):
Section 2.3 ("Related work: Learning-based / End-to-end Autonomous Driving") does not seem very relevant to the main contents of the paper, and the reviewer suggests removing this part.

Author response:
Q6 Q7 Q8: **Some mistakes.** We thank the reviewer for this suggestion. 6. We have removed the end-to-end AD part in related work. 7. We have fixed this mistake. 8. We have included these three formulas into the paragraph.

Reviewer comment (Presentation):
The input modality information in Table 1 may contain inaccuracies. For example, Talk2Car/Refer-KITTI builds on top of nuScenes/KITTI and thus can also take LiDAR as input.

Author response:
Q6 Q7 Q8: **Some mistakes.** We thank the reviewer for this suggestion. 6. We have removed the end-to-end AD part in related work. 7. We have fixed this mistake. 8. We have included these three formulas into the paragraph.

Reviewer comment (Reproducibility):
Are the object proposals boxes (N x 9) or features (N x C)?

Author response:
Q9: Yes. The size of object proposals boxes is N\*9 and the size of features is N\*C.

Reviewer comment (Evaluation):
My main concern is that the paper does not clearly explain the effectiveness of DESIGN for downstream tasks. The reviewer believes that this should be the main focus of the paper, as mentioned in the last sentence of the first paragraph of the introduction. However, most of the evaluations focus on 2D/3D dense captioning, rather than the original motivation of enabling smarter and safer autonomous vehicles. I recommend that the authors focus on demonstrating that the proposed nuDESIGN is a "new addition" for 2D/3D dense captioning in traffic scenes.

Author response:
We would like to remark on the effectiveness of the DESIGN middleware from three perspectives. **Firstly**, as demonstrated by the end-to-end driving experiment in Table.4, DESIGN can significantly decrease the collision rate by a relative margin of 25.8% (from 0.31 to 0.23), while achieving comparable L2 (from 1.03 to 1.05). We argue that collision rate is a more meaningful indicator than L2 for autonomous driving, so this result is significant enough to demonstrate the effectiveness of DESIGN. **Secondly**, we hypothesize (with all due respect) that a5o2 may feel the UniAD experiment does not contain enough datapoints. In this regard, we provide another datapoint in Table.4 that directly turns the exact real numbers into natural language descriptions. This representation underperforms DESIGN. This datapoint demonstrates that the performance gain brought by DESIGN is non-trivial and serves as another evidence of DESIGN's effectiveness. **Thirdly**, we evaluate in another setting that shows DESIGN can also improve detection performance. The results below show that the DESIGN-former outperforms BEVFormer[1] in mAP and NDS, suggesting that the cues from language helps to regularize the middle representation of BEV features and improve the performance of the bounding box detection. This serves as another evidence that DESIGN has some superiority over an existing middleware that is a subset of DESIGN (i.e., 3D bounding boxes).

|                     | mAP      | NDS      |
| ------------------- | -------- | -------- |
| BEVFormer-tiny      | 25.2     | 35.4     |
| DESIGN-former-tiny  | **27.6** | **39.4** |
| BEVFormer-small     | 37.0     | 47.9     |
| DESIGN-former-small | **37.6** | **48.7** |
| BEVFormer-base      | 41.6     | 51.7     |
| DESIGN-former-base  | **41.8** | **51.8** |

Reviewer comment (Experiments):
The experiment on the end-to-end driving model is not convincing. While the authors show an approach to link the language model with the end-to-end model, the results do not explicitly demonstrate that the learned representations enable human-like reasoning. I suggest that the authors consider scenarios studied in ReasonNet (Shao et al., "ReasonNet: End-to-End Driving with Temporal and Global Reasoning, CVPR 2023"), which require explicit modeling of socially semantic spatial-temporal relationships.

Author response:
We would like to respond to this concern from three perspectives: (1) The ReasonNet paper has been cited and discussed in an updated version, but annotating the ReasonNet dataset with DESIGN is not feasible in the rebuttal window. (2) Our UniAD experiment (Table.4) has a module that requires interaction with GPT-3.5, which is demonstrated in Fig.14 and Fig.16. We hypothesize (with all due respect) that a5o2 feels the cases shown in these two figures are not human-like enough, so we provide two more cases into Fig.18.

Reviewer comment (Evaluation):
The smoother trajectory shown in Figure 15 is also not convincing, considering the significant effort in creating a dataset and large language models. The results could be generated through post-processing of the generated trajectories. Is it necessary to include the proposed method? I suggest that the authors consider scenarios that are challenging for existing methods.

Author response:
We thank the reviewer for this suggestion. We provide more visualization results of our driving model in challenging scenarios in Fig.19. For example, in the last example of Fig.19, a white truck in ego lane blocks the way. The UniAD tries to merge to the right lane and collides with the silver car on the right lane, while our driving model stops to avoid the collision with right car.

Reviewer comment (Novelty):
Limitation of DESIGN. While the middleware captures a wide range of information, two critical aspects are missing: uncertain intents of an object and the ability to facilitate forecasting. These aspects have been extensively studied and found to be valuable in the field of autonomous driving. I would like to learn the authors' thoughts on these aspects.

Author response:
To clarify, the DESIGN representation contains the moving status of 3D bounding boxes. For example, this is a ground truth caption in Fig.12: A stroller in the front left of ego car about 11 meters away is moving slowly to ego car in the crosswalk.

Here 'moving slowly to ego car' covers the moving trend of the object. As for 'uncertain intents', we think the rough descriptions of 'slowly' or 'quickly' have intrinsically modeled the uncertainty.

Reviewer comment (Experiments):
There is a lack of comparisons with existing baselines for 3D dense captioning. For example:
    1. Chen et al., "End-to-End 3D Dense Captioning with Vote2Cap-DETR, CVPR 2023."
    2. Cai et al., "3djcg: A unified framework for joint dense captioning and visual grounding on 3d point clouds, CVPR 2022."
    3. Chen et al., "D3net: A speaker-listener architecture for semi-supervised dense captioning and visual grounding in rgb-d scans, ECCV 2022."
    
    Without a direct comparison, we cannot conclude whether the proposed method achieves state-of-the-art performance.

Author response:
|                      | Input | C@0.25 | B-4@0.25 | M@0.25 | R@0.25 | C@0.5 | B-4@0.5 | M@0.5 | R@0.5 |
| -------------------- | ----- | ------ | -------- | ------ | ------ | ----- | ------- | ----- | ----- |
| Vote2cap + MLE       | C+L   | 187.6  | 57.1     | 40.5   | 74.5   | 189.9 | 57.3    | 40.9  | 74.8  |
| Vote2cap with + SCST | C+L   | 198.3  | 58.0     | 41.9   | 75.6   | 200.1 | 58.4    | 42.1  | 76.1  |
| Ours                 | C+L   | 218.0  | 60.2     | 43.8   | 76.2   | 220.3 | 61.5    | 50.9  | 80.1  |

We thank a5o2 for pointing out these references and we have cited them in the updated paper. We compare with the newest SOTA baseline Vote2Cap-DETR in the figure below. There are two settings: the first one exploits a standard cross entropy loss (denoted as maximum likelihood estimation, MLE) and the second one exploits a self-critical sequence training (denoted as SCST) that optimizes towards a better CIDEr metric. As for 3DJCG and D3Net, since they are older baselines and involve a substantial modification to the dataloader for joint captioning and grounding, we have not yet evaluated in the rebuttal window.