Paper: Radar Spectra-language Model for Automotive Scene Parsing (ICLR-2024)

Reviewer comment (Experiments):
Please consider comparing your proposed model against some existing adapted language models for driving scene understanding.

Author response:
A review of VLMs for automotive scene understanding was added to the related work section (edited part highlighted in blue). All methods are quite new - concurrent to our work. None of the reviewed methods is quite applicable to our setting of retrieval – where data is scarce and semantic ground truth is absent (and off-the-shelf VLM performance is unsatisfactory).

Reviewer comment (Novelty):
Most of the components from the proposed framework are from existing works. It is hard to find any novel technical designs in the presented framework. Please better clarify the technical novelty and theoretical contributions of the presented work.

Author response:
Our paper introduces a novel task of extraction of semantic information from radar spectra – an otherwise generally non human - interpretable modality, and introduces an approach to tackle it. Both of the above are the main novelties of the paper. It is fortunate, that the encoder of an existing object detection and segmentation network (FFT-RadNet Rebut et al. (2022)) can be used as radar encoder of the radar spectra language model. This way, not another specific architecture has to be designed for one specific use case. 

In addition, we describe a novel process of generating captions for and fine-tuning an image-language model to extract general semantic information from AD scenes.  

Finally, to validate our radar-spectra model we modify an object detection model acting on spectra by  injecting the embedding vector from our spectra encoder, and demonstrate it improves SotA on the RADIal dataset. 

The goal of the work is to explore the semantic content of radar spectra, which is otherwise very difficult.  

In the updated paper version, we explain our contributinos in a clearer manner (highlighted by blue text color). We will try to clarify this further in the final version.

Reviewer comment (Presentation):
Would you consider giving an overview of your proposed framework at the beginning of the methodology section, which can help the readers better understand the work?

Author response:
Done. Overview provided at the begining of the "Proposed Approach" Section 3.

Reviewer comment (Evaluation):
The computation efficiency should be discussed, which is critical in automotive scene understanding.

Author response:
We agree that computational efficiency is an important property for models running on an embedded device. Nevertheless, the main focus of the paper is to a) get a better understanding of the content of radar spectra and b) to investigate the benefits of the learnt features for downstream tasks. The focus is not on computational efficiency and the method is not intended to be run on an edge device, but rather is a research tool.  

In other words, the current work aims to provide a tool to examine the semantic content of radar spectra, rather than provide a perception algorithm for an autonomous vehicle. For this reason, computational efficiency is not a major consideration.

Reviewer comment (Evaluation):
Would you consider presenting some visualization results of object detection and free space segmentation to qualitatively verify the effectiveness of your proposed method?

Author response:
We added a new appendix section A.6 with visualization of object detection and segmentation, and a brief discussion.

Reviewer comment (Novelty):
While there are not many works on radar-language models, there are extensive works on adapting large-language models for driving scene understanding. Would you consider discussing the relations and differences between your work and existing works in the related work section?

Author response:
Done, added a review of VLMs for automotive scene understanding. Most works are very new, indeed concurrent to ours. None takes our approach, contrary to us relying on annotated data or using a frozen VLM. (And of course none does a radar spectra language model).

Reviewer comment (Presentation):
The author seems to lack paper writing skills. All the figures are unaesthetic bitmaps with low resolution and some of the figures are not necessary. For Figure 4a, it is better to use formulation instead of python code to describe the loss functions. For Figure 4b, such a simple architecture may be put in the supplement material.

Author response:
Figures were re-exported in vector format.  

Figure 4 has been moved to the appendix and re-numbered to Figure 6. Formulation of loss functions was added in Section A.2.  

The reason we used python code was to be consistent with the CLIP paper Radford et al. 2021, which defined its InfoNCE loss with python code.

Reviewer comment (Experiments):
Changing the position encoding without finetuning may cause performance drop, and splitting the image may break some objects on the edge. A better and more common way is to pad black pixels to the top and bottom of the image.

Author response:
We always fine-tuned the model when using each of the adaptation methods we reported ( (1) positional encoding interpolation, (2) multiple square crops). 

When creating crops we made sure there's an overlap, hopefully mitigating effects like mentioned - of breaking an object on the edge, i.e. an object would usually appear in its entirety at least in one of the crops.  

Following this comment we tried using zero-padding, which worsened results (average weighted accuracy in the experiment did not exceed 0.5).  

We believe that the reason is that images in the RADIal dataset have aspect ratio of nearly 1:2, while the network input is 244x244 square. In this case (resize and) padding results in half the input image being black, with effective image height just 122 pixels, and many details are lost – resulting in the observed performance hit.

Reviewer comment (Evaluation):
For the detection and segmentation downstream tasks, it is better to show some cases that the pretrained model helps improve the performance, not just numbers.

Author response:
We added a new appendix section A.6 with visualization of object detection and segmentation, and a brief discussion.

Reviewer comment (Evaluation):
For autonomous driving tasks, the localization ability is more important than the classification. Could you provide some visualizations such as attention map or GradCAM to see if the retrieved objects are corresponding to the right location?

Author response:
We added a new appendix section A.4 with visualization of VLM activations using GradCAM before and after fine-tunning.

Reviewer comment (Writing):
For Equation 4, a period should be added in the end of the formula.

Author response:
All done. Thanks for pointing out!

Reviewer comment (Writing):
All the quotation marks are single quotes. Please use backquote in front of the quoted phrases.

Author response:
All done. Thanks for pointing out!

Reviewer comment (Writing):
Some of the RADIal is mistaken by RADiaL.

Author response:
All done. Thanks for pointing out!

Reviewer comment (Writing):
Overall, the proposed model is novel and the zero-shot results are interesting and impressive. However, the writing and lack of discussion lowered the final score. I would like to increase my score if the authors would polish the paper and redraw all the figures using vectorgraph.

Author response:
We thank the reviewer for review and useful suggestions!

We uploaded a final version with updated - hopefully, slightly improved - figures.

Reviewer comment (Evaluation):
No discussion on what is still hard to do or not reliable.

Author response:
> R2:W1 No discussion on what is still hard to do or not reliable. Also analysis of the varying the difficulty of the input scenes would help answer the previous question. 

A discussion section was incorporated (page 10). The changes in the updated paper version are highlighted with blue text color.

Reviewer comment (Evaluation):
Analysis of the varying the difficulty of the input scenes would help answer the previous question.

Author response:
> R2:W1 No discussion on what is still hard to do or not reliable. Also analysis of the varying the difficulty of the input scenes would help answer the previous question. 

A discussion section was incorporated (page 10). The changes in the updated paper version are highlighted with blue text color.

Reviewer comment (Novelty):
What self-driving related take would Radar-spectra be able to do well, while other modality cannot or struggle with?

Author response:
> R2:Q1 What self-driving related take would Radar-spectra be able to do well, while other modality cannot or struggle with? 

Radar has several advantages compared to other modalities, e.g., it is robust to difficult weather conditions like rain or sun glare, and it can directly measure the relative radial velocity of other objects. Moreover it can detect objects at large distances, depending on the sensor for example up to about 350m.  Nevertheless, the focus of the paper is not a comparison of radar to other modalities. We investigate the semantic content of radar spectra, which is otherwise difficult to interpret, unlike other modalities.

Reviewer comment (Writing):
Pg. 3, "Prompt Generation" section: (e.g. a photo of a {}) -- {} symbol should be replaced.

Author response:
> R2:Q2 Typos: Pg. 3, "Prompt Generation" section: (e.g. a photo of a {}) -- {} symbol should be replaced. 

Done! Thank you for pointing this out!

Reviewer comment (Novelty):
The framework seems to be a simple combination of existing methods. I didn’t see the specific design for the radar spectra language model.

Author response:
Our paper introduces a novel task of extraction of semantic information from radar spectra – an otherwise generally non human - interpretable modality, and introduces an approach to tackle it. Both of the above are the main novelties of the paper.  

In addition, we describe a novel process of generating captions for and fine-tuning an image-language model to extract general semantic information from AD scenes.  

Finally, to validate our radar-spectra model we modify an object detection model acting on spectra by  injecting the embedding vector from our spectra encoder, and demonstrate it improves SotA on the RADIal dataset.

Reviewer comment (Experiments):
The experiment of detection is not compared with SOTA methods such as RODNet.

Author response:
Thank you for pointing this out! The reference in table 3 is the FFT-RadNet model from Rebut et al. (2022) - it has been fixed in the updated paper version. FFT-RadNet is the SOTA model for the RADIal dataset, to which we compare our proposed methods. We consider the RADIal dataset for the evaluation of the proposed radar spectra language model. Unfortunatelly, RODNet cannot be applied to this dataset directly, since RODNet is designed to work on a sequence of complex valued range-azimuth-spectra, the data provided in the CRUW dataset. In contrast, the RADIal dataset contains range-Doppler spectra, where the azimuth dimension has not been processed yet.  

Following this comment we've added results of two more SotA methods for RADIal to the comparison. We've extended Sec. 4.3, in particular the ``Results" paragraph to accomodate for the changes. Modified portions of the manuscript are highlighted in blue.

Reviewer comment (Presentation):
What is [20] in Table 3?

Author response:
Thank you for pointing this out! The reference in table 3 is the FFT-RadNet model from Rebut et al. (2022) - it has been fixed in the updated paper version. FFT-RadNet is the SOTA model for the RADIal dataset, to which we compare our proposed methods. We consider the RADIal dataset for the evaluation of the proposed radar spectra language model. Unfortunatelly, RODNet cannot be applied to this dataset directly, since RODNet is designed to work on a sequence of complex valued range-azimuth-spectra, the data provided in the CRUW dataset. In contrast, the RADIal dataset contains range-Doppler spectra, where the azimuth dimension has not been processed yet.  

Following this comment we've added results of two more SotA methods for RADIal to the comparison. We've extended Sec. 4.3, in particular the ``Results" paragraph to accomodate for the changes. Modified portions of the manuscript are highlighted in blue.

Reviewer comment (Reproducibility):
If the description includes multiple object information, how do you align the text information with the corresponding object?

Author response:
In this work, we formulate queries as descriptions on scene level, we do not align text information with a specific object explicitly. When generating captions, we have the entire information of the scene. In conclusion – the alignment problem does not arise, at least for the setting described in the paper.

Reviewer comment (Evaluation):
If the description is not accurate will the information mislead the model?

Author response:
In general, yes -  imprecise descriptions may have a negative effect of training. For example, using inaccurate descriptions for highway scenes, e.g. "there are 3 cars in the street" (as opposed to "on the highway"), decreased the quality of the scene classification. That's why we don't use GPT augmentations of captions, as it sometimes adds information that is not relevant to the image like "it's a sunny day, car goes down the street" for a picture with cloudy weather. Meanwhile, we added augmentations that are supposed not to violate caption-image aligning, e.g. "it's a photo of the environment with a car" and "there is a car".

Just to be sure we would like to clarify that the generated captions are not completely random: The captions generated for a frame are based on the semantic segmentation of that frame, so the captions include the actual contents of the frame (portions of the captions are randomly dropped for the sake of data augmentation - this is why the captions are "random", but the remainder still reflects the contents of the particular frame).
 
The caption for a frame might be inaccurate only if its semantic segmentation is erroneous (in terms of frame contents, not in terms of object boundaries) - however, note that even noisy captions might not mislead the model, especially if the dataset is big and diverse enough: large caption data scrapped from the internet that large VLMs are trained on is also noisy. Nevertheless, VLMs show impressive performance. One reason might be that soft cost function / training process (contrastive learning) can deal with noisy labels.
 
In our answer to Q1 above we purely hypothetically refer to systematic errors in the captions that effect a large portion of the data - for example, replacing "highway" by "street" in say 30% of the cases. In contrast, the process we use for generating captions generally produces nearly perfect results on the level of scene description, i.e. it rarely misses completely scene content or makes up scene content in practice.
 
The process of caption generation is described in section 3.2 and Figure 4 contains examples for generated captions. They may not describe the entirety of the scene contents, but are generally correct, and therefore helpful for contrastive training.

We adapted the caption generation explanation in section 3.2 to make it clearer and reflect the above - we uploaded an updated version, changes marked in blue.