Paper: P-MapNet: Far-seeing Map Constructer Enhanced by both SDMap and HDMap Priors (ICLR-2024)

Reviewer comment (Theory):
The paper emphasizes the limitation of relying on HDMaps for autonomous vehicles to operate outside regions with this infrastructure, yet it relies heavily on HDMap priors to refine outputs and address issues such as broken and unnecessarily curved results. Additionally, the approach of generating HDMap with prior information from HDMap seems counterintuitive and unreasonable.

Author response:
We actually didn't have the HDMap, we just pre-train the mask auto-encoding in the train split dataset and then evaluate in the val split dataset. Furthermore, to verify the generalizability of the HDMap Prior Module, we conducted cross-data experiments as detailed in **Appendix B.2**. In these experiments, we pre-trained the model on the train split of the Argoverse2 dataset and then evaluated it on the validation data of nuScenes.This approach also yielded an improvement in the mIoU.
Additionnally, a difference between our MAE and the conventional MAE may be the cause of this confusion. Specifically, a conventional MAE takes RGB image patches and directly regresses RGB image values of the whole image. Here, the input and output of our HDmap MAE are both rasterized segmentation masks. In our method, the output segmentation mask is supervised by a pixel-wise cross-entropy loss instead of the typical regression loss in a conventional MAE. As such our MAE can be naturally used for refinement during inference since the head is a segmentation head. To note, this segmentation mask MAE is exactly doing masked auto-encoding.

Reviewer comment (Experiments):
The paper falls short of providing the results under the setting of camera-only modality and combining both SDMap prior and HDMap prior modules. This omission raises concerns about the true impact of HDMap priors on the overall performance.

Author response:
We have updated the relevant experiments in **Tab.2**, and conducted relevant experiments in **Fig. 5** to obtain the visualization results and quantification results of the camera-only modality utilizing post-processing vectorization.

Reviewer comment (Experiments):
In terms of vectorization baseline results, the authors only reproduce HDMapNet under their new settings on the NuScenes dataset, without conducting a comparison with other state-of-the-art methods, both vectorized and rasterized-to-vectorized. This limits the thoroughness of the performance evaluation.

Author response:
We further demonstrate the effectiveness of SD map prior by incorporating it into the state-of-the-art vectorized map constructor MapTR. The results are presented in **Appendix A.1** and **Fig.5**. It shows that the SD map prior can also improve the performance of MapTR on various ranges. Since the MAE HD map modelling method does not apply to MapTR, it is not evaluated.

Meanwhile, we would like to point out that dense rasterized prediction on the BEV view still has advantages over direct vectorized map prediction methods. Vectorized representation is intrinsically troubled by control point selection ambiguity, which negatively impacts generalization in real-world autonomous driving applications. We believe in the future, rasterized map constructer and vectorized map constructor would co-exist in the academia and industry.

| Range               | Method     | Divider | PedCrossing | Boundary | mAP                            |
|---------------------|------------|---------|--------------|----------|--------------------------------|
|    | MapTR      | 12.69   | 7.17         | 4.23     | 8.03                           |
|  $240\times 60m$   | MapTR-SDMap | **22.74** |  16.34    | 10.53    | 16.53 (+8.50)            |
|                     | P-MapNet   | 14.51   | **25.63**        | **28.11** |  **22.75**                          |

Reviewer comment (Experiments):
Could you please explain the reasons for selecting rasterized representation and employing post-processing for vectorized results instead of directly using a vectorized network? In Section 2.1, you mentioned the limitations of methods relying solely on onboard sensors, but it is not clear how this relates to the chosen representation. Additionally, you mentioned that your network is designed in a BEV dense prediction manner and the structured output space of BEV HDMap cannot be guaranteed. Given these factors, why not use the vectorized representation directly, which naturally addresses the problem and eliminates the need for additional MAE pretraining methodology?

Author response:
We further demonstrate the effectiveness of SD map prior by incorporating it into the state-of-the-art vectorized map constructor MapTR. The results are presented in **Appendix A.1** and **Fig.5**. It shows that the SD map prior can also improve the performance of MapTR on various ranges. Since the MAE HD map modelling method does not apply to MapTR, it is not evaluated.

Meanwhile, we would like to point out that dense rasterized prediction on the BEV view still has advantages over direct vectorized map prediction methods. Vectorized representation is intrinsically troubled by control point selection ambiguity, which negatively impacts generalization in real-world autonomous driving applications. We believe in the future, rasterized map constructer and vectorized map constructor would co-exist in the academia and industry.

| Range               | Method     | Divider | PedCrossing | Boundary | mAP                            |
|---------------------|------------|---------|--------------|----------|--------------------------------|
|    | MapTR      | 12.69   | 7.17         | 4.23     | 8.03                           |
|  $240\times 60m$   | MapTR-SDMap | **22.74** |  16.34    | 10.53    | 16.53 (+8.50)            |
|                     | P-MapNet   | 14.51   | **25.63**        | **28.11** |  **22.75**                          |

Reviewer comment (Experiments):
Can you report the results under the camera-only modality and when both SDMap prior and HDMap prior modules are combined, to accurately reflect the genuine influence of HDMap priors?

Author response:
We have updated the relevant experiments in **Tab.2**, and conducted relevant experiments in **Fig. 5** to obtain the visualization results and quantification results of the camera-only modality utilizing post-processing vectorization.

Reviewer comment (Evaluation):
In relation to Table 2, could you explain the significant decrease in frames per second (FPS) and how this might impact downstream or practical applications?

Author response:
Regarding the issue of model runtime, we conducted further experiments to break down the inference time and discovered that the refinement module consumes a significant amount of time. In fact, the HDMap Prior Module is an optional component and can be chosen based on specific requirements. The detailed experimental results are presented in **Table 13**.
| Component     | Runtime (ms)     | Proportion | 
|---------------------|------------|---------|
|  Image backbone  | 7.56      | 7.63%   | 
|  View transformation | 3.25 | 3.28% | 
|  Lidar backbone | 17.60  |17.76%  |
|SDMap prior module|4.40|4.45%|
|HDMap prior module|66.12|66.87%|
|Total |98.87| 100%|

Reviewer comment (Experiments):
Your SDMap Prior Module aligns misaligned SDMap with BEV features using multi-head cross-attention. However, what if there are localization errors present in the BEV features, which is a common occurrence in both camera-only and camera+lidar models?

Author response:
However, localization errors only influence the SDMap obtained from the global SDMAP, the BEV feature is online generated from the onboard sensors.

Reviewer comment (Writing):
In your summary of contributions, "artefacts" should be corrected to "artifacts."

Author response:
We have changed the spelling of the word.

Reviewer comment (Novelty):
The idea of building long-distance HD maps has been explored in previous works [4,5], the authors should clearly state the difference and the superiority of the proposed framework.

Author response:
Our approach fundamentally differs from the method used in Neural Map Prior[1]. In [1], a Neural Map must be pre-constructed to enhance the online generation of HDMaps, which is then updated as the vehicle repeatedly traverses the same area, aiming to obtain a more accurate HDMap. Additionally, the furthest distance covered in their paper is 160m by 100m.
Conversely, our method, based on the SDMap Prior Module, predicts HDMaps at ultra-long distances online, without the need for a prior traversal of the area. Furthermore, the maximum distance in our results extends to 240m by 60m.

Reviewer comment (Evaluation):
The proposed approach involves a large computation burden and has lower inference speeds.

Author response:
Regarding the issue of model runtime, we conducted further experiments to break down the inference time and discovered that the refinement module consumes a significant amount of time. In fact, the HDMap Prior Module is an optional component and can be chosen based on specific requirements. The detailed experimental results are presented in **Table 13**.
| Component     | Runtime (ms)     | Proportion | 
|---------------------|------------|---------|
|  Image backbone  | 7.56      | 7.63%   | 
|  View transformation | 3.25 | 3.28% | 
|  Lidar backbone | 17.60  |17.76%  |
|SDMap prior module|4.40|4.45%|
|HDMap prior module|66.12|66.87%|
|Total |98.87| 100%|

Reviewer comment (Experiments):
The authors evaluate the proposed framework on the benchmark with the max range of 240x60 while I'm concerned about the superiority of the proposed framework compared to the methods trained with the same range. In addition, I'm concerned about whether the proposed method has a limited range, and what the range is when SD maps do not work.

Author response:
We further demonstrate the effectiveness of SD map prior by incorporating it into the state-of-the-art vectorized map constructor MapTR. The results are presented in **Appendix A.1** and **Fig.5**. It shows that the SD map prior can also improve the performance of MapTR on various ranges. Since the MAE HD map modelling method does not apply to MapTR, it is not evaluated.

Meanwhile, we would like to point out that dense rasterized prediction on the BEV view still has advantages over direct vectorized map prediction methods. Vectorized representation is intrinsically troubled by control point selection ambiguity, which negatively impacts generalization in real-world autonomous driving applications. We believe in the future, rasterized map constructer and vectorized map constructor would co-exist in the academia and industry.

Regarding the effective distance of SDMap, due to GPU memory constraints, we only supported testing up to 240 meters. Theoretically, SDMap has priors at all distances, and in ranges beyond sensor perception, SDMap alone provides valuable information. The model could potentially guess the form of the HDMap based on SDMap, although the generated HDMap may not be very accurate due to misalignment issues.
| Range               | Method     | Divider | PedCrossing | Boundary | mAP                            |
|---------------------|------------|---------|--------------|----------|--------------------------------|
|    | MapTR      | 12.69   | 7.17         | 4.23     | 8.03                           |
|  $240\times 60m$   | MapTR-SDMap | **22.74** |  16.34    | 10.53    | 16.53 (+8.50)            |
|                     | P-MapNet   | 14.51   | **25.63**        | **28.11** |  **22.75**                          |

Reviewer comment (Experiments):
The experiments about the downsampling factors of the SD map.

Author response:
We have completed relevant experiments, see Appendix A.4 for details. SDMap downsampling factors are the same as factor **d**.

Reviewer comment (Experiments):
Experimental results about P-MapNet(S+H) without lidar.

Author response:
We have updated relevant experiments in **Tab.2**

Reviewer comment (Reproducibility):
In Sec.4.3, it's unclear how the initial maps are used in the mask image modelling and how the maps are refined during inference.

Author response:
A difference between our MAE and the conventional MAE may be the cause of this confusion. Specifically, a conventional MAE takes RGB image patches and directly regresses RGB image values of the whole image. Here, the input and output of our HDmap MAE are both rasterized segmentation masks. In our method, the output segmentation mask is supervised by a pixel-wise cross-entropy loss instead of the typical regression loss in a conventional MAE. As such our MAE can be naturally used for refinement during inference since the head is a segmentation head. To note, this segmentation mask MAE is exactly doing masked auto-encoding.

We have further refined the description and ablation experiments of the HDMap Prior Module, as detailed in **Section 4.3** and **Appendix B**.

Reviewer comment (Experiments):
I'm concerned about whether the proposed framework can be applied to vectorized methods, such as MapTR[1], though this paper tries to vectorize the map through post-processing.

Author response:
We further demonstrate the effectiveness of SD map prior by incorporating it into the state-of-the-art vectorized map constructor MapTR. The results are presented in **Appendix A.1** and **Fig.5**. It shows that the SD map prior can also improve the performance of MapTR on various ranges. Since the MAE HD map modelling method does not apply to MapTR, it is not evaluated.

Meanwhile, we would like to point out that dense rasterized prediction on the BEV view still has advantages over direct vectorized map prediction methods. Vectorized representation is intrinsically troubled by control point selection ambiguity, which negatively impacts generalization in real-world autonomous driving applications. We believe in the future, rasterized map constructer and vectorized map constructor would co-exist in the academia and industry.

Regarding the effective distance of SDMap, due to GPU memory constraints, we only supported testing up to 240 meters. Theoretically, SDMap has priors at all distances, and in ranges beyond sensor perception, SDMap alone provides valuable information. The model could potentially guess the form of the HDMap based on SDMap, although the generated HDMap may not be very accurate due to misalignment issues.
| Range               | Method     | Divider | PedCrossing | Boundary | mAP                            |
|---------------------|------------|---------|--------------|----------|--------------------------------|
|    | MapTR      | 12.69   | 7.17         | 4.23     | 8.03                           |
|  $240\times 60m$   | MapTR-SDMap | **22.74** |  16.34    | 10.53    | 16.53 (+8.50)            |
|                     | P-MapNet   | 14.51   | **25.63**        | **28.11** |  **22.75**                          |

Reviewer comment (Experiments):
If a vectorization modeling approach is used, there might not be such discontinuities in results. This article should conduct further experiments to validate this matter.

Author response:
We further demonstrate the effectiveness of SD map prior by incorporating it into the state-of-the-art vectorized map constructor MapTR. The results are presented in **Appendix A.1** and **Fig.5**. It shows that the SD map prior can also improve the performance of MapTR on various ranges. Since the MAE HD map modelling method does not apply to MapTR, it is not evaluated.

Meanwhile, we would like to point out that dense rasterized prediction on the BEV view still has advantages over direct vectorized map prediction methods. Vectorized representation is intrinsically troubled by control point selection ambiguity, which negatively impacts generalization in real-world autonomous driving applications. We believe in the future, rasterized map constructer and vectorized map constructor would co-exist in the academia and industry.
| Range               | Method     | Divider | PedCrossing | Boundary | mAP                            |
|---------------------|------------|---------|--------------|----------|--------------------------------|
|    | MapTR      | 12.69   | 7.17         | 4.23     | 8.03                           |
|  $240\times 60m$   | MapTR-SDMap | **22.74** |  16.34    | 10.53    | 16.53 (+8.50)            |
|                     | P-MapNet   | 14.51   | **25.63**        | **28.11** |  **22.75**                          |

Reviewer comment (Experiments):
The metrics of vectorization results should be compared with vectorized modeling methods.

Author response:
We further demonstrate the effectiveness of SD map prior by incorporating it into the state-of-the-art vectorized map constructor MapTR. The results are presented in **Appendix A.1** and **Fig.5**. It shows that the SD map prior can also improve the performance of MapTR on various ranges. Since the MAE HD map modelling method does not apply to MapTR, it is not evaluated.

Meanwhile, we would like to point out that dense rasterized prediction on the BEV view still has advantages over direct vectorized map prediction methods. Vectorized representation is intrinsically troubled by control point selection ambiguity, which negatively impacts generalization in real-world autonomous driving applications. We believe in the future, rasterized map constructer and vectorized map constructor would co-exist in the academia and industry.
| Range               | Method     | Divider | PedCrossing | Boundary | mAP                            |
|---------------------|------------|---------|--------------|----------|--------------------------------|
|    | MapTR      | 12.69   | 7.17         | 4.23     | 8.03                           |
|  $240\times 60m$   | MapTR-SDMap | **22.74** |  16.34    | 10.53    | 16.53 (+8.50)            |
|                     | P-MapNet   | 14.51   | **25.63**        | **28.11** |  **22.75**                          |

Reviewer comment (Experiments):
The mask proportion of MAE should undergo some ablation experiments.

Author response:
We have added relevant ablation experiments. Specifically, in **Appendix B.5**, we conducted 25%, 50%, and 75% experiments. Too high a mask ratio will lead to the lack of valid information and the actual refinement process of the input difference is large, too low a mask ratio can not force the network to capture the HDMap priors。
| Mask Proportion  | Divider | PedCrossing | Boundary | mAP                            |
|--------------|---------|--------------|----------|--------------------------------|
|    25%  | 64.8   | 51.4        |67.6     | 61.27                           |
|    50% | **65.3** | 52.0    | **68.0**  | **61.77**             |
|    75%   | 64.7   | **52.1**        | 67.7 | 61.50                         |

Reviewer comment (Evaluation):
The benefits brought by the prior information of HDMap are too small.

Author response:
This issue is also present in the work of MapPrior[1]. Essentially, our goal is to map the output of the SDMap Prior Module to the space of HDMaps. However, this process might not map onto the ground truth with high precision, hence the modest improvement in the mean Intersection over Union (mIoU) metric, which measures accuracy.In order to evaluate the realism after refining, we use a type of Perceptual Metrics, specifically LPIPS [2], to do so, and the results show that our HDMap Prior module has a significant improvement on predicting after refining than without refining. As detailed in **Appendix B.1**.
| Method          | mIoU↑  | LPIPS↓ |  
|-----------------|--------|--------|
| Baseline        | 49.07  | 0.7872 |   
| P-MapNet (S)    | 60.20  | 0.7607 |   
| P-MapNet (S+H)  | **61.77**  | **0.7124**|

Reviewer comment (Evaluation):
There are some data inaccuracies in mIoU in Table 2.

Author response:
It may be a problem with our table layout. You may have misread the results of only cam and fusion model. We have revised the table **Tab.2**.

Reviewer comment (Experiments):
What data should be used to train this MAE-based ViT? And would the model overfit if all the data is utilized?

Author response:
Initially, we pre-trained our model using the train split of the nuScenes dataset, followed by evaluations on the validation split. This approach might have led to some overfitting. To verify the generalizability of the HDMap Prior Module, we conducted cross-data experiments as detailed in **Appendix B.2**. In these experiments, we pre-trained the model on the train split of the Argoverse2 dataset and then evaluated it on the validation data of nuScenes.This approach also yielded an improvement in the mIoU.

Reviewer comment (Experiments):
The benchmark is not compared reasonable. Comparison with recent advance works is needed.

Author response:
Dear reviewer, thank you for the professional feedbacks. We have updated the manuscript and marked the changes in red. Here are my answers to your related questions.
## Weaknesses.1 & Questions.1:

We further demonstrate the effectiveness of SD map prior by incorporating it into the state-of-the-art vectorized map constructor MapTR. The results are presented in **Appendix A.1** and **Fig.5**. It shows that the SD map prior can also improve the performance of MapTR on various ranges. Since the MAE HD map modelling method does not apply to MapTR, it is not evaluated.

Meanwhile, we would like to point out that dense rasterized prediction on the BEV view still has advantages over direct vectorized map prediction methods. Vectorized representation is intrinsically troubled by control point selection ambiguity, which negatively impacts generalization in real-world autonomous driving applications. We believe in the future, rasterized map constructer and vectorized map constructor would co-exist in the academia and industry.
| Range               | Method     | Divider | PedCrossing | Boundary | mAP                            |
|---------------------|------------|---------|--------------|----------|--------------------------------|
|    | MapTR      | 12.69   | 7.17         | 4.23     | 8.03                           |
|  $240\times 60m$   | MapTR-SDMap | **22.74** |  16.34    | 10.53    | 16.53 (+8.50)            |
|                     | P-MapNet   | 14.51   | **25.63**        | **28.11** |  **22.75**                          |

Reviewer comment (Writing):
Some parts of the proposed method is not clarified clearly, such as the HDMap Refinement Module.

Author response:
## Weaknesses.2 & Questions.2:
A difference between our MAE and the conventional MAE may be the cause of this confusion. Specifically, a conventional MAE takes RGB image patches and directly regresses RGB image values of the whole image. Here, the input and output of our HDmap MAE are both rasterized segmentation masks. In our method, the output segmentation mask is supervised by a pixel-wise cross-entropy loss instead of the typical regression loss in a conventional MAE. As such our MAE can be naturally used for refinement during inference since the head is a segmentation head. To note, this segmentation mask MAE is exactly doing masked auto-encoding.

We have further refined the description and ablation experiments of the HDMap Prior Module, as detailed in **Section 4.3** and **Appendix B**.

Reviewer comment (Experiments):
Utilizing pretrained MAE as second-stage refinement is interesting. However, its generalization as a pretrained model is more worthy of exploration.

Author response:
## Weaknesses.4 & Questions.4:
Indeed, we have also considered this aspect and supplemented our method with a cross-dataset experiment, detailed in **Appendix B.2** and **Table 11**. We initially pre-trained the HDMap prior module on the train split of the Argoverse 2 dataset and then tested it on nuScenes val dataset. This approach also yielded an improvement in the mIoU.

Reviewer comment (Experiments):
It is reasonable to compare with recent works with advance performance (e.g. [1], [2]).

Author response:
Dear reviewer, thank you for the professional feedbacks. We have updated the manuscript and marked the changes in red. Here are my answers to your related questions.
## Weaknesses.1 & Questions.1:

We further demonstrate the effectiveness of SD map prior by incorporating it into the state-of-the-art vectorized map constructor MapTR. The results are presented in **Appendix A.1** and **Fig.5**. It shows that the SD map prior can also improve the performance of MapTR on various ranges. Since the MAE HD map modelling method does not apply to MapTR, it is not evaluated.

Meanwhile, we would like to point out that dense rasterized prediction on the BEV view still has advantages over direct vectorized map prediction methods. Vectorized representation is intrinsically troubled by control point selection ambiguity, which negatively impacts generalization in real-world autonomous driving applications. We believe in the future, rasterized map constructer and vectorized map constructor would co-exist in the academia and industry.
| Range               | Method     | Divider | PedCrossing | Boundary | mAP                            |
|---------------------|------------|---------|--------------|----------|--------------------------------|
|    | MapTR      | 12.69   | 7.17         | 4.23     | 8.03                           |
|  $240\times 60m$   | MapTR-SDMap | **22.74** |  16.34    | 10.53    | 16.53 (+8.50)            |
|                     | P-MapNet   | 14.51   | **25.63**        | **28.11** |  **22.75**                          |

Reviewer comment (Writing):
Some parts of the proposed method is not clarified clearly, such as the HDMap Refinement Module. Please introduce more details about it. How can it refine the initial predictions with absent sidewalks and broken lane lines? Are there any insights?

Author response:
## Weaknesses.2 & Questions.2:
A difference between our MAE and the conventional MAE may be the cause of this confusion. Specifically, a conventional MAE takes RGB image patches and directly regresses RGB image values of the whole image. Here, the input and output of our HDmap MAE are both rasterized segmentation masks. In our method, the output segmentation mask is supervised by a pixel-wise cross-entropy loss instead of the typical regression loss in a conventional MAE. As such our MAE can be naturally used for refinement during inference since the head is a segmentation head. To note, this segmentation mask MAE is exactly doing masked auto-encoding.

We have further refined the description and ablation experiments of the HDMap Prior Module, as detailed in **Section 4.3** and **Appendix B**.

Reviewer comment (Evaluation):
The performance of run time seems not competitive. Please explain about that.

Author response:
## Weaknesses.3 & Questions.3:
Regarding the issue of model runtime, we conducted further experiments to break down the inference time and discovered that the refinement module consumes a significant amount of time. In fact, the HDMap Prior Module is an optional component and can be chosen based on specific requirements. The detailed experimental results are presented in **Table 13**.
| Component     | Runtime (ms)     | Proportion | 
|---------------------|------------|---------|
|  Image backbone  | 7.56      | 7.63%   | 
|  View transformation | 3.25 | 3.28% | 
|  Lidar backbone | 17.60  |17.76%  |
|SDMap prior module|4.40|4.45%|
|HDMap prior module|66.12|66.87%|
|Total |98.87| 100%|

Reviewer comment (Experiments):
Utilizing pretrained MAE as second-stage refinement is interesting. However, its generalization as a pretrained model is more worthy of exploration. The reviewer wonder how it works when it come to other dataset, e.g., pretrained on nuscenes while inferenced on Ago.

Author response:
## Weaknesses.4 & Questions.4:
Indeed, we have also considered this aspect and supplemented our method with a cross-dataset experiment, detailed in **Appendix B.2** and **Table 11**. We initially pre-trained the HDMap prior module on the train split of the Argoverse 2 dataset and then tested it on nuScenes val dataset. This approach also yielded an improvement in the mIoU.