Paper: LaneSegNet: Map Learning with Lane Segment Perception for Autonomous Driving (ICLR-2024)

Reviewer comment (Experiments):
The definition of "long range" is not clearly defined in the paper. The authors mention that the OpenLaneV2 dataset is reannotated using the proposed lane segment manner, resulting in lanes being broken into multiple segments. By using shorter lane segments, I don't think "long range" is challange.

Author response:
Thanks for your question. We have analyzed the data of lane segments and the average length is 21.44m and the standard deviation is 15.86. Compared with OpenLane-V2 whose average length is 24.11m and the standard deviation is 18.47, the length of lane segments is only a little bit shorter than that of centerlines. So “long range” still remains a great challenge for lane segments.

Besides, it is also demonstrated in the following results where the increased perception range (30m $\to$ 50m) has led to a significant performance drop in all metrics.

| Setting                                  | mAP  | AP$_{ls}$ | AP$_{ped}$ | TOP$_{lsls}$ | AE$_{type}$ | AE$_{dist}$ |
| ---------------------------------------- | :--: | :-------: | :--------: | :----------: | :---------: | :---------: |
| $\pm30\textit{m} \times \pm15\textit{m}$ | 43.3 |   42.5    |    44.2    |     17.7     |     6.7     |    0.536    |
| $\pm50\textit{m} \times \pm25\textit{m}$ | 32.6 |   32.3    |    32.9    |     8.1      |     9.2     |    0.673    |

Reviewer comment (Novelty):
In the context of map learning, the novelty of topology prediction in this paper is limited.

Author response:
Agreed. We haven’t proposed any specific topology module or design to elevate our model’s performance on topology prediction. But thanks to the designing merits of our representation, we have also witnessed a significant performance enhancement in topology prediction as shown in Tab. 3, demonstrating the training friendliness of lane segments. Indeed, we're looking forward to more methods adopting our formulation and exploring the possibilities of employing more novel topological designs in this task.

Reviewer comment (Experiments):
MapTRV2 is a stronger baseline but the relative results are not compared in the experiment section.

Author response:
Thank you for your feedback. We acknowledge the importance of MapTRv2 as a baseline and have accordingly conducted a comparative analysis, the results of which are presented in the accompanying table. Utilizing the same R50 backbone and BEVFormer encoder as MapTR and LaneSegNet, our findings indicate that while MapTRv2 does demonstrate enhanced performance over MapTR, LaneSegNet continues to outperform MapTRv2 in a fair comparison.  We have added the results into the revised paper.

| Setting    | mAP  | AP$_{ls}$ | AP$_{ped}$ | AE$_{dist}$ |
| ---------- | :--: | :-------: | :--------: | :---------: |
| MapTR      | 27.0 |   25.9    |    28.1    |    0.695    |
| MapTRv2    | 28.5 |   26.6    |    30.4    |    0.702    |
| LaneSegNet | 32.6 |   32.3    |    32.9    |    0.673    |

Reviewer comment (Reproducibility):
How to innitialized multiple reference points uniformly inside a lane segment through positional query is not discussed detaily.

Author response:
Thank you for your insightful comment. In response to your suggestion, we have enhanced the details in the revised manuscript

In the process of heads-to-regions, it is critical to guarantee that reference points are uniformly dispersed within a lane segment. To achieve this, we methodically allocate four reference points along two predicted boundaries of a lane segment. Consequently, lane attention is directed to each local area along the lane segment through distinct heads, enabling long range attention and precise local feature extraction. 

In the first decoder layer, due to the absence of geometric guidance from previous layers' predictions, the identity initialization mechanism generates only one reference point from each positional query embedding. Thus, the eight reference points utilized in the first layer's lane attention are positioned identically. This deliberate design choice aims to stabilize the learning of positional priors by filtering out the distraction of intricate geometries.

Reviewer comment (Novelty):
Motivation for a new task - I'm not convinced this task is necessary. Can the authors elaborate why this is a more suitable representation as opposed to the representations in VectorMapNet, MapTR, others?

Author response:
We agree that the map-element-wise abstraction adopted by VectorMapNet and MapTR provides a suitable preliminary representation for online mapping. However, as shown in Fig.1(b),  the representation of map elements fails to adequately indicate topology relationships at intersections and/or situations where lanes diverge/merge. But as demonstrated in previous works such as STSU [1], OpenLane-V2 [2], TopoMLP [3], SMERF [4], etc., such topology relationships are very crucial for autonomous driving, without which vehicles cannot trustfully predict its next action such as changing lanes legally (in this case), turning around and so on. Indeed, lane segment can greatly combine the advantages of both representations.

> [1] Can Y B, Liniger A, Paudel D P and Van Gool L, Structured bird’s-eye-view traffic scene understanding from onboard images. In ICCV, 2021.  
> [2] Wang H, Li T, Li Y, et al. Openlane-v2: A topology reasoning benchmark for scene understanding in autonomous driving. In NeurIPS Track Datasets and Benchmarks, 2023.  
> [3] Wu D, Chang J, Jia F, et al. TopoMLP: An Simple yet Strong Pipeline for Driving Topology Reasoning. arXiv preprint arXiv:2310.06753, 2023.  
> [4] Luo K Z, Weng X, Wang Y, et al. Augmenting Lane Perception and Topology Understanding with Standard Definition Navigation Maps. arXiv preprint arXiv:2311.04079, 2023.

Reviewer comment (Writing):
Writing - the majority of Section 3 (method) was not detailed enough. Section 3.3 (training loss) in particular describes several losses very briefly and with no citations.

Author response:
Thanks for your valuable advice. Considering that the encoder, predictor, and losses applied in our framework are universally adopted by previous works, we saved such space for those more important things in our first submission. More implementation information has already been added to Appendix A.3, including a more detailed explanation of our training loss. We've also accordingly added the following related citations and corresponding references to Sec.3.3 Training Loss in the manuscript:

> [1] Milletari F, Navab N, Ahmadi S A. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV. 2016.  
> [2] Lin T Y, Goyal P, Girshick R, et al. Focal loss for dense object detection. In ICCV. 2017.

Reviewer comment (Experiments):
Comparison with prior work - OpenLane-V2 is relatively new and baseline numbers are from authors implementations. It would help calibrate numbers if they applied their architecture to the original tasks conducted in nuScenes.

Author response:
Thanks and Agreed. We have re-implemented related experiments on nuScenes' original individual task--centerline detection and the results are shown in the following table. Compared with our baseline TopoNet, LaneSegNet outperforms by +6.1 in DET$\_{l}$, +6.4 in TOP$\_{ll}$, and +10.0 in OLS, which is consistent with our experiment results in OpenLane-V2.

| Method         | DET$_{l}$ | TOP$_{ll}$ |   OLS    |
| -------------- | :-------: | :--------: | :------: |
| MapTR          |   8.3     |    0.1     |   11.5   |
| TopoNet        |   24.3    |    2.5     |   20.1   |
| **LaneSegNet** | **30.4**  |  **8.9**   | **30.1** |

Reviewer comment (Presentation):
Figures should be more informative - Figure 2 looks like a generic map prediction architecture and Figure 3 does not help illustrate the lane attention module.

Author response:
Thanks for your valuable suggestion. To fairly validate the representation advantages, we adopt a generic map prediction architecture as shown in Figure 2. We intend to modify Figure 2 to to better highlight the unique aspects of our design in the revised paper. Additionally, we are enthusiastic about the potential for future methods to adopt and expand upon our representational ideology, leading to innovative designs.

Regarding Figure 3, we have made significant revisions to provide a clearer depiction of the lane attention module, particularly emphasizing the heads-to-region mechanism. Detailed implementation aspects and further illustrations have also been incorporated into Appendix A.1.3 to enhance understanding. We appreciate your valuable contribution to improving our paper.

Reviewer comment (Theory):
Take for example a simple road with two lanes (V1, V2 from left to right) - with the proposed representation, this would be represented as V1 = {V1_left, V1_center, V1_right} and V2 = {V2_left, V2_center, V2_right}. How is the consistency of V1_right and V2_left enforced? It seems redundant to regress additional left/right lanes instead of simply regressing centerline + adding left/right lane id attributes.

Author response:
Thank you for highlighting this question. Indeed, there can be instances of inconsistency between V1$\_{right}$ and V2$\_{left}$ in our proposed representation. To address this, we have incorporated advanced post-processing techniques, such as the union-find algorithm, which have demonstrated efficacy in resolving these discrepancies. Actually, such a problem may also exists in other online mapping algorithms, whether it is MapTR, PivotNet, or any other previous methods. Direct results of such map learning methods are usually redundant under a fixed confidence threshold. They may also require corresponding post-processing to simplify the output. 

With respect to the second question, it's true that regressing centerline + adding left/right id attributes can integrate enough semantic information, but the merits of performance enhancement brought by the additional geometrical supervision will be deprived at the same time. In addition, we have also conducted such experiments but insight comes that a strong reasoning capability is strongly bound with a strong prediction performance and thus the absence of supervision of the left/right lane divider leads to a drop in the model's reasoning performance in previous experiments as shown in Tab. 4.

Reviewer comment (Experiments):
Why is Argoverse 2 mentioned on page 7?

Author response:
The OpenLane-V2 dataset is partially built upon the Argoverse 2 dataset. So the original map data has to be firstly re-collected from the Argoverse 2 dataset before further alignment or processment. More details of the data processing approach are elaborated in Appendix A.2.

Reviewer comment (Writing):
Metrics: it would be helpful to readers to define the acronyms for OLS and DET

Author response:
Thanks for your advice. OLS is short for *OpenLane-V2 Score* which is a comprehensive metric defined in OpenLane-V2 while DET represents the detection metric of certain attributes which are usually given in subscripts and calculated in the way of mAP. We've added this paraphrase to the manuscript in the form of footnotes on page 7.

Reviewer comment (Experiments):
The model uses one-to-one optimal assignment between predictions and ground truth with the Hungarian algorithm, which is usually slow and unstable and difficult to train, particularly in the beginning.

Author response:
Thanks for your suggestion. In our experiments, we adopted the one-to-one optimal assignment strategy from the DETR [1] model, which helps eliminate the need for post-processing like non-maximum suppression (NMS). However, as you noted, this strategy is usually slow and unstable and difficult to train, particularly in the beginning. To address this, we've refined the Deformable Attention [2] module with lane attention and identical initialization. These adjustments have improved the model's convergence, as shown in our results Tab. 8. Additionally, we are considering the one-to-many assignment strategy as a future exploration direction.

> [1] Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers. In ECCV, 2020.  
> [2] Zhu X, Su W, Lu L, et al. Deformable detr:
Deformable transformers for end-to-end object detection. In ICLR, 2020.

Reviewer comment (Evaluation):
While metrics look promising, the metrics are still pretty low. It would be more interested in showing what type of lanes the model can do so well that it can be used for autonomous driving.

Author response:
The low metrics are primarily due to two factors. First, compared to existing HD map learning tasks, whose perception range is $\pm30\textit{m} \times \pm15\textit{m}$ , we have expanded the range to $\pm50\textit{m} \times \pm25\textit{m}$. Second, the metrics are defined relatively stringently and compactly. 

Taking two main metrics AP$\_{ls}$ and TOP$\_{lsls}$ as examples:

- The calculation of AP$\_{ls}$ is based on the lane segment distance between the predicted point sets and the ground truth in Equation 5. As we have expanded the range, the distance differences in point sets farther away make a great difference by leading to an excess of the matching threshold and an increase in false positives thus resulting in lower metrics. 

- The TOP$\_{lsls}$ metric requires both accurate line detection and the detection of relevant relationships, making its assessment criteria quite stringent and compact. It is indicated by previous large amounts of experiments that the upper bound of TOP$\_{lsls}$ is about the numerical level of AP$\_{ls}$‘s square.

To further illustrate this issue, we directly re-evaluated our trained model under the setting of $\pm30\textit{m} \times \pm15\textit{m}$, and the results are as follows.

| Setting                                  | mAP  | AP$_{ls}$ | AP$_{ped}$ | TOP$_{lsls}$ | AE$_{type}$ | AE$_{dist}$ |
| ---------------------------------------- | :--: | :-------: | :--------: | :----------: | :---------: | :---------: |
| $\pm30\textit{m} \times \pm15\textit{m}$ | 43.3 |   42.5    |    44.2    |     17.7     |     6.7     |    0.536    |
| $\pm50\textit{m} \times \pm25\textit{m}$ | 32.6 |   32.3    |    32.9    |     8.1      |     9.2     |    0.673    |

In addition, we provide qualitative visualization in Appendix C. We can see that laneSegNet performs well as shown in Figure 7 no matter whether it is in typical, complex, or long-tail scenarios.

Reviewer comment (Experiments):
How long is each lane segment?

Author response:
Thanks for your insightful question. We have analyzed the data of lane segments. The average length is 21.44m and the standard deviation is 15.86. Compared with OpenLane-V2 whose average length is 24.11m and the standard deviation is 18.47, the continuity of lanes is still well-preserved.

Reviewer comment (Reproducibility):
For a lane segment {A_left, A, A_right}, each appears in 3 lane segments, how to combine them to get final result for e.g. A?

Author response:
According to Sec.3.1, the geometric composition of a lane segment is defined as the vectorized centerline and its corresponding lane boundaries, while its semantics include lane segment classification and line type of the left/right lane boundary. During map learning, the decoder outputs an instance embedding corresponding to each lane segment, followed by predictors that forecast the centerline and boundaries' offset relative to the centerline, along with its associated semantics. Consequently, for the learning of a lane segment, these pieces of information are acquired through instance embedding and are inherently interlinked.

Reviewer comment (Experiments):
No ablation on the choice for the BEV encoder.

Author response:
Thanks. We have added an ablation study on the BEV encoder section. For implement convenience, we selected GKT encoder to compare with BEVFormer encoder.

| Method               |   mAP    | AP$_{ls}$ | AP$_{ped}$ | TOP$_{lsls}$ | AE$_{type}$ | AE$_{dist}$ |
| :------------------- | :------: | :-------: | :--------: | :----------: | :---------: | :---------: |
| GKT [1]              |   30.1   |   28.9    |    30.3    |     6.6      |    10.9     |    0.700    |
| **BEVFormer (Ours)** | **32.6** | **32.3**  |  **32.9**  |   **8.1**    |   **9.2**   |  **0.673**  |

> [1] Chen S, Cheng T, Wang X, et al. Efficient and robust 2d-to-bev representation learning via geometry-guided kernel transformer. arXiv preprint arXiv:2206.04584.