Paper: SEPT: Towards Efficient Scene Representation Learning for Motion Prediction (ICLR-2024)

Reviewer comment (Novelty):
Limited novelty: The model structure is not new, which is basically SceneTransformer [A], which should be considered as contributions. For the three pretraining objectives, it is very similar to Traj-MAE [B] and ForecastMAE [C], as stated by the authors as well. Compared to [B] and [C], they achieve better performance, but the paper does not give a thorough analysis regarding the difference. As a result, it is hard to determine the real working part of the method.

Author response:
> W1 (a): Limited novelty: The model structure is not new, which is basically SceneTransformer [A], which should (not) be considered as contributions.

We consider scene understanding pretraining as the primary contribution of SEPT paper.  Benefited from the pretraining scheme, our model could achieve SOTA performance with a greatly simplified architecture, as illustrated in Figure 2 of the paper. Regarding our model's relationship with SceneTransformer, while both the two models are built on transformer, there are several notable differences in their architectural designs:
1. **Encoder architecture**: SceneTransformer takes a spatiotemporal-intertwined encoding structure that performs factorized attention along both the time and space (agent & road graph) axes alternately and repeatedly, to fuse the spatiotemporal information. In contrast, SEPT adopts a spatiotemporal-separated encoding mechanism that, first processes temporal information within agent trajectories and reduces the time dimension by max pooling, then encodes the spatial relationships among agents and roads. This saves computation, allowing for more efficient training and inference.
2. **Decoder architecture**: SEPT's decoder uses N learnable queries to aggregate information from scene embeddings through cross-attention operation, with N corresponding to the number of motion modalities. On the other hand, SceneTransformer's decoder conducts self-attention across time and agent dimension on the "agent features" which is stacked N times, and it adds one-hot encoding to distinguish each predictive modality.

> W1 (b): For the three pretraining objectives, it is very similar to Traj-MAE[B] and ForecastMAE[C], as stated by the authors as well. Compared to [B] and [C], they achieve better performance, but the paper does not give a thorough analysis regarding the difference. As a result, it is hard to determine the real working part of the method.

> Q1: Could the authors explain more about the difference between your method and Traj-MAE/ForecastMAE and discuss why your method better performance than them? It might be the case that "the devil is in the details" and I think it is important to find out these details as your contributions. Otherwise, the proposed methodology is basically the same as Traj-MAE/ForecastMAE, which has nothing new.

Regarding the relationship between our method and Traj-MAE and Forecast-MAE, we would like to revise our paper and expand the related works section to highlight the main differences:

1. **Traj-MAE** designs two independent mask-reconstruction tasks on trajectories and road map input, to train its trajectory and map encoder separately. Such separation leads to a notable limitation that the spatial relationship between agents and roads are not stressed during pretraining. In contrast, SEPT's pretraining introduces the Tail prediction (TP) task, which considers both temporal and spatial information, and jointly trains TempoNet and SpaNet. Our ablation experiments clearly show that TP has a significant impact on the final performance improvement. This could be explained by the fact that TP can help the encoder better align the representations learned by the masked trajectory modeling (MTM) and masked road modeling (MRM) tasks.
2. **Forecast-MAE** has a notable difference with our method: the granularity of the tokenization for the scene inputs. Forecast-MAE uses the whole historical or future trajectory as a single token, while SEPT treats a waypoint in a trajectory as a token. Similarly, for road input, it uses a polyline of lane segment as a token, while our method uses a short road vector less than 5m. We believe that our approach can better capture the motion patterns, as well as the dependency between the historical motions and the road structure. Also, Forecast-MAE adopts a distinct masking strategy for agent trajectory by incorporating ground-truth future in pretraining, and it predict the future given its history or vice versa.

Also, we would like to share more thoughts on the performance differences:
1. Within the benchmarks of Argoverse 1 and 2, preceding SOTA methods like QCNet and ProphNet set a notable standard. SEPT was able to surpass these methods, but it required the assistance of SSL pretraining to do so. Hence, we consider the proposed SSL pretraining scheme as the primary contributor to the observed performance improvement.
2. It's worth mentioning that Traj-MAE chose AutoBots as the baseline model, which, as observed, is not a strong competitor on Argoverse 1 (minADE 0.89; minFDE 1.41). We would like to highlight that earlier this year, our prototype model, without pretraining, achieved a minADE of 0.83 and a minFDE of 1.30 on the Argoverse 1 test set. We believe that the performance improvement over a stronger baseline model lends more credibility to our method.

Reviewer comment (Novelty):
Could the authors explain more about the difference between your method and Traj-MAE/ForecastMAE and discuss why your method achieves better performance than them? It might be the case that "the devil is in the details" and I think it is important to find out these details as your contributions. Otherwise, the proposed methodology is basically the same as Traj-MAE/ForecastMAE, which has nothing new.

Author response:
> Q1: Could the authors explain more about the difference between your method and Traj-MAE/ForecastMAE and discuss why your method better performance than them? It might be the case that "the devil is in the details" and I think it is important to find out these details as your contributions. Otherwise, the proposed methodology is basically the same as Traj-MAE/ForecastMAE, which has nothing new.

Regarding the relationship between our method and Traj-MAE and Forecast-MAE, we would like to revise our paper and expand the related works section to highlight the main differences:

1. **Traj-MAE** designs two independent mask-reconstruction tasks on trajectories and road map input, to train its trajectory and map encoder separately. Such separation leads to a notable limitation that the spatial relationship between agents and roads are not stressed during pretraining. In contrast, SEPT's pretraining introduces the Tail prediction (TP) task, which considers both temporal and spatial information, and jointly trains TempoNet and SpaNet. Our ablation experiments clearly show that TP has a significant impact on the final performance improvement. This could be explained by the fact that TP can help the encoder better align the representations learned by the masked trajectory modeling (MTM) and masked road modeling (MRM) tasks.
2. **Forecast-MAE** has a notable difference with our method: the granularity of the tokenization for the scene inputs. Forecast-MAE uses the whole historical or future trajectory as a single token, while SEPT treats a waypoint in a trajectory as a token. Similarly, for road input, it uses a polyline of lane segment as a token, while our method uses a short road vector less than 5m. We believe that our approach can better capture the motion patterns, as well as the dependency between the historical motions and the road structure. Also, Forecast-MAE adopts a distinct masking strategy for agent trajectory by incorporating ground-truth future in pretraining, and it predict the future given its history or vice versa.

Also, we would like to share more thoughts on the performance differences:
1. Within the benchmarks of Argoverse 1 and 2, preceding SOTA methods like QCNet and ProphNet set a notable standard. SEPT was able to surpass these methods, but it required the assistance of SSL pretraining to do so. Hence, we consider the proposed SSL pretraining scheme as the primary contributor to the observed performance improvement.
2. It's worth mentioning that Traj-MAE chose AutoBots as the baseline model, which, as observed, is not a strong competitor on Argoverse 1 (minADE 0.89; minFDE 1.41). We would like to highlight that earlier this year, our prototype model, without pretraining, achieved a minADE of 0.83 and a minFDE of 1.30 on the Argoverse 1 test set. We believe that the performance improvement over a stronger baseline model lends more credibility to our method.

Reviewer comment (Reproducibility):
Do the authors plan to open source the code so that the community could easily reproduce the results and find out the working part?

Author response:
> Do the authors plan to open source the code so that the community could easily reproduce the results and find out the working part?

Certainly! We plan to opensource our codebase once the paper is accepted. It is a great pleasure to share our work with the community!

Reviewer comment (Novelty):
Missed reference: it seems that some classic transformer-based works are not mentioned, which might cause wrong understanding of the contributions of this work: [D] Motion Transformer with Global Intention Localization and Local Movement Refinement. NeurIPS 2022 Oral; [E] HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encoding. IEEE TPAMI; [F] Multi-modal motion prediction with transformer-based neural network for autonomous driving. ICRA 2022.

Author response:
> Missed reference: it seems that some classic transformer based works are not mentioned, which might cause wrong understanding of the contributions of this work.

We appreciate your suggestion to include more classic transformer-based works in our related works section. We agree that these works are important and relevant to our research topic, and we will review them in our revised manuscript.

Reviewer comment (Novelty):
In summary, the proposed method has strong empirical performance on the public leaderboards but the method described in the paper basically has nothing new compared to existing methods, which makes their contributions unclear. Since the authors neither provide any detailed comparisons and analysis with existing methods nor mention their plans for open source, I do not think the paper could provide much value to the community, thus I give a borderline reject. If the authors could solve the aforementioned issues, I may change my scores accordingly.

Author response:
> W1 (a): Limited novelty: The model structure is not new, which is basically SceneTransformer [A], which should (not) be considered as contributions.

We consider scene understanding pretraining as the primary contribution of SEPT paper.  Benefited from the pretraining scheme, our model could achieve SOTA performance with a greatly simplified architecture, as illustrated in Figure 2 of the paper. Regarding our model's relationship with SceneTransformer, while both the two models are built on transformer, there are several notable differences in their architectural designs:
1. **Encoder architecture**: SceneTransformer takes a spatiotemporal-intertwined encoding structure that performs factorized attention along both the time and space (agent & road graph) axes alternately and repeatedly, to fuse the spatiotemporal information. In contrast, SEPT adopts a spatiotemporal-separated encoding mechanism that, first processes temporal information within agent trajectories and reduces the time dimension by max pooling, then encodes the spatial relationships among agents and roads. This saves computation, allowing for more efficient training and inference.
2. **Decoder architecture**: SEPT's decoder uses N learnable queries to aggregate information from scene embeddings through cross-attention operation, with N corresponding to the number of motion modalities. On the other hand, SceneTransformer's decoder conducts self-attention across time and agent dimension on the "agent features" which is stacked N times, and it adds one-hot encoding to distinguish each predictive modality.

> W1 (b): For the three pretraining objectives, it is very similar to Traj-MAE[B] and ForecastMAE[C], as stated by the authors as well. Compared to [B] and [C], they achieve better performance, but the paper does not give a thorough analysis regarding the difference. As a result, it is hard to determine the real working part of the method.

> Q1: Could the authors explain more about the difference between your method and Traj-MAE/ForecastMAE and discuss why your method better performance than them? It might be the case that "the devil is in the details" and I think it is important to find out these details as your contributions. Otherwise, the proposed methodology is basically the same as Traj-MAE/ForecastMAE, which has nothing new.

Regarding the relationship between our method and Traj-MAE and Forecast-MAE, we would like to revise our paper and expand the related works section to highlight the main differences:

1. **Traj-MAE** designs two independent mask-reconstruction tasks on trajectories and road map input, to train its trajectory and map encoder separately. Such separation leads to a notable limitation that the spatial relationship between agents and roads are not stressed during pretraining. In contrast, SEPT's pretraining introduces the Tail prediction (TP) task, which considers both temporal and spatial information, and jointly trains TempoNet and SpaNet. Our ablation experiments clearly show that TP has a significant impact on the final performance improvement. This could be explained by the fact that TP can help the encoder better align the representations learned by the masked trajectory modeling (MTM) and masked road modeling (MRM) tasks.
2. **Forecast-MAE** has a notable difference with our method: the granularity of the tokenization for the scene inputs. Forecast-MAE uses the whole historical or future trajectory as a single token, while SEPT treats a waypoint in a trajectory as a token. Similarly, for road input, it uses a polyline of lane segment as a token, while our method uses a short road vector less than 5m. We believe that our approach can better capture the motion patterns, as well as the dependency between the historical motions and the road structure. Also, Forecast-MAE adopts a distinct masking strategy for agent trajectory by incorporating ground-truth future in pretraining, and it predict the future given its history or vice versa.

Also, we would like to share more thoughts on the performance differences:
1. Within the benchmarks of Argoverse 1 and 2, preceding SOTA methods like QCNet and ProphNet set a notable standard. SEPT was able to surpass these methods, but it required the assistance of SSL pretraining to do so. Hence, we consider the proposed SSL pretraining scheme as the primary contributor to the observed performance improvement.
2. It's worth mentioning that Traj-MAE chose AutoBots as the baseline model, which, as observed, is not a strong competitor on Argoverse 1 (minADE 0.89; minFDE 1.41). We would like to highlight that earlier this year, our prototype model, without pretraining, achieved a minADE of 0.83 and a minFDE of 1.30 on the Argoverse 1 test set. We believe that the performance improvement over a stronger baseline model lends more credibility to our method.

> Do the authors plan to open source the code so that the community could easily reproduce the results and find out the working part?

Certainly! We plan to opensource our codebase once the paper is accepted. It is a great pleasure to share our work with the community! 

> Missed reference: it seems that some classic transformer based works are not mentioned, which might cause wrong understanding of the contributions of this work.

We appreciate your suggestion to include more classic transformer-based works in our related works section. We agree that these works are important and relevant to our research topic, and we will review them in our revised manuscript.

Reviewer comment (Experiments):
While results are reported on both Argoverse 1 and 2, it would be good to see generalization of the proposed methods to at-least one other dataset such as waymo open dataset or nuScenes.

Author response:
We recognize the importance of testing SEPT's performance on a wider range of datasets to assess its generalization capabilities more comprehensively. Specifically, we will consider including test results on additional datasets like Waymo Open Dataset or nuScenes in the camera-ready version to provide a more thorough evaluation of our approach. Thank you for your feedback, and we will ensure that this aspect is addressed in our final submission.

Reviewer comment (Experiments):
What is the impact on the results when not using the test set for pre-training?

Author response:
For participants in the Argoverse 1 motion prediction challenge, it's a widely recognized observation that test performance often exhibits a substantial gap when compared to validation performance, potentially due to distribution mismatches. Indeed, our initial motivation for exploring pretraining, especially on the test set, as the reviewer speculates, was to address this gap. However, the experimental results did not align with our expectations. Surprisingly, this led us in a different direction, ultimately resulting in an overall improvement in prediction performance.

During our investigation, we also experimented with excluding the test set from the pretraining process. We found that this exclusion has a negligible impact on the prediction performance. We decide to keep using the test set to demonstrate that our pretraining could incorporate all the data at hand. In the next few days, we intend to conduct further experiments and will include an ablation study to provide additional insight into the use of the test set in our approach.

Reviewer comment (Novelty):
The Introduction effectively illustrates the potential of SSL in motion prediction tasks. However, it could benefit from acknowledging previous works such as Traj-MAE and Forecast-MAE that have already explored SSL in motion prediction. The three self-supervised tasks presented appear to be inspired or derived from these preceding studies.

Author response:
We appreciate your feedback and agree that our paper could benefit from acknowledging previous works, including Traj-MAE and Forecast-MAE, which have explored self-supervised learning in motion prediction.

However, we want to clarify that while our work draws inspiration from the success of self-supervised learning, it is not directly derived from Traj-MAE or Forecast-MAE. Early this year, our initial motivation for exploring pretraining was to address the validation-test performance gap of our model on Argoverse 1. The experimental results did not align with our expectations, but led us in a different direction, ultimately resulting in an overall improvement in prediction performance. We became aware of Traj-MAE and Forecast-MAE around June 2023, but by that time, our task design had already been finalized.

We will revise the introduction to include references to these works. Also, to address the concerns raised by multiple reviewers regarding the relationship between our method and Traj-MAE and Forecast-MAE, we would like to expand the related works section to highlight the main differences:
1. **Traj-MAE** designs two independent mask-reconstruction tasks on trajectories and road map input, to train its trajectory and map encoder separately. Such separation leads to a notable limitation that the spatial relationship between agents and roads are not stressed during pretraining. In contrast, SEPT's pretraining introduces the Tail prediction (TP) task, which considers both temporal and spatial information, and jointly trains TempoNet and SpaNet. Our ablation experiments clearly show that TP has a significant impact on the final performance improvement. This could be explained by the fact that TP can help the encoder better align the representations learned by the masked trajectory modeling (MTM) and masked road modeling (MRM) tasks.
2. **Forecast-MAE** has a notable difference with our method: the granularity of the tokenization for the scene inputs. Forecast-MAE uses the whole historical or future trajectory as a single token, while SEPT treats a waypoint in a trajectory as a token. Similarly, for road input, it uses a polyline of lane segment as a token, while our method uses a short road vector less than 5m. We believe that our approach can better capture the motion patterns, as well as the dependency between the historical motions and the road structure. Also, Forecast-MAE adopts a distinct masking strategy for agent trajectory by incorporating ground-truth future in pretraining, and it predict the future given its history or vice versa.

Reviewer comment (Novelty):
The framework presented in the paper is well-organized and operational, but it seems to lack novelty and groundbreaking contributions in the area of study.

Author response:
Thank you for acknowledging the organization and operational nature of our framework. While we understand that our work may not be deemed groundbreaking in the sense of a revolutionary breakthrough, we do believe that novelty lies in our introduction of self-supervised learning (SSL) to the field of motion prediction, as well as our unique task design. Our experiments and ablation studies have shown that pretraining for scene understanding significantly contributes to prediction performance. 

In response to your valuable feedback, we will also revise the paper to provide more in-depth details about the relationship between our method and Traj-MAE as well as Forecast-MAE. This will help clarify our specific contributions in the context of existing research.

Reviewer comment (Evaluation):
It's observed that the predominant improvement in SEPT's performance relative to the previous state-of-the-art isn’t chiefly attributed to SSL pretraining. This seems to be somewhat divergent from the central thesis and focus of the paper.

Author response:
Defining 'state-of-the-art' in this context might introduce ambiguity. In our interpretation, SOTA references the leading techniques showcased on specific leaderboards, such as QCNet and ProphNet on Argoverse, as well as MTR on WOMD. Traj-MAE and Forecast-MAE, while exploring SSL, did not attain SOTA performance on both Argoverse 1 and 2.

Within the benchmarks of Argoverse 1 and 2, preceding SOTA methods like QCNet and ProphNet set a notable standard. SEPT was able to surpass these methods, but it required the assistance of SSL pretraining to do so. Hence, we consider the proposed SSL pretraining scheme as the primary contributor to the observed performance improvement.

Regarding the comparison with Traj-MAE and Forecast-MAE, besides pretraining, our performance gains likely stem from various other elements, encompassing data preprocessing, input representation, model architecture, and hyperparameters. Through our exploration, we found that the agent-centric data representation and the DETR-like decoder architecture may have pivotal roles in this observed enhancement.

Reviewer comment (Novelty):
Regarding the related work, you’ve cited that the lane mask SSL task of Forecast-MAE didn’t exhibit considerable advancement in the ablation study. However, in Traj-MAE, there was a significant enhancement noted in the HDMap mask. Isn't the consistent improvement across all SSL tasks already an established finding prior to this research?

Author response:
We noticed that Traj-MAE indeed conducted an ablation study on the effectiveness of pretraining tasks, as indicated in Table 4 of the paper and Table 4 of the supplementary material. From our perspective, while the consistent improvement across SSL tasks can be considered an indicator of reasonable task design, it may not be the sole criterion for evaluating a method's efficacy.

It's worth mentioning that Traj-MAE chose AutoBots as the baseline model, which, as observed, is not a strong competitor on the Argoverse 1 leaderboard (minADE 0.89; minFDE 1.41). We would like to highlight that earlier this year, our prototype model, without extensive pretraining or hyperparameter tuning, achieved a minADE of 0.83 and a minFDE of 1.30 on the Argoverse 1 test set, with only 3.5 hours of training on a single NVIDIA 3090 Ti. We believe that the performance improvement over a stronger baseline model lends more credibility to our method.

Furthermore, we observed a notable gap between Traj-MAE's validation and test performance on Argoverse 1. While its validation performance surpassed that of ProphNet and our SEPT, its test performance was significantly worse (as elaborated in our response to your Q3). In light of this, we find Traj-MAE's ablation study, which relies on their validation performance, less convincing.

Reviewer comment (Evaluation):
In table 4, there is a noted reduction of 6.4% in minADE on the validation set relative to the baseline without SSL pretraining. Meanwhile, SEPT shows a 10.1% and 12.3% decrease in minADE compared to Traj-MAE and Forecast-MAE, respectively. Could you elucidate whether the baseline model, even without the SSL pretraining, has already exceeded the performance of previous state-of-the-art methods? And what are the primary contributors to this enhancement in performance?

Author response:
Defining 'state-of-the-art' in this context might introduce ambiguity. In our interpretation, SOTA references the leading techniques showcased on specific leaderboards, such as QCNet and ProphNet on Argoverse, as well as MTR on WOMD. Traj-MAE and Forecast-MAE, while exploring SSL, did not attain SOTA performance on both Argoverse 1 and 2.

Within the benchmarks of Argoverse 1 and 2, preceding SOTA methods like QCNet and ProphNet set a notable standard. SEPT was able to surpass these methods, but it required the assistance of SSL pretraining to do so. Hence, we consider the proposed SSL pretraining scheme as the primary contributor to the observed performance improvement.

Regarding the comparison with Traj-MAE and Forecast-MAE, besides pretraining, our performance gains likely stem from various other elements, encompassing data preprocessing, input representation, model architecture, and hyperparameters. Through our exploration, we found that the agent-centric data representation and the DETR-like decoder architecture may have pivotal roles in this observed enhancement.

Reviewer comment (Evaluation):
When it comes to model efficiency, the paper only compares with QCNet which is one of the largest models among all comparison methods. Since time-consuming is one of the main drawbacks of previous methods as is claimed in Section 1, it would be good to know the exact inference speeds of SEPT and the other comparison methods.

Author response:
Our paper primarily focuses on self-supervised scene understanding, which enhances prediction accuracy with a greatly simplified network design. We included QCNet in our comparisons because it was the previous SOTA on Argoverse 1 and 2 leaderboard, and intended to highlight that our method achieves SOTA performance with a reduced computational cost. In the paper, we indeed mentioned that “These empirical techniques consequently lead to an intricate and time-consuming information processing pipeline”. However, we acknowledge that (1) time-saving is not the primary objective of our research, and (2) the information processing pipeline is not the sole determinant of inference efficiency. Therefore, we plan to remove the claim of "time-consuming" from that sentence in our revised paper.

In response to your inquiry about the exact inference speeds of SEPT, we conducted comprehensive tests under various circumstances, as outlined below:
1. GPU inference during training (batch size = 96): 26 iteration/s, or 2496 samples/s, on an NVIDIA RTX 3090 Ti;
2. GPU inference at deployment using ONNX (batch size = 1): 4.5 ms/sample, on an NVIDIA GTX 1060;
3. CPU inference at deployment using ONNX (batch size = 1, single-threaded): 27.8 ms/sample, on an Intel Core i5-8400.

These results demonstrate that our model's inference speeds are competitive and suitable for use within an autonomous driving pipeline.

Reviewer comment (Evaluation):
It would be good if the authors could talk more about their research findings in addition to presenting their approach, which I believe will inspire other researchers.

Author response:
We appreciate your feedback and agree with your suggestion. In our revised manuscript, we will expand our discussion to include a dedicated part on our research findings. This part will emphasize the significance of pretraining and its task design in the context of our work, addressing the concerns raised by multiple reviewers.