Paper: RedMotion: Motion Prediction via Redundancy Reduction (ICLR-2024)

Reviewer comment (Experiments):
As a work focused on efficiency, no statistics about inference memory footprint or inference time are reported. I suggest the authors report these related statistics and compare with open-sourced classic baselines like MTR/QCNet. Otherwise, the claimed advantage of reduction can not be known.

Author response:
* Issue 1: As a work focused on efficiency, no statistics about inference memory footprint or inference time are reported.

We want to highlight that our work is not focused on efficiency but on self-supervised representation learning.

Reviewer comment (Evaluation):
Self-defined metrics without proper justification. For Waymo Open Motion dataset, widely used (and official) metrics are minADE_6, minFDE_6, mAP averaged over 3s, 5s, 8s. I do not see the necessity of using these similar but different metrics in the paper. For fair comparision, I suggest reporting the official metrics in the paper.

Author response:
* Issue 3: Self-defined metrics without proper justification and no test results.

We have added a bar plot of the number of trajectory proposals in Appendix C. Furthermore, it was our misunderstanding that we interpreted the following sentence as, each prediction can contain up to 8 trajectory proposals per agent: "Each ScenarioPredictions proto within the submission corresponds to a single scenario in the test set and contains up to 8 predictions for the objects listed in the tracks_to_predict field of the scenario"  [[https://waymo.com/open/challenges/2023/motion-prediction/#submit](https://waymo.com/open/challenges/2023/motion-prediction/#submit)]

The current challenge rules only allow up to 6 proposals per agent, therefore we retrained our models with 6 trajectory proposals and submitted the results of our best model to the challenge leaderboard: [[Waymo Prediction Challenge Entry](https://waymo.com/open/challenges/entry/?challenge=MOTION_PREDICTION&challengeId=MOTION_PREDICTION_2023&emailId=824a6c20-3f34&timestamp=1700489974797915
)]

Reviewer comment (Evaluation):
No test set results. Waymo Open Motion Leaderboard is easy to submit and I think it is **very necessay** to report the results on it to avoid overfitting the validation set.

Author response:
* Issue 3: Self-defined metrics without proper justification and no test results.

We have added a bar plot of the number of trajectory proposals in Appendix C. Furthermore, it was our misunderstanding that we interpreted the following sentence as, each prediction can contain up to 8 trajectory proposals per agent: "Each ScenarioPredictions proto within the submission corresponds to a single scenario in the test set and contains up to 8 predictions for the objects listed in the tracks_to_predict field of the scenario"  [[https://waymo.com/open/challenges/2023/motion-prediction/#submit](https://waymo.com/open/challenges/2023/motion-prediction/#submit)]

The current challenge rules only allow up to 6 proposals per agent, therefore we retrained our models with 6 trajectory proposals and submitted the results of our best model to the challenge leaderboard: [[Waymo Prediction Challenge Entry](https://waymo.com/open/challenges/entry/?challenge=MOTION_PREDICTION&challengeId=MOTION_PREDICTION_2023&emailId=824a6c20-3f34&timestamp=1700489974797915
)]

Reviewer comment (Experiments):
I missed a more clear explanation on the motion prediction baseline used when not applying any pre-training.

Author response:
* Q1: More clear explanation on the motion prediction baseline used when not applying any pre-training.

Learning global environment features represented as RED tokens is our contribution, hence we remove the parallel decoder in the road environment encoder for the baseline version (see Figure 1). To allow a fair comparison, we increase the token dimension for the baseline from 128 to 192 to give both models a similar capacity (RedMotion baseline 9.9M params vs. RedMotion 9.2M params). We added this explicit size comparison to Section 4.1.

Reviewer comment (Novelty):
The first claimed contribution is that the authors design an architecture that reduces the variable-length set of map objects to an encoding of fixed-length, but it is unclear how this encoding is different from "latent query attention" used in Wayformer, which also achieves the same. The submission claims that compared to Wayformer, it encodes past agent trajectories and environment context with separated encoders to offer more flexibility for specialized self-supervised pre-training. However, Wayformer also compares against a "late fusion" variant that seems very similar to the proposed architecture. Clarification is needed on these architectural choices, the comparison to Wayformer, and the reason for excluding Wayformer from Table 2.

Author response:
* Q1: How are road environment descriptors (RED) different from latent queries in Wayformer?

The methods are related, yet they primarily differ in two aspects: firstly, in their respective positions within the model, and secondly, in their intended purposes. Our RED are learned by an internal decoder and serve as input for a motion head, while latent queries form a part of the transformer decoder that serves as the motion head.

RED provide environment context (map and current agent state) and are learned with self-supervised learning, while latent queries can be understood as trajectory anchors and are learned using supervised learning.

Building on this difference, we have analyzed the effect of combining RED and latent queries. Specifically, we implemented a version of our model named RedMotion tra-dec, which uses a transformer decoder as the motion head. In this way, the RED are learned via redundancy reduction, thereby providing context that is subsequently transformed into trajectory proposals by the latent queries (refer to Sec. 4.2 for results).

* **Issue 1:** Latent querys interpretation and issues with the Wayformer architecture.
  * We don’t think that there is any issue with the Wayformer architecture, but we see key difference to our architecture and modules. In our opinion, there are two forms of latent queries used in the Wayformer architecture: learned queries in the encoder and in the decoder.
    * In the decoder (Sec. 3.3 Wayformer paper, learned seeds in Fig. 1) learned queries are used to attend to the encoded features and transform them into trajectory proposals. We think that these are inspired by the “learned anchors” from Waymo’s previous works [MultiPath](https://arxiv.org/abs/1910.05449) or [MultiPath++](https://arxiv.org/abs/2111.14973v3). The key difference in our model, is that these elements are part of the motion head. They are learned through supervised learning and can be integrated into our model (tra-dec config).
    * In the encoder(s) (Sec. 3.2 Wayformer paper) learned queries are used to reduce the computational cost for processing long input sequences. These are inspired by the [Set Transformer](https://proceedings.mlr.press/v97/lee19d.html) and [Perceiver](https://proceedings.mlr.press/v139/jaegle21a.html) architectures. In contrast, we are using [local attention](https://arxiv.org/abs/2004.05150) in our road environment encoder to lower the computational complexity (see Sec. 3.2 or Fig. 4 in our work) and a memory efficient implementation of cross attention to fuse features (see Fig. 5).

Reviewer comment (Theory):
For the second contribution of pre-training the map encoding to be invariant to small rotations and translations, it is unclear whether this is desirable. If the map rotates, the motion prediction should also rotate to some extent. The principle behind this pre-training objective is not entirely clear.

Author response:
* Q2: Is a map encoding that is invariant to small rotations and translations undesirable for trajectory prediction?

In self-supervised pre-training, we learn similar embeddings for moderately augmented environments as correctly described by you. During subsequent fine-tuning, we use these embeddings as "good initialization" to learn trajectory prediction. At this phase, we do not augment environments anymore to learn representations that are responsive to translation and rotation.

Similarly, self-supervised methods in computer vision typically learn augmentation invariant features during pretraining, while during fine-tuning for object detection augmentation-sensitive features are learned.

Reviewer comment (Experiments):
Section 4.1 "dataset" - The motivation for studying the semi-supervised setting for motion prediction is lacking. Since collecting training data only requires driving through cities and running perception models, it seems all of the data is always labeled. If the argument is that the approach can fine-tune motion prediction on maps without motion data, this should be explicitly demonstrated, or pre-training should be done on a dataset containing only maps without motion data. If the argument is that the pre-trained RED tokens are useful for many downstream tasks, then more tasks than just motion prediction should be benchmarked.

Author response:
* Q3: Motivation for studying the semi-supervised setting for motion prediction? Collecting training data only requires driving through cities and running perception models.

Generating trajectory prediction datasets can involve labor-intensive post-processing, such as filtering false positive detections and trajectory smoothing, etc.

* **Issue 2:** The authors should perform their experiments in a setting where there is a large quantity of map data without trajectories
  * We want to highlight our approach represents a common practice in self-supervised representation learning. This approach aligns with seminal works in computer vision, such as [[SimCLR](https://proceedings.mlr.press/v119/chen20j.html)] , [[SwaV](https://proceedings.neurips.cc/paper/2020/hash/70feb62b69f16e0238f741fab228fec2-Abstract.html)], which employ this setting for labeled datasets such as ImageNet.

* **Issue 3:** Trajectory labels can be generated fully automated.
  * Although we are uncertain about the processing pipelines in large companies and organizations like Waymo or Motional, our experience suggests that collecting trajectory datasets invariably involves a significant amount of manual labor. This is due to the fact that fusion and perception systems can sometimes produce inaccuracies, such as jumps.

Reviewer comment (Writing):
Section 3.2 "Figure 3c shows the vocabulary size" - It is unclear if "vocabulary size" is the right term. Vocabulary size usually refers to the size of a set of discrete options, but Figure 3c appears to show sequence lengths, and none of the inputs to the model are discretized.

Author response:
* Q5: Figure 3c, Is "vocabulary size" the right term?

We describe the number of discrete options with vocabulary size. The sequence length for environment tokens is 1200 and we have added these information in the revised paper.

Reviewer comment (Evaluation):
Section 4.2 "Interestingly, the Scene Transformer in its joint prediction configuration performs worse than in its marginal prediction version" - This is actually expected; joint prediction should be better if minADE/minFDE are evaluated at the scene level, and marginal prediction should be better if evaluated at the per-agent level.

Author response:
* Q7: Section 4.2 "Interestingly, the Scene Transformer in its joint prediction configuration performs worse than in its marginal prediction version”

We have revised this paragraph.

Reviewer comment (Evaluation):
Section 4.2 "our model with 16 trajectory proposals can outperform all other methods" - While a plot of minFDE as a function of samples is interesting, including a 16-sample version in Table 2 might be too unfair of a comparison.

Author response:
* Q8: Plot minFDE as a function.

We have added a bar plot of the number of trajectory proposals in Appendix C. Furthermore, it was our misunderstanding that we interpreted the following sentence as, each prediction can contain up to 8 trajectory proposals per agent: "Each ScenarioPredictions proto within the submission corresponds to a single scenario in the test set and contains up to 8 predictions for the objects listed in the tracks_to_predict field of the scenario" [[https://waymo.com/open/challenges/2023/motion-prediction/#submit](https://waymo.com/open/challenges/2023/motion-prediction/#submit)]

The current challenge rules only allow up to 6 proposals per agent, therefore we retrained our models with 6 trajectory proposals and submitted the results of our best model to the challenge leaderboard: [[Waymo Prediction Challenge Entry](https://waymo.com/open/challenges/entry/?challenge=MOTION_PREDICTION&challengeId=MOTION_PREDICTION_2023&emailId=824a6c20-3f34&timestamp=1700489974797915
)]

Reviewer comment (Evaluation):
Table 2 - The authors do not report mAP. Since the MTR and MTR++ models are chosen based on mAP, this might lead to lower values for minFDE/minADE, making the current comparison not exactly apples-to-apples.

Author response:
* Q9: Table 2 - is there a reason the authors don't report mAP?

We have added the official challenge metric Soft mAP.

* **Issue 4:** How to interpret the scores in the Waymo Challenge?
  * Our interpretation is that our model can achieve good performance in the distance error metrics (FDE, ADE), but is worse in the mAP score than SOTA methods. Consequently, we view our model's performance as competitive, rather than SOTA. Please note that achieving SOTA performance is not the focus of our work. We have rather shown performances for reference since we also do not investigate ensembling, which is required to achieve a mAP of 0.4764 (6 models with 128 proposals each, vs. 1 model with 6 proposals in our approach).

Reviewer comment (Novelty):
Clarification is needed on how the proposed architecture improves over Wayformer. While it would be acceptable if any architecture were chosen to study self-supervised pre-training, since the architecture design is emphasized as a contribution, the reviewer wants to understand that contribution better.

Author response:
* Q1: How are road environment descriptors (RED) different from latent queries in Wayformer?

The methods are related, yet they primarily differ in two aspects: firstly, in their respective positions within the model, and secondly, in their intended purposes. Our RED are learned by an internal decoder and serve as input for a motion head, while latent queries form a part of the transformer decoder that serves as the motion head.

RED provide environment context (map and current agent state) and are learned with self-supervised learning, while latent queries can be understood as trajectory anchors and are learned using supervised learning.

Building on this difference, we have analyzed the effect of combining RED and latent queries. Specifically, we implemented a version of our model named RedMotion tra-dec, which uses a transformer decoder as the motion head. In this way, the RED are learned via redundancy reduction, thereby providing context that is subsequently transformed into trajectory proposals by the latent queries (refer to Sec. 4.2 for results).

* **Issue 1:** Latent querys interpretation and issues with the Wayformer architecture.
  * We don’t think that there is any issue with the Wayformer architecture, but we see key difference to our architecture and modules. In our opinion, there are two forms of latent queries used in the Wayformer architecture: learned queries in the encoder and in the decoder.
    * In the decoder (Sec. 3.3 Wayformer paper, learned seeds in Fig. 1) learned queries are used to attend to the encoded features and transform them into trajectory proposals. We think that these are inspired by the “learned anchors” from Waymo’s previous works [MultiPath](https://arxiv.org/abs/1910.05449) or [MultiPath++](https://arxiv.org/abs/2111.14973v3). The key difference in our model, is that these elements are part of the motion head. They are learned through supervised learning and can be integrated into our model (tra-dec config).
    * In the encoder(s) (Sec. 3.2 Wayformer paper) learned queries are used to reduce the computational cost for processing long input sequences. These are inspired by the [Set Transformer](https://proceedings.mlr.press/v97/lee19d.html) and [Perceiver](https://proceedings.mlr.press/v139/jaegle21a.html) architectures. In contrast, we are using [local attention](https://arxiv.org/abs/2004.05150) in our road environment encoder to lower the computational complexity (see Sec. 3.2 or Fig. 4 in our work) and a memory efficient implementation of cross attention to fuse features (see Fig. 5).

Reviewer comment (Theory):
A more in-depth explanation of the principle behind the pre-training task is needed. While Table 2 suggests it is effective, it is unclear why this pre-training objective makes sense for learning good representations of map information for motion prediction.

Author response:
* Q2: Is a map encoding that is invariant to small rotations and translations undesirable for trajectory prediction?

In self-supervised pre-training, we learn similar embeddings for moderately augmented environments as correctly described by you. During subsequent fine-tuning, we use these embeddings as "good initialization" to learn trajectory prediction. At this phase, we do not augment environments anymore to learn representations that are responsive to translation and rotation.

Similarly, self-supervised methods in computer vision typically learn augmentation invariant features during pretraining, while during fine-tuning for object detection augmentation-sensitive features are learned.