Paper: Don't Reinvent the Steering Wheel (ICLR-2024)

Reviewer comment (Novelty):
This paper talks about the advantages of using the bicycle kinematic model as the prior for trajectory prediction. However, none of the three proposed formulations used the actual bicycle kinematic model. From the result in Table 1, Formulation 1 has the best performance, and Formulation 1 is simply to predict deltas between waypoints.

Author response:
- “None of the three proposed formulations used the actual bicycle kinematic model.” 

We performed derivations and experiments in increasing order of complexity (first-order formulations involving velocity and heading first, then second-order formulations). We will include further results centered around steering and acceleration in future revisions.

Reviewer comment (Evaluation):
The evaluation result lacks some important information. The tables only show the relative improvement from the MTR baseline. However, it is not clear whether the MTR numbers are from the MTR paper, the WOD leaderboard, or from the authors' own implementation. I highly recommend the authors submit their solution to the WOD leaderboard and report the numbers from the leaderboard.

Author response:
- “The tables only show the relative improvement from the MTR baseline. However, it is not clear whether the MTR numbers are from the MTR paper, the WOD leaderboard, or from the authors’ own implementation.” 

The baseline MTR numbers represent our own re-training of a downscaled MTR model, whose absolute performance results will be included in future revision. Here are the absolute baseline numbers for your convenience: 

Baseline (Downscaled and re-trained MTR)

mAP - 0.3373

minADE - 0.6734

minFDE - 1.4153

MissRate - 0.1712

Original MTR (Full-size, not re-trained)

mAP - 0.4129

minADE - 0.6050

minFDE - 1.2207

MissRate - 0.1351

Reviewer comment (Theory):
I am not sure sigma_{x + vx * dt} = sigma_x + sigma_{vx} * dt is correct. The summation of two Gaussian variables x + y should be sigma_{x + y} = \sqrt{sigma_x ^ 2 + sigma_y ^ 2}.

Author response:
- “I am not sure sigma_{x + vx * dt} = sigma_x + sigma_{vx} * dt is correct. The summation of two Gaussian variables x + y should be sigma_{x + y} = \sqrt{sigma_x ^ 2 + sigma_y ^ 2}.”

Thank you for catching this! We will fix this and update the paper.

Reviewer comment (Novelty):
Lack of novelty / contributions. Trajectory forecasting with kinematics models exists in literature. The given kinematic model derivations are quite straightforward.

Author response:
Thank you for your suggestions. We clarify how we approach kinematic priors in our method differently from existing work in the general response above, and will make sure to incorporate revisions accordingly in the paper. We agree that the idea of using kinematic priors is not new to trajectory forecasting tasks, and thus will re-frame the contributions accordingly, as we believe previous work considered kinematic models *deterministically* rather than *probabilistically*, which remains as our key contribution over prior works and the key differentiator.  This formulation offers several benefits as we stated above in the general response.

While the kinematic model derivations are straightforward, the additional novelty is the consideration of the relationship between variances of kinematic variables and resulting trajectories. Mean kinematic variables and mean trajectories are considered in previous work directly, but the *disadvantage is the loss of expressiveness reflected in trajectory distributions when assuming a Gaussian model of decision making*.   *This issue* is what we address in the paper.

Hopefully this answers some of your questions, and thank you for your review!

Reviewer comment (Novelty):
As mentioned above, kinematics models have been applied for ML trajectory forecasting and hence, the contributions of this paper seem minor.

Author response:
Thank you for your suggestions. We clarify how we approach kinematic priors in our method differently from existing work in the general response above, and will make sure to incorporate revisions accordingly in the paper. We agree that the idea of using kinematic priors is not new to trajectory forecasting tasks, and thus will re-frame the contributions accordingly, as we believe previous work considered kinematic models *deterministically* rather than *probabilistically*, which remains as our key contribution over prior works and the key differentiator.  This formulation offers several benefits as we stated above in the general response.

While the kinematic model derivations are straightforward, the additional novelty is the consideration of the relationship between variances of kinematic variables and resulting trajectories. Mean kinematic variables and mean trajectories are considered in previous work directly, but the *disadvantage is the loss of expressiveness reflected in trajectory distributions when assuming a Gaussian model of decision making*.   *This issue* is what we address in the paper.

Hopefully this answers some of your questions, and thank you for your review!

Reviewer comment (Experiments):
The paper was unclear about the baseline performance. Specifically, did the authors compare to the original MTR, or a re-implemented and retrained version of MTR? I found the following statements in the paper:  
> We implement kinematic priors on state-of-the-art method Motion Transformer (MTR) Shi et al. (2022), which serves as our baseline method.  

> ... we downscale the model from its original size of 65 million parameters to 2 million parameters and reproduce baseline results in our experiments  

> We compare our model against the state-of-the-art architecture...

In all tables in both the main paper and the appendix, the results are written as relative to the baseline, and I could not find absolute numbers for the baseline anywhere. Thus I am lead to believe that the results are for the smaller model, which likely performs worse than the original MTR? If this is true, I believe the authors need to release the absolute performance of their baseline. The reason that the absolute performance is important to me is that it is unclear whether the gains of the kinematic model hold when absolute performance increases as you scale the model, etc.

Author response:
- “Specifically, did the authors compare to the original MTR, or a re-implemented and retrained version of MTR?”

We compare it to the re-implemented and re-trained version of MTR, which can be expected to have lower performance due to the sheer difference in number of parameters. We do this mostly due to limitations on available computational resources. 

- “I could not find absolute numbers for the baseline anywhere”. 

We should have included this in the appendix and will add it in revisions. For your convenience, we include the absolute performance of the downscaled baseline here. 
                                                                    
Baseline (Downscaled and re-trained MTR) 

*mAP* - 0.3373

*minADE* - 0.6734	

*minFDE* - 1.4153

*MissRate* - 0.1712

Original MTR (Full-size, not re-trained)
 
*mAP* - 0.4129

*minADE* - 0.6050

*minFDE* - 1.2207

*MissRate* - 0.1351

- “The reason that the absolute performance is important to me is that its unclear whether the gains of the kinematic model hold when absolute performance increases as you scale the model, etc.”

We agree with this intuition, and also observe from preliminary training curves that gains from the kinematic model diminish as the model size increases. However, we believe that state-of-the-art models should also consider models with smaller numbers of trainable parameters for the sake of *practicality and portability*. While the benefit of kinematic priors diminish with greater model sizes, we believe that they will be beneficial in the event that such models need to be downscaled and fit onto a robot. 

We hope this answers some of your questions, and thank you again for the feedback!

Reviewer comment (Writing):
The phrase "but also scales well in size" is unclear if this is a good or a bad thing. The authors should clarify this better.

Author response:
“but also scales well in size”; is this a good thing or bad thing? 

We mention this as a good thing in the context of performance, but also a bad thing in the context of portable autonomous driving models for real-time inference. We observe trends in recent years that SOTA models tend to become larger and larger–combining SOTA from several tasks together for full-stack autonomous driving may prove to be computationally costly if model sizes continue to scale along with performance. We would like to motivate work in the direction of smaller models with more hand crafted kinematic modeling or classical simulation techniques for sake of *portability* or unseen domain situations with data scarcity (while still preserving performance).

We will clarify this confusion in future revisions of the paper.

Reviewer comment (Reproducibility):
The phrase "we downscale the model" is unclear; the authors should explain exactly how this is done.

Author response:
“we downscale the model” how? 

Since MTR is a transformer-based model, we downscale the model directly by reducing the number of heads, number of encoder and decoder layers, and hidden features. The full details on model parameters used for results can be found in Table 8 of the appendix. We will reference this Table in the revision.

Reviewer comment (Experiments):
For the experiments with noisy trajectories, it is unclear if noise is added during training as well. This would be good to clarify.

Author response:
“For the experiments with noisy trajectories, do you add noise during training as well?”

For experiments regarding noisy trajectories, we do not add noise during training. The experiments were inspired by the scenario where training data is clean and well-processed, but sensors may be noisy in the real world. This experiment examined the robustness of our model in comparison to direct prediction on positional waypoint trajectories.