Paper: Trajeglish: Traffic Modeling as Next-Token Prediction (ICLR-2024)

Reviewer comment (Experiments):
motivation of using tokenization (compared with using the actual values as in most of existing work in Appendix B) is not very clear.

Author response:
“motivation of using tokenization (compared with using the actual values as in most of existing work in Appendix B) is not very clear” + “What’s the motivation of using a small vocabulary compared with using the actual values as in most of existing work (as in Appendix B)?” - Our motivation stems from the impressive results we’ve seen from applying discrete sequence models in other continuous domains, for instance Paella [1], Stable Diffusion [2] Parti [3], and GAIA-1 [4]. We share the intuition behind these works that sacrificing a small amount of resolution (e.g. a few centimeters in our case as reported in Figure 12 and Figure 13) can be well worth the expressiveness and stability of the categorical distribution compared to a mixture-of-x parameterization, for instance. Empirically, this tradeoff appears to be well worth exploring, as our method outperforms all prior work based on diffusion, VAEs, and mixture-of-x modeling on the WOMD sim agents benchmark [(see our note above)](https://openreview.net/forum?id=Z59Rb5bPPP&noteId=VvSB6yXfHY).

Reviewer comment (Evaluation):
Improvements in Table 1 seem quite small. Can you show standard deviations for the results?

Author response:
“Improvements in Table 1 seem quite small. Can you show standard deviations for the results?” We would like to highlight that for the mADE and ADE reported in Table 1, the increase in performance due to intra-timestep conditioning and tokenization regularization is quite significant, improving by 6% and 2.3% respectively. We additionally quantify the performance gap due to these modeling choices in Figure 9 and Figure 10 which show significant improvements when intra-timestep coupling is taken into account. Finally, trajeglish improves on interaction metrics by 4.4% over all prior work on the official WOMD sim agents benchmark, a large jump in performance over all prior work [(see our note above)](https://openreview.net/forum?id=Z59Rb5bPPP&noteId=VvSB6yXfHY).

Reviewer comment (Experiments):
the baseline details are not given (e.g., “The “marginal” baseline is an equally important baseline designed to mimic the behavior of models such as Wayformer (Nayakanti et al., 2022) and MultiPath++ (Varadarajan et al., 2021) that are trained to model the distribution over single agent trajectories instead of multi-agent scene-consistent trajectories.” However, it is unclear if this baseline really can achieve similar performance as Wayformer / MultiPath++ as the authors did not give further details) and it is hard for one to assess if they are really strong baselines.

Author response:
“it is unclear if this baseline really can achieve similar performance as Wayformer / MultiPath++ as the authors did not give further details) and it is hard for one to assess if they are really strong baselines” - In [our note above](https://openreview.net/forum?id=Z59Rb5bPPP&noteId=VvSB6yXfHY), we report results on the WOMD sim agents benchmark which features an official submission for Wayformer as well as a submission based on MultiPath++. Trajeglish improves over Wayformer by 9.2%, and MultiPath++ by 5.4%. We hope these direct comparisons are convincing for the reviewer. We are also updating our draft to include more details about our implementation of the “no-intra” and “marginal” baselines referenced in Table 1.

Reviewer comment (Novelty):
motivation of having a model that take order into account is not very convincing.

Author response:
“motivation of having a model that take order into account is not very convincing” - We note that the focus of our work is on how we apply discrete sequence modeling to multi-agent trajectories, and less about the design of our encoder for scene context. We are aware of prior work that studies how to adjust the attention mechanism to maintain permutation equivariance while still leveraging tracking information, such as AgentFormer [5]. However, we find we are able to achieve state of the art performance for traffic modeling by simply randomizing the order during training, so we reserve the improvement of our architecture for encoding scene context for later work.

Reviewer comment (Novelty):
missing some relevant work on multi-agent trajectory prediction:  
Hivt: Hierarchical vector transformer for multiagent motion prediction, Z. Zhou, L. Ye, J. Wang, K. Wu, and K. Lu.  
Language-Guided Traffic Simulation via Scene-Level Diffusion, Z. Zhong, D. Rempe, Y. Chen, B. Ivanovic, Y. Cao, D. Xu, M. Pavone, B. Ray.

Author response:
Additional papers to cite - We note that HiVT studies how mixture-of-x distributions can be used to model single-agent trajectories and “Language-Guided Traffic Simulation via Scene-Level Diffusion” focuses on how text can be used to control generation of a static traffic scenario, so we don’t consider them directly relevant to our work which studies how discrete sequence models can be applied to closed-loop traffic simulation. However, if the reviewer feels strongly that these are relevant, we can add these citations.

Reviewer comment (Experiments):
What’s the motivation of using a small vocabulary compared with using the actual values as in most of existing work (as in Appendix B)?

Author response:
“motivation of using tokenization (compared with using the actual values as in most of existing work in Appendix B) is not very clear” + “What’s the motivation of using a small vocabulary compared with using the actual values as in most of existing work (as in Appendix B)?” - Our motivation stems from the impressive results we’ve seen from applying discrete sequence models in other continuous domains, for instance Paella [1], Stable Diffusion [2] Parti [3], and GAIA-1 [4]. We share the intuition behind these works that sacrificing a small amount of resolution (e.g. a few centimeters in our case as reported in Figure 12 and Figure 13) can be well worth the expressiveness and stability of the categorical distribution compared to a mixture-of-x parameterization, for instance. Empirically, this tradeoff appears to be well worth exploring, as our method outperforms all prior work based on diffusion, VAEs, and mixture-of-x modeling on the WOMD sim agents benchmark [(see our note above)](https://openreview.net/forum?id=Z59Rb5bPPP&noteId=VvSB6yXfHY).

Reviewer comment (Presentation):
The provided video is a bit confusing. How do you control other vehicles that are neither replay nor trajeglish? In some videos the legends only show these two types but there are vehicles of other colors.

Author response:
“The provided video is a bit confusing. How do you control other vehicles that are neither replay nor trajeglish?” - All agents in our visualization and evaluation are either controlled by replay or trajeglish. For the left 3 panes of the video, we control all vehicles in the scene with trajeglish and title these panes “Full Autonomy '' to indicate the setting. These are the rollouts we use to evaluate our method in Table 1, Figure 9, and the WOMD sim agents benchmark (see our note to all reviewers). For the rightmost pane, only the blue agent is controlled by trajeglish and the rest are on replay, as we denote in the legend. These are the rollouts we use to evaluate our method in Figure 10.

    To be clear, our ultimate goal is to control agents in simulation such that they interact in realistic ways to any number of agents controlled by a black-box AV system. In the absence of a black-box AV system at hand to test, we use replay as a surrogate black-box controller. The purpose of our experiments in the "partial control setting" is to demonstrate the property of trajeglish agents that they interact with other agents in a closed-loop setting independent of the underlying method that controls the other agents.

Reviewer comment (Evaluation):
Figure 9 why the collision rate decreases when the rollout becomes longer?

Author response:
“Figure 9 why the collision rate decreases when the rollout becomes longer?” - the reason is that for the rollouts evaluated in Figure 9, we adjust the padding of the WOMD scenarios to prevent agents from reappearing if they ever disappear. As a result, the number of agents in the scene decreases over time, so the number of collisions also decreases over time. We use this setting for evaluation because our tokenizer relies on the existence of the previous state in order to tokenize, so we are unable to tokenize states that do not have a recorded state in the timestep before. On the other hand, for the WOMD sim agents benchmark [(see our note above)](https://openreview.net/forum?id=Z59Rb5bPPP&noteId=VvSB6yXfHY), we evaluate in the setting where all agents visible at the tenth timestep need to be simulated for all 80 timesteps into the future, as required by the benchmark. We will add these details on how padding is taken into account in each of our evaluation settings to our paper.

Reviewer comment (Experiments):
For the experiments, 16 scenarios are sampled for every clip in WOMD? Or only 16 clips in total?

Author response:
“For the experiments, 16 scenarios are sampled for every clip in WOMD? Or only 16 clips in total?” 16 scenarios are sampled per clip for each of the 44k WOMD validation clips.

Reviewer comment (Presentation):
Can you also provide some qualitative visualization for nuscenes in Appendix?

Author response:
“Can you also provide some qualitative visualization for nuscenes in Appendix?” Yes, we will update our paper before the end of the discussion period.

Reviewer comment (Novelty):
Similar contributions as the recently published “MotionLM: Multi-Agent Motion Forecasting as Language Modeling” (https://arxiv.org/pdf/2309.16534.pdf)

Author response:
"Similar contributions as the recently published MotionLM" + “How does your approach compare to MotionLM?” - The main difference between our work and MotionLM is the application they target; MotionLM targets online deployment, and therefore models 1- and 2-agent (x,y) trajectories at 2hz. Since we target offline simulation of agents, we model up to 24-agent (x,y,h) trajectories at 10hz. The differences between the distributions that these methods seek to model give rise to differences in tokenization schemes; MotionLM discretizes the second-order derivatives of motion in a global coordinate frame, whereas we leverage the fact that we decode heading to discretize first-order derivatives of motion in the per-timestep Frenet frame of the agent. As a practical consideration, the difference in target distribution also results in significantly different sequence lengths modeled between the two approaches; sequences modeled by MotionLM are of length 2 agents * 16 timesteps = 32, while sequences modeled by trajeglish are of length 24 agents * 63 timesteps = 1512. Finally, while temporal causality is shown to be a useful inductive bias for MotionLM and the attention mask used by the autoregressive decoder does not take into account intra-timestep conditioning, our goal in Section 2 is to show that temporal causality is actually required for our application of controlling reactive agents in simulation, and we explicitly measure in Figures 9, 10, and 11 to what extent intra-timestep conditioning affects performance, especially when only a single timestep of trajectory context is provided at initialization.

    While we plan to add a citation for MotionLM to our paper, we would also like to make it clear that MotionLM was released on arxiv on September 28, the same day that we submitted our work to ICLR. As a result, we hope that the reviewer recognizes our work and MotionLM as fully concurrent successful applications of discrete sequence modeling to two different variants of motion prediction.

Reviewer comment (Experiments):
As a result of this simpler benchmark and missing comparisons to any prior architecture, this paper does not address the key question of whether the proposed method is competitive to the current state-of-the-art despite its simplicity. At a glance, it seems to be much worse, with a minADE >3m in comparison to the SoTA methods with minADE < 1m on the more challenging standard WOMD setting.

Author response:
“At a glance, it seems to be much worse, with a minADE >3m in comparison to the SoTA methods with minADE < 1m” - The minADE in Table 1 is calculated by sampling scenarios given only 1 timestep of ground-truth history. We expect the optimal minADE for this setting to be significantly higher than the optimal minADE when 11 timesteps are given as input, as is the case for all 3 WOMD benchmarks related to motion prediction. For an apples-to-apples comparison of minADE to prior work, we report a minADE of 1.68 meters on the WOMD sim agents benchmark (see our [note to all reviewers](https://openreview.net/forum?id=Z59Rb5bPPP&noteId=VvSB6yXfHY)), which is competitive with that of other methods, although we did not optimize hyperparameters for this particular metric since it is not included in the benchmark’s ranking metric. In the paper, we will be more precise about the evaluation setting we use for Table 1.

Reviewer comment (Presentation):
The paper is not self-contained, with important details (e.g., related work and several figures referenced during discussions in the main paper) only available in the appendix

Author response:
“The paper is not self-contained, with important details (e.g., related work and several figures referenced during discussions in the main paper) only available in the appendix” + “The clarity of Figure/Table captions and their placement within the document could be improved” - We appreciate this feedback. We are reorganizing the paper to make sure the related work and main figures are included in the body of the paper. We will also edit the captions for clarity, and post an updated version before the end of the discussion period.