Original: Exploring the Temporal Constraint for Trajectory Prediction Anonymous Author(s) AffiliationAddress email
Revised: Towards Capturing the Temporal Dynamics for Trajectory Prediction: a Coarse-to-Fine Approach

Original: Abstract:
Revised: Abstract:

Original: Trajectory prediction is one of the basic tasks in the autonomous driving field, which aims to predict the future position of other agents around the egovehicle so that a safe yet efficient driving plan could be generated in the downstream module.
Revised: Trajectory prediction is one of the basic tasks in the autonomous driving ﬁeld, which aims to predict the future position of other agents around the ego vehicle so that a safe yet efﬁcient driving plan could be generated in the downstream module.

Original: Recently, deep learning based methods dominate the field.
Revised: Recently, deep learning based methods dominate the ﬁeld.

Original: Stateof-the-art (SOTA) methods usually follow an encoder-decoder paradigm.
Revised: Stateof-the-art (SOTA) methods usually follow an encoder-decoder paradigm.

Original: Specifically, the encoder is responsible for extracting information from agents’ historystates and HD-Map and providing a representation vector for each agent.
Revised: Specifically, the encoder is responsible for extracting information from agents’ history states and HD-Map and providing a representation vector for each agent.

Original: Takingthese vectors as input, the decoder predicts multi-step future positions for eachagent, which is usually accomplished by a single multi-layer perceptron (MLP)to directly output a Tx2 tensor.
Revised: Taking these vectors as input, the decoder predicts multi-step future positions for each agent, which is usually accomplished by a single multi-layer perceptron (MLP) to directly output a Tx2 tensor.

Original: Though models with adoptation of MLP decoderhave dominated the leaderboard of multiple datasets, ‘the elephant in the room’is that the temporal correlation among future time-steps is ignored since there is no direct constraint between output neurons of a MLP.
Revised: Though models with adoptation of MLP decoder have dominated the leaderboard of multiple datasets, ‘the elephant in the room is that the temporal correlation among future time-steps is ignored since there is no direct relation among output neurons of a MLP.

Original: In this work, we examinethis design choice and investigate several ways to apply the temporal inductivebias into the generation of future trajectories on top of a SOTA encoder.
Revised: In this work, we examine this design choice and investigate several ways to apply the temporal inductive bias into the generation of future trajectories on top of a SOTA encoder.

Original: We findthat simply using autoregressive RNN to generate future positions would lead tosignificant performance drop even with techniques such as history highway andteacher forcing.
Revised: We ﬁnd that simply using autoregressive RNN to generate future positions would lead to signiﬁcant performance drop even with techniques such as history highway and teacher forcing.

Original: Instead, taking scratch trajectories generated by MLP as input,an additional refinement module based on structures with temporal prior such as
Revised: Instead, taking scratch trajectories generated by MLP as input, an additional reﬁnement module based on structures with temporal prior such as

Original: RNN or 1D-CNN could remarkably boost the accuracy.
Revised: RNN or 1D-CNN could remarkably boost the accuracy.

Original: Furthermore, we examineseveral objective functions to emphasize the temporal constraint.
Revised: Furthermore, we examine several objective functions to emphasize the temporal priors.

Original: By the combination of aforementioned techniques to introduce the temporal prior, we improvethe top-ranked method’s performance by a large margin and achieve SOTA resulton the Waymo Open Motion Challenge.
Revised: By the combination of aforementioned techniques to introduce the temporal prior, we improve the topranked method’s performance by a large margin and achieve SOTA result on the Waymo Open Motion Challenge.

Original:  Introduction
Revised:  Introduction

Original: Trajectory prediction aims to predict agents’ positions in the next several seconds, which is oneof the essential tasks for the autonomous driving.
Revised: Trajectory prediction aims to predict agents’ positions in the next several seconds, which is one of the essential tasks for the autonomous driving.

Original: Recently, deep learning based methods havedominated the field and they usually follow an encoder-decoder paradigm [1, 2].
Revised: Recently, deep learning based methods have dominated the ﬁeld and they usually follow an encoder-decoder paradigm [1, 2].

Original: The encoder takesall agents’ history states (position, velocity, heading, etc .) and HD-Map (lanes, road lines, trafficlights, etc .)
Revised: The encoder takes all agents’ history states (position, velocity, heading, etc .) and HD-Map (lanes, road lines, trafﬁc lights, etc .)

Original: as inputs and outputs a representation vector for each agent.
Revised: as inputs and outputs a representation vector for each agent.

Original: The decoder then generatesfuture trajectories for the agents based on these vectors.
Revised: The decoder then generates future trajectories for the agents based on these vectors.

Original: We observe that current top-rankedmethods on multiple datasets 1 all adopt an MLP as the decoder to output a T × 2 tensor andthen reshape it into the final prediction .
Revised: We observe that current top-ranked methods [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14] on multiple datasets 1 all adopt an MLP as the decoder to output a T × 2 tensor and then reshape it into the ﬁnal prediction .

Original: Though the MLP design achieves satisfying performance empirically, we point out that it ignoresthe temporal correlation among the future time-steps.
Revised: Though the MLP design achieves satisfying performance empirically, we point out that it ignores the temporal correlation among the future time-steps.

Original: Mathematically, we denote the hidden representation vector of an agent as H ∈ R H and its predicted future trajectory as τ ∈ R T × 2 where Tindicates the length of prediction horizon and 2 indicates the 2D coordinate.
Revised: Mathematically, we denote the hidden representation vector of an agent as H ∈ R H and its predicted future trajectory as τ ∈ R T × 2 where T indicates the length of prediction horizon and 2 indicates the 2D coordinate.

Original: The MLP accomplishesthe mapping:
Revised: The MLP accomplishes the mapping:

Original: [Equation]
Revised: [Equation]

Original: where W 1 ∈ R H′
Revised: where W 1 ∈ R H

Revised: [Equation]

Original: × H and W 2 ∈ R (2 T ) × H ′ are weight matrices, and δ ( · ) is a non-linear function.
Revised: × H and W 2 ∈ R (2 T ) × H (cid:48) are weight matrices, and δ ( · ) is a non-linear function.

Original: We can find that every two rows of W 2 contains the weights to generate the position for one specifictime-step.
Revised: We can ﬁnd that every two rows of W 2 contains the weights to generate the position for one speciﬁc time-step.

Original: Since aforementioned SOTA works adopt either per-step negative log likelihood (NLL)loss or per-step mean squared error (MSE) loss, the back-propagation updating for every two rowsof W 2 is independent.
Revised: Since aforementioned SOTA works adopt either per-step negative log likelihood (NLL) loss or per-step mean squared error (MSE) loss, the back-propagation updating for every two rows of W 2 is independent.

Original: Such design is based on the assumption of conditional independence amongfuture time-steps.
Revised: Such design is based on the assumption of conditional independence among future time-steps.

Original: However, the assumption is untenable in real-world scenarios and thus could beharmful for the prediction.
Revised: However, the assumption is untenable in real-world scenarios and thus could be harmful for the prediction.

Original: In this work, we aim to discuss about ‘the elephant in the room’ and explore several approaches tomodel the temporal correlation for future time-steps on top of a SOTA encoder.
Revised: In this work, we aim to discuss about ‘the elephant in the room’ and explore several approaches to model the temporal correlation for future time-steps on top of a SOTA encoder.

Original: We find that simplyreplacing the MLP with an autoregressive RNN leads to significant performance drop, even withtechniques such as teacher forcing [18] or history highway [19] to alleviate its drawbacks of longterm dependencies and cumulative error.
Revised: We ﬁnd that simply replacing the MLP with an autoregressive RNN leads to signiﬁcant performance drop, even with techniques such as teacher forcing [18] or history highway [19] to alleviate its drawbacks of long-term dependencies and cumulative error.

Original: However, when first using an MLP to generate a scratchtrajectory and then using a structure with temporal inductive bias (RNN/1D-CNN) to refine it, the SOTA model’s performance could be boosted.
Revised: Inspired by the success of reﬁne modules in the deep learning based methods [20, 21, 22, 23] which ease the optimization difﬁculty of a single decoder head, we ﬁnd that when ﬁrst using an MLP to generate a scratch trajectory and then using a structure with temporal inductive bias (RNN/1D-CNN) to reﬁne it, the SOTA model’s performance could be boosted .

Original: We also explore different training objectives to applytemporal constraints on the output besides modifications of the neural network structure.
Revised: We also explore different training objectives to apply temporal priors on the output besides modiﬁcations of the neural network structure.

Original: We findthat directly fitting the velocity degenerates the performance, while accumulating the velocity intothe displacement ( i.e. , calculating the integral of velocity over time) and still using the coordinatesas the target would lead to better prediction.
Revised: We ﬁnd that directly ﬁtting the velocity degenerates the performance, while accumulating the velocity into the displacement ( i.e. , calculating the integral of velocity over time) and still using the coordinates as the target would lead to better prediction.

Original: By combining these discovered effective techniques,we significantly boost the performance of a SOTA model on Waymo Open Motion Dataset, whichverifies the benefit of utilizing the temporal prior.
Revised: By combining these discovered effective techniques, we signiﬁcantly boost the performance of a SOTA model on Waymo Open Motion Dataset, which veriﬁes the beneﬁt of utilizing the temporal prior.

Original: In summary, the contributions of the paper are:
Revised: In summary, the contributions of the paper are:

Original: • We explore ways to incorporate the temporal structure in the decoder and propose an effective scratch-then-temporally-refine paradigm.
Revised: • We explore ways to incorporate the temporal structure in the decoder and propose an effective scratch-then-temporally-reﬁne paradigm.

Original: By using an MLP and a temporal modulein two steps respectively, it explicitly models the temporal correlation among future timesteps without the burden of training an autoregressive RNN.• We examine several objective functions to apply temporal constraints on the output.
Revised: By using an MLP and a temporal module in two steps respectively, it explicitly models the temporal correlation among future time steps without the burden of training an autoregressive RNN. • We examine several objective functions to add temporal priors to the output.

Original: Wereport the explicit improvements by first generating velocity and then accumulating temporally into coordinates to calculate per-step loss.
Revised: We report the notable improvements by ﬁrst generating velocity and then accumulating temporally into coordinates to calculate per-step loss.

Original: • We conduct thorough ablation study on the Waymo Open Motion Dataset to test different ways of introducing the temporal prior into the decoding process.
Revised: • We conduct thorough ablation study on the Waymo Open Motion Dataset to test different ways of introducing the temporal prior into the decoding process.

Original: With the combination ofour effective attempts, the performance of a SOTA encoder is boosted by a large margin onWaymo Public Leaderboard.
Revised: With the combination of our effective attempts, the performance of a SOTA encoder is boosted by a large margin on Waymo Public Leaderboard.

Original: We consider the correlation among future time steps as a non-negligible factor, which is not exploitedby current SOTA methods.
Revised: We consider the correlation among future time steps as a non-negligible factor, which is not exploited by current SOTA methods.

Original: We hope the attempts and successful parts illustrated in this study couldprovide useful information for this line of study in the trajectory prediction community.
Revised: We hope the attempts and successful parts illustrated in this study could provide useful information for this line of study in the trajectory prediction community.

Original:  Related Works
Revised:  Related Works

Original: Pioneering works of deep learning based trajectory prediction models usually adopt an RNN decoderto generate future trajectories in an autoregressive way.
Revised: Pioneering works of deep learning based trajectory prediction models usually adopt an RNN decoder to generate future trajectories in an autoregressive way.

Original: Social LSTM [20] applies an LSTM decoderafter the LSTM encoder, and adopts the NLL loss for each time-step.
Revised: Social LSTM [24] applies an LSTM decoder after the LSTM encoder, and adopts the NLL loss for each time-step.

Original: Its extension, Social GAN [21],uses GAN [22] to encode different modalities and adopts per-step MSE loss.
Revised: Its extension, Social GAN [25], uses GAN [26] to encode different modalities and adopts per-step MSE loss.

Original: DESIRE [23] employsa conditional variational autoencoder to sample a diverse set of hypothetical modalities and adoptsa GRU to generate future predictions.
Revised: DESIRE [27] employs a conditional variational autoencoder to sample a diverse set of hypothetical modalities and adopts a GRU to generate future predictions.

Original: MFP [24] introduces a latent distribution to model the joint modality of multiple agents and utilizes an RNN for decoding as well.
Revised: MFP [28] introduces a latent distribution to model the joint modality of multiple agents and utilizes an RNN for decoding as well.

Original: Mulitpath [25] leveragesa fixed set of future state-sequence anchors that correspond to modes of the trajectory distributionand regresses offsets along with uncertainties with respect to anchor waypoints by RNN.
Revised: Mulitpath [29] leverages a ﬁxed set of future state-sequence anchors that correspond to modes of the trajectory distribution and regresses offsets along with uncertainties with respect to anchor waypoints by RNN.

Revised: IDE-Net [30] applies additional loss to infer the interaction types in an unsupervised fashion.

Original:  Trajecton++ [26] utilizes a GRU as the decoder to predict a bivariate Gaussian distribution over controlactions.
Revised: To capture the temporal dynamics, Trajecton++ [31] utilizes a GRU as the decoder to predict a bivariate Gaussian distribution over control actions.

Revised: By applying dynamically-extended unicycle models over the predicted control signals, it is able to guarantee that its trajectory samples are dynamically feasible.

Original: DKM [27] designs an autoregressive differentiable dynamics layer to generate the controlsignal per-step.
Revised: DKM [32] designs an autoregressive differentiable dynamics layer to generate the control signal per-step.

Original: Though it is intuitive to use an autoregressive RNN to generate the sequence of future positions dueto its recurrent nature, modern methods with a concise MLP decoder outperform previous RNNbased ones and dominate multiple public motion prediction leaderboards, including Waymo [10],nuScenes [17], Argoverse [5] and INTERACTION [14].
Revised: Though it is intuitive to use an autoregressive RNN to generate the sequence of future positions due to its recurrent nature, modern methods with a concise MLP decoder outperform previous RNNbased ones and dominate multiple public motion prediction leaderboards, including Waymo [9], nuScenes [17], Argoverse [15] and INTERACTION [16].

Original: [28] utilizes CNN to encode the scene andgenerates trajectories directly by an MLP, while LaneGCN [2] encodes the scene by four distinctgraph convolutional network (GCN) layers.
Revised: [33] utilizes CNN to encode the scene and generates trajectories directly by an MLP, while LaneGCN [2] encodes the scene by four distinct graph convolutional network (GCN) layers.

Original: PGP [15] traverses the lane graph to incorporate the lanetopology while LaPred [16] applies attention mechanism on lanes.
Revised: PGP [13] traverses the lane graph to incorporate the lane topology while LaPred [14] applies attention mechanism on lanes.

Original: HiVT [4] designs a hierarchicalway to aggregate the local and global information efficiently.
Revised: HiVT [4] designs a hierarchical way to aggregate the local and global information efﬁciently.

Original: SceneTransformer [7] encodes thespatial and temporal information in a factorized way.
Revised: SceneTransformer [6] encodes the spatial and temporal information in a factorized way.

Original: HDGT [9] models the driving scene as aheterogeneous graph.
Revised: HDGT [8] models the driving scene as a heterogeneous graph.

Original: TPCN [29] and its extension DCMS [3] encodes the scene in the form of pointcloud and apply an MLP on the pooled instance vector to generate future trajectories.
Revised: TPCN [34] and its extension DCMS [3] encodes the scene in the form of point cloud and apply an MLP on the pooled instance vector to generate future trajectories.

Original: VectorNet [1]encodes both the agents and lanes as vectors on a global graph and decodes the future trajectoriesbased on the corresponding agents’ vectors with an MLP.
Revised: VectorNet [1] encodes both the agents and lanes as vectors on a global graph and decodes the future trajectories based on the corresponding agents’ vectors with an MLP.

Original: Its extension TNT [30] and DenseTNT [8]generate trajectories in a two-stage way: the stage one aims to predict the target/goal of agentsand the stage two aims to complete the entire trajectories conditioned on the predicted end pointby an MLP.
Revised: Its extension TNT [35] and DenseTNT [7] generate trajectories in a two-stage way: the stage one aims to predict the target/goal of agents and the stage two aims to complete the entire trajectories conditioned on the predicted end point by an MLP.

Original: GOHOME [12] and its extension THOMAS [13] output heatmaps for goals and thensample trajectories based on the selected goals.
Revised: GOHOME [11] and its extension THOMAS [12] output heatmaps for goals and then sample trajectories based on the selected goals.

Original: LaneRCNN [31] follows the goal-based predictionframework as well but they complete the trajectories with a polynomial prior and use an MLP torefine the polyline.
Revised: LaneRCNN [36] follows the goal-based prediction framework as well but they complete the trajectories with a polynomial prior and use an MLP to reﬁne the polyline.

Original: In PLOP [32], means of the distribution of future coordinates are generated inpolynomials of degree 4 of time.
Revised: In PLOP [37], means of the distribution of future coordinates are generated in polynomials of degree 4 of time.

Original: In the recent study - Multipath++ [6], they find that both polynomialrepresentations and control signal representations lead to degenerated performance compared todirectly regressing the coordinates.
Revised: In the recent study - Multipath++ [5], they ﬁnd that both polynomial representations and control signal representations lead to degenerated performance compared to directly regressing the coordinates.

Original: Revisiting the existing literature, we can find that decoders and training objectives with differenttypes of temporal constraint are proposed.
Revised: Revisiting the existing literature, we can ﬁnd that decoders and training objectives with different types of temporal structure are proposed.

Original: However, they are usually entangled with the encodermodule and other settings, which makes it hard to draw a clear conclusion about the advantagesand disadvantages of these designs.
Revised: However, they are usually entangled with the encoder module and other settings, which makes it hard to draw a clear conclusion about the advantages and disadvantages of these designs.

Original: In this work, we examine different choice of applying temporalconstraints in the generation process of future trajectories while keep other settings such as thedataset, the encoder, the training schedule, and the hyperparameter the same for a fair comparison.
Revised: In this work, we examine different choice of applying temporal structure in the generation process of future trajectories while keep other settings such as the dataset, the encoder, the training schedule, and the hyperparameter the same for a fair comparison.

Original: To the best of our knowledge, we are the first to explore this design choice with the modern vectorbased encoder on large-scale autonomous driving datasets.
Revised: To the best of our knowledge, we are the ﬁrst to explore this design choice with the modern vector-based encoder on large-scale autonomous driving datasets.

Original:  Problem Formulation
Revised:  Problem Formulation

Original: Given history states { S ihis | i = 1 , ..., N } of N agents such as their coordinate, velocity, and headingand their surrounding information S env including lanes, traffic lights, etc ., the goal is to predict theirfuture positions in the next T time-steps { τ i ∈ R T × 2 | i = 1 , ..., N } .
Revised: Given history states { S ihis | i = 1 , ..., N } of N agents such as their coordinate, velocity, and heading and their surrounding information S env including lanes, trafﬁc lights, etc ., the goal is to predict their future positions in the next T time-steps { τ i ∈ R T × 2 | i = 1 , ..., N } .

Original: Current SOTA methods usually follow the encoder-decoder paradigm.
Revised: Current SOTA methods usually follow the encoder-decoder paradigm.

Original: The encoder takes all agents’ histories and environmentinformation as inputs and outputs a representation vector H i ∈
Revised: The encoder takes all agents’ histories and environment information as inputs and outputs a representation vector H i ∈

Original: R H for each agent i:
Revised: R H for each agent i:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Conditioned on agents’ hidden feature H i , the decoder aims to predict the distributions of theirfuture trajectories respectively:
Revised: Conditioned on agents’ hidden feature H i , the decoder aims to predict the distributions of their future trajectories respectively:

Original: [Equation]
Revised: [Equation]

Original: Y ti is agent
Revised: Y ti is agent

Original: i ’s distribution of states at future time-step t .
Revised: i ’s distribution of states at future time-step t .

Original: As mentioned in Sec. 1, most SOTA works generate coordinates of different future time-steps withan MLP and adopt a step-wise MSE or NLL loss, which assumes that the distributions of states on different future time-steps are conditionally independent:
Revised: As mentioned in Sec. 1, most SOTA works generate coordinates of different future time-steps with an MLP and adopt a step-wise MSE or NLL loss, which assumes that the distributions of states on different future time-steps are conditionally independent:

Original: [Equation]
Revised: [Equation]

Original: We argue that this assumption ignores the temporal correlation among future time-steps, whichcould result in less consistent and physically infeasible trajectories.
Revised: We argue that this assumption ignores the temporal correlation among future time-steps, which could result in less consistent and physically infeasible trajectories.

Original: In this work, we explore twoparts to apply the temporal constraint: the mapping function ϕ ( H i )
Revised: In this work, we explore two parts to apply the temporal priors: the mapping function φ ( H i )

Original: =
Revised: =

Original: ˆ τ i =
Revised: ˆ τ i =

Original: { ˆ Y ti | t = 1 , ...T } andthe objective function L ( τ i , ϕ ( H i )) .
Revised: { ˆ Y ti | t = 1 , ...T } and the objective function L ( τ i , φ ( H i )) .

Original: Experiments Setting: We examine different decoders and objective functions on the validation setof Waymo Open Motion [10].
Revised: Experiments Setting: We examine different decoders and objective functions on the validation set of Waymo Open Motion [9].

Original: For all models, we adopt the same encoder module - HDGT [9], atop-ranked encoder on Waymo Leaderboard.
Revised: For all models, we adopt the same encoder module - HDGT [8], a top-ranked encoder on Waymo Leaderboard.

Original: All experiments are conducted with the same settingand hyperparameters.
Revised: All experiments are conducted with the same setting and hyperparameters.

Original: Please refer to Sec.
Revised: Please refer to Sec.

Original: 9 for more details.
Revised: 9 for more details.

Original: The following metrics are used: a) minADE (Minimum Average Distance Error): the minimum value of the Euclidean distance betweenthe prediction and ground truth averaged by the prediction length T , for K required predictions.
Revised: The following metrics are used: a) minADE (Minimum Average Distance Error): the minimum value of the Euclidean distance between the prediction and ground truth averaged by the prediction length T , for K required predictions.

Original: b)minFDE (Minimum Final Distance Error): similar to minADE, despite that it only calculates theerror at the final time-step T .
Revised: b) minFDE (Minimum Final Distance Error): similar to minADE, despite that it only calculates the error at the ﬁnal time-step T .

Original: c) MR (Missing Rate): the ratio of whether the Euclidean distancebetween the prediction and ground truth at the final time-step T for all K predictions is larger than2 meters.
Revised: c) MR (Missing Rate): the ratio of whether the Euclidean distance between the prediction and ground truth at the ﬁnal time-step T for all K predictions is larger than 2 meters.

Original:  Temporal Enhanced Decoder
Revised:  Temporal Enhanced Decoder

Original: [Figure]
Revised: [Figure]

Original: 4.1 Autoregressive

Original: [Figure]
Revised: [Figure]

Original: The autoregressive RNN predicts future trajectories in an iterative way with the assumption thatthe coordinate distribution of each time-step depends on that from previous time-steps (for notationbrevity, we drop the subscript i of agents in the following sections):
Revised: The autoregressive RNN predicts future trajectories in an iterative way with the assumption that the coordinate distribution of each time-step depends on that from previous time-steps (for notation brevity, we drop the subscript i of agents in the following sections):

Original: [Equation]
Revised: [Equation]

Original: The generation process is implemented by maintaining an internal state vector in RNN.
Revised: The generation process is implemented by maintaining an internal state vector in RNN.

Original: [Table]
Revised: [Table]

Original: Two commonly used RNN structures are
Revised: Two commonly used RNN structures are

Original: LSTM [33] and GRU [34] and we adopt an
Revised: LSTM [38] and GRU [39] and we adopt an

Original: LSTM here for its high capacity 2 .
Revised: LSTM here for its high capacity 2 .

Original: Meanwhile, there are two widely known drawbacks of RNN: the difficulty to capturelong-term dependency due to the implicitpast information retention mechanism [35]and the cumulative error [18].
Revised: Meanwhile, there are two widely known drawbacks of RNN: the difﬁculty to capture long-term dependency due to the implicit past information retention mechanism [40] and the cumulative error [18].

Original: As forthe first problem, inspired by highway
Revised: As for the ﬁrst problem, inspired by highway

Original: RNN [36, 37], we implement a variant of
Revised: RNN [41, 42], we implement a variant of

Original: LSTM which takes the vector H as an additional input at each time-step so that theLSTM. history information is always accessible.
Revised: LSTM which takes the vector H as an additional input at each time-step so that the history information is always accessible.

Original: To alleviate the cumulative error of autoregressive model,we adopt the teacher forcing with scheduled sampling strategy [18].
Revised: To alleviate the cumulative error of autoregressive model, we adopt the teacher forcing with scheduled sampling strategy [18].

Original: Specifically, at each time-step,the LSTM would have a possibility p GT to take the ground-truth (GT) state as input instead of thepredicted one, and the possibility p GT decays as the training goes by.
Revised: Speciﬁcally, at each time-step, the LSTM would have a possibility p GT to take the ground-truth (GT) state as input instead of the predicted one, and the possibility p GT decays as the training goes by.

Original: In this way, the model couldbe updated with less noise at the start of training while at the end it could learn to robustly do theprediction based on its previous output [18].
Revised: In this way, the model could be updated with less noise at the start of training while at the end it could learn to robustly do the prediction based on its previous output [18].

Original: To futher explore the differences between the RNN and MLP decoder, we design a ‘mix’ of them.
Revised: To futher explore the differences between the RNN and MLP decoder, we design a ‘mix’ of them.

Original: We use an LSTM to autoregressively generates l steps’representation vectors and then use an MLP to predict positions of T/l time-steps with each step’svector as input.
Revised: We use an LSTM to autoregressively generates l steps’ representation vectors and then use an MLP to predict positions of T/l time-steps with each step’s vector as input.

Original: In Figure 1, we give illustrations of the aforementioned decoders.
Revised: In Figure 1, we give illustrations of the aforementioned decoders.

Original: Table 1 is the performance of thedifferent decoder structures on Waymo Motion Dataset (validation set).
Revised: Table 1 is the performance of the different decoder structures on Waymo Motion Dataset (validation set).

Original: We can conclude that:
Revised: We can conclude that:

Original: • Autoregressive RNN performs worse than the MLP by a large margin, which explains why most SOTA methods adopt the latter one.
Revised: • Autoregressive RNN performs worse than the MLP by a large margin, which explains why most SOTA methods adopt the latter one.

Original: • The cumulative error is one cause of autoregressive RNN’s bad performance.
Revised: • The cumulative error is one cause of autoregressive RNN’s bad performance.

Original: The teacher forcing technique could alleviate this problem, but its performance is still worse than MLP.
Revised: The teacher forcing technique could alleviate this problem, but its performance is still worse than MLP.

Original: • Concatenating the history feature H with each time-step’s state as input to RNN could not solve the long-term dependency problem.
Revised: • Concatenating the history feature H with each time-step’s state as input to RNN could not solve the long-term dependency problem.

Original: Indeed, it leads to worse performance than vanilla RNN.
Revised: Indeed, it leads to worse performance than vanilla RNN.

Original: It might come from the difficulty of updating H for all T time-steps as theinput to RNN.
Revised: It might come from the difﬁculty of updating H for all T time-steps as the input to RNN.

Original: • The mix of the two structure has the performance between the MLP and RNN decoder.
Revised: • The mix of the two structure has the performance between the MLP and RNN decoder.

Original: By reducing the autoregressive steps, the optimization becomes easier.
Revised: By reducing the autoregressive steps, the optimization becomes easier.

Original: However, it is still much worse than a concise MLP decoder.
Revised: However, it is still much worse than a concise MLP decoder.

Original: In summary, we find that the inherent issues of autoregressive RNN make it hard to compete with the
Revised: In summary, we ﬁnd that the inherent issues of autoregressive RNN make it hard to compete with the

Original: MLP decoder in the trajectory prediction task.
Revised: MLP decoder in the trajectory prediction task.

Original: We should explore other ways to apply the temporalinductive bias on the network structure.
Revised: We should explore other ways to apply the temporal inductive bias on the network structure.

Original: [Figure]
Revised: [Figure]

Original: Through Section 4.1, we find that directly utilizing RNN to predict trajectories is hard because ithas to generate in an autoregressive fashion, which is troublesome.
Revised: Through Section 4.1, we ﬁnd that directly utilizing RNN to predict trajectories is hard because it has to generate in an autoregressive fashion, which is troublesome.

Original: Thus, to utilize the RNN whileavoiding the autoregressive part, we need to generate proper input states to RNN for each time-step.
Revised: Thus, to utilize the RNN while avoiding the autoregressive part, we need to generate proper input states to RNN for each time-step.

Original: We notice the fact that a simple MLP is proven to be able to do the prediction accurately in theexisting literature, which inspires us to train an MLP to generate scratch trajectories and thenusing structures with temporal prior (RNN or 1D-CNN) to refine the scratch.
Revised: We notice the fact that a simple MLP is proven to be able to do the prediction accurately in the existing literature, which inspires us to train an MLP to generate scratch trajectories and then using structures with temporal prior (RNN or 1D-CNN) to reﬁne the scratch.

Original: Formally, given the hidden feature H from the encoder, we first train an MLP with the output shapeof T × 2 , denoted as ϕ scratch:
Revised: Formally, given the hidden feature H from the encoder, we ﬁrst train an MLP with the output shape of T × 2 , denoted as φ scratch:

Original: [Equation]
Revised: [Equation]

Original: Based on the scratch trajectories { ˆ Y t scratch | t = 1 , ...T } , we train a refinement module ϕ refine torefine them:{ ˆ [Equation] t offset [Equation]
Revised: Based on the scratch trajectories { ˆ Y t scratch | t = 1 , ...T } as well as the feature H from the encoder which encodes agents interactions and map information, we train a reﬁnement module φ reﬁne to reﬁne them:

Original: [Equation]

Original: = ϕ refine ( { ˆ Y t scratch | t

Original: [Equation]
Revised: [Equation]

Original: 1 , ...T } ) , (7) where the objective for Y t offset is the difference between ˆ Y t scratch and the ground-truth Y t .
Revised: where the objective for Y t offset is the difference between ˆ Y t scratch and the ground-truth Y t .

Original: Figure 2 gives an illustration of the proposed paradigm.
Revised: Figure 2 gives an illustration of the proposed paradigm.

Original: [Table]
Revised: [Table]

Original: Note that ϕ scratch and ϕ refine could be trainedin an end-to-end fashion.
Revised: Note that φ scratch and φ reﬁne could be trained in an end-to-end fashion.

Original: In experiments, wehave different choices for ϕ refine: a) another
Revised: In experiments, we have different choices for φ reﬁne: a) another

Original: MLP, similar to [3], which takes the flattenedscratch trajectory as the input and output a T × 2tensor.
Revised: MLP, similar to [3], which takes the ﬂattened scratch trajectory as the input and output a T ×tensor.

Original: b) an LSTM which takes the scratch trajectory as the input of the t th time-step .
Revised: b) an LSTM which takes the scratch trajectory as the input of the t th time-step .

Original: c) 1DCNN, which serves as the local temporal information filter.
Revised: c) 1DCNN, which serves as the local temporal information ﬁlter.

Original: The results are shown in Table 2.
Revised: The results are shown in Table 2.

Original: Additionally, for the variant with best performance, 1D-CNN, we also conduct experimentson cascading multiple refinement modules to explore the effects of the multi-granularity refinement.
Revised: Additionally, for the variant with best performance, 1D-CNN, we also conduct experiments on cascading multiple reﬁnement modules to explore the effects of the multi-granularity reﬁnement.

Original: We can conclude that: a) An additional refinement module could improve the performance explicitly,even if it is simply another MLP.
Revised: We can conclude that: a) An additional reﬁnement module could improve the performance notablely, even if it is simply another MLP.

Original: b) Refinement modules with temporal inductive bias (RNN and 1DCNN) perform better than the one without such bias.
Revised: b) Reﬁnement modules with temporal inductive bias (RNN and 1DCNN) perform better than the one without such bias.

Original: c) By cascading multiple refinement modules,the performance could be further boosted.
Revised: c) By cascading multiple reﬁnement modules, the performance could be further boosted.

Original: In summary, we find that the proposed scratch-then-temporally-refine paradigm could combine thebest of two worlds: the easy optimization of the MLP and the temporal inductive bias of RNN/1DCNN.
Revised: In summary, we ﬁnd that the proposed scratch-then-temporally-reﬁne paradigm could combine the best of two worlds: the easy optimization of the MLP and the temporal inductive bias of RNN/1DCNN.

Original:  Temporally Correlated Objective Function
Revised:  Temporally Correlated Objective Function

Original: Decoder （ �𝐱𝐱 𝟐𝟐 , �𝐲𝐲） （
Revised: Decoder [Equation] [Equation]

Original: �𝐱𝐱 𝟏𝟏 , �𝐲𝐲） （ �𝐱𝐱 𝟑𝟑 , �𝐲𝐲）
Revised: [Equation]

Original: （ 𝐱𝐱 𝟐𝟐 , 𝐲𝐲） （ 𝐱𝐱 𝟏𝟏 , 𝐲𝐲） （ 𝐱𝐱 𝟑𝟑 , 𝐲𝐲）
Revised: ） （ 𝐱𝐱 𝟏𝟏 , 𝐲𝐲） （ 𝐱𝐱 𝟑𝟑 , 𝐲𝐲）

Original: Coordinate Loss
Revised: Coordinate Loss

Original: Decoder （ �𝐯𝐯 𝐱𝐱𝟐𝟐 , �𝐯𝐯 𝒚𝒚 ） （ �𝐯𝐯 𝐱𝐱𝟏𝟏 , �𝐯𝐯 𝒚𝒚 ） （ �𝐯𝐯 𝐱𝐱𝟑𝟑 , �𝐯𝐯 𝒚𝒚 ） （ 𝐯𝐯 𝐱𝐱𝟐𝟐 , 𝐯𝐯 𝐲𝐲 ） （ 𝐯𝐯 𝐱𝐱𝟏𝟏 , 𝐯𝐯 𝐲𝐲 ） （ 𝐯𝐯 𝐱𝐱𝟑𝟑 , 𝐯𝐯 𝐲𝐲 ）
Revised: Decoder （ �𝐯𝐯 𝐱𝐱𝟐𝟐 , �𝐯𝐯 𝒚𝒚 ） （ �𝐯𝐯 𝐱𝐱𝟏𝟏 , �𝐯𝐯 𝒚𝒚 ） （ �𝐯𝐯 𝐱𝐱𝟑𝟑 , �𝐯𝐯 𝒚𝒚 ） （ 𝐯𝐯 𝐱𝐱𝟐𝟐 , 𝐯𝐯 𝐲𝐲 ） （ 𝐯𝐯 𝐱𝐱𝟏𝟏 , 𝐯𝐯 𝐲𝐲 ） （ 𝐯𝐯 𝐱𝐱𝟑𝟑 , 𝐯𝐯 𝐲𝐲 ）

Original: Velocity Loss
Revised: Velocity Loss

Original: Cumulative Coordinate Loss (Ours) （
Revised: Cumulative Coordinate Loss (Ours) （

Original: �𝐯𝐯 𝐱𝐱𝟐𝟐 , �𝐯𝐯 𝒚𝒚 ） （ �𝐯𝐯 𝐱𝐱𝟏𝟏 , �𝐯𝐯 𝒚𝒚 ） （ �𝐯𝐯 𝐱𝐱𝟑𝟑 , �𝐯𝐯 𝒚𝒚 ） （ 𝐱𝐱 𝟐𝟐 , 𝐲𝐲） （ 𝐱𝐱 𝟏𝟏 , 𝐲𝐲） （ 𝐱𝐱 𝟑𝟑 , 𝐲𝐲）
Revised: �𝐯𝐯 𝐱𝐱𝟐𝟐 , �𝐯𝐯 𝒚𝒚 ） （ �𝐯𝐯 𝐱𝐱𝟏𝟏 , �𝐯𝐯 𝒚𝒚 ） （ �𝐯𝐯 𝐱𝐱𝟑𝟑 , �𝐯𝐯 𝒚𝒚 ） （ 𝐱𝐱 𝟐𝟐 , 𝐲𝐲） （ 𝐱𝐱 𝟏𝟏 , 𝐲𝐲） （ 𝐱𝐱 𝟑𝟑 , 𝐲𝐲）

Original: In addition to designing the neural networkstructure with the temporal prior, the temporal constraintamong future time-steps could be applied on the objective function part as well.
Revised: In addition to designing the neural network with the temporal structure, the temporal priors among future time-steps could be applied on the objective function part as well.

Original: Existing works [1,7, 38, 6] usually calculate the regression loss per time-step and then take the mean of all time-stepsas the overall loss:
Revised: Existing works [1, 6, 30, 5] usually calculate the regression loss per time-step and then take the mean of all time-steps as the overall loss:

Original: [Equation]
Revised: [Equation]

Original: where ˆ Y t and Y t are the predicted and ground-truth coordinate at time-step t respectively, and
Revised: where ˆ Y t and Y t are the predicted and ground-truth coordinate at time-step t respectively, and

Original: L step is the loss function applied on each time-step, which could either be the MSE loss [1, 8]or NLL loss [7, 6].
Revised: L step is the loss function applied on each time-step, which could either be the MSE loss [1, 7] or NLL loss [6, 5].

Original: This objective function assumes that agents’ future positions are independentwith each other, which ignores the physical constraint and could result in kinematically infeasibleprediction.
Revised: This objective function assumes that agents’ future positions are independent with each other, which ignores the physical constraint and could result in kinematically infeasible prediction.

Original: Additionally, for the long-term trajectory prediction, agents might move hundreds of meters in the predicted interval.
Revised: Additionally, for the long-term trajectory prediction, agents might move hundreds of meters in the predicted interval.

Original: As a result, the magnitude of the late time-steps’ coordinates mightbe much larger than the early ones, which brings difficulty for the learning process.
Revised: As a result, the magnitude of the late time-steps’ coordinates might be much larger than the early ones, which brings difﬁculty for the learning process.

Original: One intuitive solution for the aforementiond issues is to predict the control signal instead [27, 26]and the coordinate could be obtained by physical models during inference.
Revised: One intuitive solution for the aforementiond issues is to predict the control signal instead [32, 31, 43] and the coordinate could be obtained by physical models during inference.

Original: In this way, the networkexplicitly learns the dynamics which provides strong prior knowledge to the model while keeping theoutput magnitude of different time-steps similar.
Revised: In this way, the network explicitly learns the dynamics which provides strong prior knowledge to the model while keeping the output magnitude of different time-steps similar.

Original: However, in our empirical experiments, predictingvelocity leads to performance drop, which is consistent with the conclusion in [6].
Revised: However, in our empirical experiments, predicting velocity leads to performance drop, which is consistent with the conclusion in [5].

Original: One burden forpredicting the control signal is the cumulative error which makes the model sensitive to noise.
Revised: One burden for predicting the control signal is the cumulative error which makes the model sensitive to noise.

Original: Witha prediction error occurring at the first time-step, the entire generated trajectory would shift from theground-truth, even if predictions at all other time-steps are correct.
Revised: With a prediction error occurring at the ﬁrst time-step, the entire generated trajectory would shift from the ground-truth, even if predictions at all other time-steps are correct.

Revised: The third is the proposed cumulative coordi nate loss.

Original: [Table]
Revised: [Table]

Original: To alleviate the problem, we propose tolet the model output velocity while calculating the regression loss between theaccumulation of the models’ output andthe ground-truth coordinate.
Revised: To alleviate the problem [Table] we propose to let the model output velocity while calculating the regression loss between the accumulation of the models’ output and the ground-truth coordinate.

Original: Formally,given the predicted velocity { (cid:99) ˙
Revised: Formally, given the predicted velocity { (cid:99) ˙

Original: Y t | t =1 , ..., T } and the ground-truth coordinate { Y t | t = 1 , ..., T } , the objectivefunction is: 
Revised: Y t | t = 1 , ..., T } and the GT coordinate { Y t | t = 1 , ..., T } , the objective is: [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: In this way, we combine the best of the two worlds: the output is in the form of velocity whichintroduces temporal correlation among time-steps while the early time-step’s influence on its latercoordinates is considered by the accumulation of the loss.
Revised: In this way [Table] we combine the best of the two worlds: the output is in the form of velocity which introduces temporal correlation among time-steps while the early time-step’s inﬂuence on its later coordinates is considered by the accumulation of the loss.

Original: Figure.
Revised: Figure.

Original: 3 illustrates the three objectivefunctions mentioned above and Table.
Revised: 3 illustrates the three objective functions mentioned above and Table.

Original: 3 gives their performance.
Revised: 3 gives their performance.

Original: The proposed cumulative coordinate loss function achieves the best performance.
Revised: The proposed cumulative coordinate loss function achieves the best performance

Revised: [Table]

Original:  Results on the Public Leaderboard
Revised:  Results on the Public Leaderboard

Original: [Table]
Revised: [Table]

Original: In this section, we provide resultson the test set of Waymo Open Motion 3 to verify that our conclusions are not overfitting the validation set, as shown in Table 4.
Revised: Here we provide results on the test set of Waymo Open Motion 3 to verify that our conclusions are not overﬁtting the validation set, as shown in Table 4.

Original: Wecan find that by combining the proposed techniques, the SOTA encoder
Revised: We can ﬁnd that by combining the proposed techniques, the SOTA encoder

Original: HDGT’s performance could be further boosted and achieves competitive performance on the learderboard 4 .
Revised: HDGT’s performance could be further boosted and achieves competitive performance on the learderboard 4 .

Original:  Ablation Study
Revised:  Ablation Study

Original:  In this section, we study the effects of the combination of the proposed techniques.
Revised: [Table] In this section [Table] we study the effects of the combination of the proposed techniques.

Original: In addition tothe commonly used metrics - minADE/minFDE/MR, to evaluate the physical feasibility of predictedtrajectories under different settings, we calculate the Turning Radius Infeasibility ( TRI ) [6].
Revised: In addition to the commonly used metrics - minADE/minFDE/MR, to evaluate the physical feasibility of predicted trajectories under different settings, we calculate the Turning Radius Infeasibility ( TRI ) [5].

Original: Ifthe turning radius (the circumradius constituting three consecutive waypoints) along the predictedtrajectories is less than a certain threshold, it is treated as a violation.
Revised: If the turning radius (the circumradius constituting three consecutive waypoints) along the predicted trajectories is less than a certain threshold, it is treated as a violation.

Original: We set this threshold as 3.5m- the approximate minimum turning radius threshold for a midsize sedan, following [6]. We also [Table]
Revised: We set this threshold as 3.5m - the approximate minimum turning radius threshold for a midsize sedan, following [5].

Original:  evaluate the smoothness of the trajectory by calculating the acceleration and jerk, i.e. , the second andthird time derivative of the position.
Revised: We also evaluate the smoothness of the trajectory by calculating the acceleration and jerk, i.e. , the second and third time derivative of the position.

Original: According to [40, 41], human drivers’ maximum accelerationis around 5 . 

Revised: According to [45, 46], human drivers’ maximum acceleration is around 5 .

Original: m / s 2 , and that of jerk is 2 . 

Revised: m / s 2 , and that of jerk is 2 .

Original: m / s 3 .
Revised: m / s 3 .

Original: Thus, we define the Unsmooth Ratio ( UR ) as: if atany time-step of a trajectory, its corresponding acceleration is larger than 5 . [-
-] m / s 2 or jerk is largerthan 2 . 

Revised: Thus, we deﬁne the Unsmooth Ratio ( UR ) as: if at any time-step of a trajectory, its corresponding acceleration is larger than 5 . [-
-]0 m / s 2 or jerk is larger than 2 .

Original: m / s 3 , it is considered as an unsmooth step.
Revised: m / s 3 , it is considered as an unsmooth step.

Original: The ablation study results are shown in Table 5.
Revised: The ablation study results are shown in Table 5.

Original: We can conclude that: both techniques could improve the distance-based metrics of the baseline model and the combination of them could furtherboost the performance.
Revised: We can conclude that: both techniques could improve the distance-based metrics of the baseline model and the combination of them could further boost the performance.

Original: As for the Turning Radius Infeasibility (TRI) and Unsmooth Ratio (UR), thetemporal refinement module could make generated trajectories much more smooth and physicallyfeasible while the cumulative coordinate loss is harmful for them.
Revised: As for the Turning Radius Infeasibility (TRI) and Unsmooth Ratio (UR), the temporal reﬁnement module could make generated trajectories much more smooth and physically feasible while the cumulative coordinate loss is harmful for them.

Original: We think it is because predictions of different time-steps have more information interaction in the temporal refinement module,which is helpful for their consistency.
Revised: We think it is because predictions of different time-steps have more information interaction in the temporal reﬁnement module, which is helpful for their consistency.

Original: However, the cumulative coordinate loss entangles all theinformation by the accumulation process which might be unstable compared to direct regression.
Revised: However, the cumulative coordinate loss entangles all the information by the accumulation process which might be unstable compared to direct regression.

Revised: Also, to evaluate the impact of this modiﬁcation is on the inference time of the origin model, we calculcate the mean time and std of 30 runs while keeping the environment, devices, hyperparameters like batch-size the same.

Revised: The computational burden of Cumulative Coordinate Loss is neglectable while the 5 cascaded reﬁne modules takes 3-5% more inference time.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we explore ways of applying the temporal constraint on the generation process oftrajectories.
Revised: In this paper, we explore ways of applying the temporal priors on the generation process of trajectories.

Original: We find that intuitively using an autoregressive RNN would lead to degenerated performance compared to the commonly used MLP decoder.
Revised: We ﬁnd that intuitively using an autoregressive RNN would lead to degenerated performance compared to the commonly used MLP decoder.

Original: To combine the best of two methods, wepropose a scratch-then-temporally-refine paradigm, which first generates a scratch trajectory by an MLP and then incorporates the temporal structure to refine the trajectory.
Revised: To combine the best of two methods, we propose a scratch-then-temporally-reﬁne paradigm, which ﬁrst generates a scratch trajectory by an MLP and then incorporates the temporal structure to reﬁne the trajectory.

Original: We also find that directlyfitting the velocity degenerates the performance, while accumulating the velocity and then still usingthe coordinate as the target would lead to better performance.
Revised: We also ﬁnd that directly ﬁtting the velocity degenerates the performance, while accumulating the velocity and then still using the coordinate as the target would lead to better performance.

Original: Finally, by combining these discovered effective techniques, we boost the SOTA encoder - HDGT’s performance by a large marginin the Waymo Motion Leaderboard, which verifies the benefits of utilizing our proposed temporalprior.
Revised: Finally, by combining these discovered effective techniques, we boost the SOTA encoder - HDGT’s performance by a large margin in the Waymo Motion Leaderboard, which veriﬁes the beneﬁts of utilizing our proposed temporal prior.

Original:  Implementation Details
Revised:  Implementation Details

Original: We adopt HDGT [9] as the encoder for all models with the same hyperparameters.
Revised: We adopt HDGT [8] as the encoder for all models with the same hyperparameters.

Original: We use all agentsin the scene as a sample for the wholeness of the information while [9] uses every target agent andits surrounding 16 agents as a sample.
Revised: We use all agents in the scene as a sample for the wholeness of the information while [8] uses every target agent and its surrounding 16 agents as a sample.

Original: We use max-pooling as the aggregation function for the lanenode for less computational cost, which we find has few degeneration of the performance.
Revised: We use max-pooling as the aggregation function for the lane node for less computational cost, which we ﬁnd has few degeneration of the performance.

Original: We trainall ablation models with 30 epochs for quick verification of the idea and train the final submissionto the leaderboard with 120 epochs which has similar training steps with [9].
Revised: We train all ablation models with 30 epochs for quick veriﬁcation of the idea and train the ﬁnal submission to the leaderboard with 120 epochs which has similar training steps with [8].

Original:  Limitation and Future Work
Revised:  Limitation and Future Work

Original: Experiments are only conducted on one SOTA encoder on the Waymo Open Motion Dataset(WOMP).
Revised: Experiments are only conducted on one SOTA encoder on the Waymo Open Motion Dataset (WOMP).

Original: Though it is the largest public dataset for motion prediction, there still could be potential biased conclusion.
Revised: Though it is the largest public dataset for motion prediction, there still could be potential biased conclusion.

Original: To address this issue, in the future works, we might implement more SOTAencoders and conduct experiments on more datasets to further verify the conclusion.
Revised: To address this issue, in the future works, we might implement more SOTA encoders and conduct experiments on more datasets to further verify the conclusion.

Revised: Autoregressive RNN, as an structure naturally with temporal prior, did not work well under the simple attempts tried in this paper.

Revised: More sophisticated and well-designed strcture could be explored such as [47, 48] for better convergence and performance.

Revised: Acknowledgments

Revised: This work was partly supported by NSFC (62206172, 62222607), Shanghai Municipal Science and

Revised: Technology Major Project (2021SHZDZX0102), and Shanghai Committee of Science and Technology (21DZ1100100).