Motion Forecasting Transformer with Global Intention Localization and Local Movement Refinement
Anonymous Author(s)
AffiliationAddress email
Abstract
Predicting multimodal future behavior of traffic participants is essential for roboticvehicles to make safe decisions.
Existing works explore to directly predict futuretrajectories based on latent features or utilize dense goal candidates to identifyagent’s destinations, where the former strategy converges slowly since all motionmodes are derived from the same feature while the latter strategy has efficiencyissue since its performance highly relies on the density of goal candidates.
Inthis paper, we propose the Motion TRansformer (MTR) framework that modelsmotion prediction as the joint optimization of global intention localization andlocal movement refinement.
Instead of using goal candidates, MTR incorporatesspatial intention priors by adopting a small set of learnable motion query pairs.
Each motion query pair takes charge of trajectory prediction and refinement for aspecific motion mode, which stabilizes the training process and facilitates bettermultimodal predictions.
Experiments show that MTR achieves state-of-the-artperformance on both the marginal and joint motion prediction challenges, ranking1 st on the leaderbaords of Waymo Open Motion Dataset.
Code will be available.
Introduction
Motion forecasting is a fundamental task of modern autonomous driving systems.
It has beenreceiving increasing attention in recent years [18, 43, 27, 52, 33] as it is crucial for robotic vehiclesto understand driving scenes and make safe decisions.
Motion forecasting requires to predict futurebehaviors of traffic participants by jointly considering the observed agent states and road maps, whichis challenging due to inherently multimodal behaviors of the agent and complex scene environments.
To cover all potential future behaviors of the agent, existing approaches mainly fall into two differentlines: the goal-based methods and the direct-regression methods.
The goal-based methods [18, 56]adopt dense goal candidates to cover all possible destinations of the agent, predicting the probabilityof each candidate being a real destination and then completing the full trajectory for each selectedcandidate.
Although these goal candidates alleviate the burden of model optimization by reducingtrajectory uncertainty, their density largely affects the performance of these methods: fewer candidateswill decrease the performance while more candidates will greatly increase computation and memorycost.
Instead of using goal candidates, the direct-regression methods [33, 44] directly predict a setof trajectories based on the encoded agent feature, covering the agent’s future behavior adaptively.
Despite the flexibility in predicting a broad range of agent behaviors, they generally converge slowlyas various motion modes are required to be regressed from the same agent feature without utilizingany spatial priors.
They also tend to predict the most frequent modes of training data since thesefrequent modes dominate the optimization of the agent feature.
In this paper, we present a unifiedframework, namely Motion TRansformer (MTR), which takes the best of both types of methods.
In our proposed MTR, we adopt a small set of novel motion query pairs to model motion predictionas the joint optimization of two tasks:
The first global intention localization task aims to roughly identify agent’s intention for achieving higher efficiency, while the second local movement refinementtask aims to adaptively refine each intention’s predicted trajectory for achieving better accuracy.
Ourapproach not only stabilizes the training process without depending on dense goal candidates but alsoenables flexible and adaptive prediction by enabling local refinement for each motion mode.
Specifically, each motion query pair consists of two components, i.e. , a static intention query and adynamic searching query.
The static intention queries are introduced for global intention localization,where we formulate them based on a small set of spatially distributed intention points.
Each staticintention query is the learnable positional embedding of an intention point for generating trajectory ofa specific motion mode, which not only stabilizes the training process by explicitly utilizing differentqueries for different modes, but also eliminates the dependency on dense goal candidates by requiringeach query to take charge of a large region.
The dynamic searching queries are utilized for localmovement refinement, where they are also initialized as the learnable embeddings of the intentionpoints but are responsible for retrieving fine-grained local features around each intention point.
Forthis purpose, the dynamic searching queries are dynamically updated according to the predictedtrajectories, which can adaptively gather latest trajectory features from a deformable local region foriterative motion refinement.
These two queries complement each other and have been empiricallydemonstrated their great effectiveness in predicting multimodal future motion.
Besides that, we alsopropose a dense future prediction module.
Existing works generally focus on modeling the agentinteraction over past trajectories while ignoring the future trajectories’ interaction.
To compensate forsuch information, we adopt a simple auxiliary regression head to densely predict future trajectoryand velocity for each agent, which are encoded as additional future context features to benefit futuremotion prediction of our interested agent.
The experiments show that this simple auxiliary task workswell and remarkably improves the performance of multimodal motion prediction.
Our contributions are three-fold: (1) We propose a novel motion decoder network with a newconcept of motion query pair, which adopts two types of queries to model motion prediction as jointoptimization of global intention localization and local movement refinement.
It not only stabilizesthe training with mode-specific motion query pairs, but also enables adaptive motion refinementby iteratively gathering fine-grained trajectory features.
(2) We present an auxiliary dense futureprediction task to enable the future interactions between our interested agent and other agents.
Itfacilitates our framework to predict more scene-compliant trajectories for the interacting agents.
By adopting these techniques, we propose MTR framework that explores transformer encoder-decoderstructure for multimodal motion prediction.
Our approach achieves state-of-the-art performanceon both the marginal and joint motion prediction benchmarks of Waymo Open Motion Dataset(WOMD) [13], outperforming previous best ensemble-free approaches with + 8 .
48% mAP gains formarginal motion prediction and +7 .
98% mAP gains for joint motion prediction.
As of 19 May 2022,our approach ranks 1 st on both the marginal and joint motion prediction leaderboards of WOMD.
Related Work
Motion Prediction for Autonomous Driving.
Recently, motion prediction has been extensivelystudied due to the growing interest in autonomous driving, and it typically takes road map andagent history states as input.
To encode such scene context, early works [34, 29, 5, 11, 55, 3, 8]typically rasterize them into an image so as to be processed with convolutional neural networks(CNNs).
LaneGCN [25] builds a lane graph toscalability capture map topology.
VectorNet [15] iswidely adopted by recent works [18, 40, 33, 44] due to its efficiency and scalability, where both roadmaps and agent trajectories are represented as polylines.
We also adopt this vector representation,but instead of building global graph of polylines, we propose to adopt transformer encoder onlocal connected graph, which not only better maintains input locality structure but also is morememory-efficient to enable larger map encoding for long-term motion prediction.
Given the encoded scene context features, existing works explore various strategies to model multimodal future motion.
Early works [1, 19, 37, 41, 38] propose to generate a set of trajectory samplesto approximate the output distribution.
Some other works [9, 20, 31, 35, 39] parameterize multimodal predictions with Gaussian Mixture Models (GMMs) to generate compact distribution.
HOMEseries [17, 16] generate trajectories with sampling on a predicted heatmap.
IntentNet [7] considersintention prediction as a classification with 8 high level actions, while [27] proposes a region-basedtraining strategy.
Goal-based methods [56, 38, 14, 28] are another kinds of models where they firstestimate several goal points of the agents and then complete full trajectory for each goal. TransformerEncoder TransformerDecoderLayer …
DenseFuturePrediction … DynamicMapCollection GMMPrediction 𝒦×(2+2) 𝒦×1×𝒦×𝐷 𝒦×(2+2) 𝒦×𝑇×𝒦×𝐷 MotionQueryPair
PolylineEncoder
PredictedTrajectoryInitialization
QueryContentInitialization
QueryUpdating ×𝑁
Agentfeatures
Mapfeature
Predictedtrajectory
Trajectory-specificmapfeature
Interestedagent
Otheragent …
LocalGraph
QueryContentFeature
PredictedTrajectory
Key&Value
QueryPosition QueryContent [Figure] a
[Figure]
（ b ） （ c ） Recently, the large-scale Waymo Open Motion Dataset (WOMD) [13] is proposed for long-termmotion prediction. To address this challenge, DenseTNT [18] adopts a goal-based strategy to classifyendpoint of trajectory from dense goal points.
Other works directly predict the future trajectoriesbased on the encoded agent features [33] or latent anchor embedding [44].
However, the goal-basedstrategy has the efficiency concern due to a large number of goal candidates, while the directregression strategy converges slowly as the predictions of various motion modes are regressed fromthe same agent feature.
In contrast, our approach adopts a small set of learnable motion query pairs,which not only eliminate the large number of goal candidates but also alleviate the optimizationburden by utilizing mode-specific motion query pairs for predicting different motion modes.
Some very recent works [42, 22, 21] also achieve top performance on WOMD by exploring Mix-andMatch block [42], a variant of MultiPath++ [22] or heterogeneous graph [21].
However, they generallyfocus on exploring various structures for encoding scene context, while how to design a better motiondecoder for multimodal motion prediction is still underexplored.
In contrast, our approach focuses onaddressing this challenge with a novel transformer-based motion decoder network.
Transformer.
Transformer [45] has been widely applied in natural language processing [10, 2]and computer vision [12, 47, 4, 46, 53].
Our approach is inspired by DETR [4] and its follow-upworks [58, 30, 26, 54], especially DAB-DETR [26], where the object query is considered as thepositional embedding of a spatial anchor box.
Motivated by their great success in object detection,we introduce a novel concept of motion query pair to model multimodal motion prediction with priorintention points, where each motion query pair takes charge of predicting a specific motion mode andalso enables iterative motion refinement by combining with transformer decoders.
Motion TRansformer (MTR)
We propose Motion TRansformer (MTR), which adopts a novel transformer encoder-decoder structurewith iterative motion refinement for predicting multimodal future motion.
The overall structure isillustrated in Figure 1.
In Sec. 3.1, we introduce our encoder network for scene context modeling.
In
Sec. 3.2, we present motion decoder network with a novel concept of motion query pair for predictingmultimodal trajectories.
Finally, in Sec. 3.3, we introduce the optimization process of our framework.
3.1 Transformer Encoder for Scene Context Modeling
The future behaviors of the agents highly depend on the agents’ interaction and road map.
To encodesuch scene context, existing approaches have explored various strategies by building global interacting graph [15, 18] or summarizing map features to agent-wise features [33, 44].
We argue that the localitystructure is important for encoding scene context, especially for the road map.
Hence, we propose atransformer encoder network with local self-attention to better maintain such structure information.
Input representation.
We follow the vectorized representation [15] to organize both input trajectoriesand road map as polylines.
For the motion prediction of a interested agent, we adopt the agent-centricstrategy [56, 18, 44] that normalizes all inputs to the coordinate system centered at this agent.
Then,a simple polyline encoder is adopted to encode each polyline as an input token feature for thetransformer encoder.
Specifically, we denote the history state of N a agents as A in ∈ N N a × t × C a ,where t is the number of history frames, C a is the number of state information ( e.g. , location, headingangle and velocity), and we pad zeros at the positions of missing frames for trajectories that haveless than t frames.
The road map is denoted as M in ∈ R N m × n × C m , where N m is the number of mappolylines, n is the number of points in each polyline and C m is the number of attributes of each point( e.g. , location and road type).
Both of them are encoded by a PointNet-like [36] polyline encoder as:
[Equation]
where MLP ( · ) is a multilayer perceptron network, and ϕ is max-pooling to summarize each polylinefeatures as agent features A p ∈ R N a × D and map features M p ∈ R N m × D with feature dimension D .
Scene context encoding with local transformer encoder.
The local structure of scene context isimportant for motion prediction.
For example, the relation of two parallel lanes is important formodelling the motion of changing lanes, but adopting attention on global connected graph equallyconsiders relation of all lanes.
In contrast, we introduce such prior knowledge to context encoder byadopting local attention, which better maintains the locality structure and are more memory-efficient.
Specifically, the attention module of j -th transformer encoder layer can be formulated as:
[Equation]
where MultiHeadAttn ( · , · , · ) is the multi-head attention layer [45], G {+
+}
=
[ A p , M p ] ∈ N ( N a + N m ) ×
Dconcatenating the features of agents and map, and κ ( · ) denotes k -nearest neighbor algorithm to findk closest polylines for each query polyline.
PE denotes sinusoidal position encoding of input tokens,where we utilize the latest position for each agent and utilize polyline center for each map polyline.
Thanks to such local self-attention, our framework can encode a much larger area of scene context.
The encoder network finally generates both agent features A past ∈ R N a × D and map features M ∈ R N m × D , which are considered as the scene context inputs of the following decoder network.
Dense future prediction for future interactions.
Interactions with other agents heavily affect behaviors of our interested agent, and the pioneer works propose to model the multi-agent interactions withhub-host based network [59], dynamic relational reasoning [24], social spatial-temporal network [51],etc.
However, most existing works generally focus on learning such interactions over past trajectorieswhile ignoring the interactions of future trajectories.
Therefore, considering that the encoded features A have already learned rich context information of all agents, we propose to densely predict bothfuture trajectories and velocities of all agents by adopting a simple regression head on A :
[Equation]
where S i ∈
R N a × 4 includes future position and velocity of each agent at time step i , and T is thenumber of future frames to be predicted.
The predicted trajectories S 1: T are encoded by adoptingthe same polyline encoder as Eq.
(1) to encode the agents’ future states as features A future ∈ R N a × D ,which are then utilized to enhance the above features A by using a feature concatenation and three MLP layers as A =
MLP ([ A past , A future ]) .
This auxiliary task provides additional future contextinformation to the decoder network, facilitating the model to predict more scene-compliant futuretrajectories for the interested agent.
The experiments in Table 3 demonstrates that this simple andlight-weight auxiliary task can effectively improve the performance of multimodal motion prediction.
3.2 Transformer Decoder with Motion Query Pair
Given the scene context features, a transformer-based motion decoder network is adopted for multimodal motion prediction, where we propose motion query pair to model motion prediction as thejoint optimization of global intention localization and local movement refinement.
Each motionquery pair contains two types of queries, i.e. , static intention query and dynamic searching query,for conducting global intention localization and local movement refinement respectively.
As shown in Figure 2, our motion decoder network contains stacked transformer decoder layers for iterativelyrefining the predicted trajectories with motion query pairs.
Next, we illustrate the detailed structure.
Global intention localization aims to localize agent’s potential motion intentions in an efficientand effective manner.
We propose static intention query to narrow down the uncertainty of futuretrajectory by utilizing different intention queries for different motion modes.
Specifically, we generate [Figure]
𝐼: (𝒦×2) 𝐶 ! :
(𝒦×𝐷) 𝑌 "!
: (𝒦×2)
Sine+MLP
Sine+MLP + + Q K V
Q K V 𝐼: (𝒦×2) 𝐶 !
#$ : (𝒦×𝐷) 𝑌 "! #$ : (𝒦×2) 𝑍 $:"!
#$ : (𝒦×𝑇×6)
[Figure]
i
[Figure]
K representative intention points
I
by adopting k-means clustering algorithm onthe endpoints of ground-truth (GT) trajectories, where each intention point represents animplicit motion mode that considers both motion direction and velocity.
We model eachstatic intention query as the learnable positional embedding of the intention point as:
[Equation]
where PE( · ) is the sinusoidal position encoding, and Q I ∈ R K× D .
Notably, each intention query takes charge of predicting trajectories for a specific motion mode, whichstabilizes the training process and facilitatespredicting multimodal trajectories since eachmotion mode has their own learnable embedding.
Thanks to their learnable and adaptiveproperties, we only need a small number ofqueries ( e.g. , 64 queries in our setting) forefficient intention localization, instead of using densely-placed goal candidates [56, 18]to cover the destinations of the agents.
Local movement refinement aims to complement with global intention localization by iterativelygathering fine-grained trajectory features for refining the trajectories.
We propose dynamic searchingquery to adaptively probe trajectory features for each motion mode.
Each dynamic searching query isalso the position embedding of a spatial point, which is initialized with its corresponding intentionpoint but will be dynamically updated according to the predicted trajectory in each decoder layer.
Specifically, given the predicted future trajectories Y j 1: T = { Y ji ∈ R K× 2 | i = 1 , · · · , T } in j -thdecoder layer, the dynamic searching query of ( j + 1) -th decoder layer is updated as follows:
[Equation]
As shown in Figure 3, for each motion query pair, we propose a dynamic map collection module toextract fine-grained trajectory features by querying map features from a trajectory-aligned local region,which is implemented by collecting L polylines whose centers are closest to the predicted trajectory.
As the agent’s behavior largely depends on road maps, this local movement refinement strategyenables to continually focus on latest local context information for iterative motion refinement.
Attention module with motion query pair.
In each decoder layer, static intention query is utilized topropagate information among different motion intentions, while dynamic searching query is utilizedto aggregate trajectory-specific features from scene context features.
Specifically, we utilize staticintention query as the position embedding of self-attention module as follows:
[Equation]
where C j − 1 ∈ R K× D is query content features from ( j − 1) -th decoder layer, C 0 is initialized tozeros, and C j sa ∈ R
K× D is the updated query content.
Next, we utilize dynamic searching query asquery position embedding of cross attention to probe trajectory-specific features from the outputsof encoder.
Inspired by [30, 26], we concatenate content features and position embedding for bothquery and key to decouple their contributions to the attention weights.
Two cross-attention modulesare adopted separately for aggregating features from both agent features A and map features M as:
[Equation]
Motion Query
Motion Query
Motion Query0.
[Figure]
Prediction of layer 𝑗
Static Intention
[Figure]
Dynamic Searching Query
Predictedtrajectory
Trajectory-specificmapfeatures where [ · , · ] indicates feature concatenation, α ( M ) is the aforementioned dynamic map collectionmodule to collect L trajectory-aligned map features for motion refinement. Note that for simplicity, in Eq. (6) and (7), we omit the residual connection and feed-forward network in transformer layer [45].
Finally, C j ∈ R K× D is the updated query content features for each motion query pair in j -th layer.
Multimodal motion prediction with Gaussian Mixture Model.
For each decoder layer, we appenda prediction head to C j for generating future trajectories.
As the behaviors of the agents are highlymultimodal, we follow [9, 44] to represent the distribution of predicted trajectories with Gaussian
Mixture Model (GMM) at each time step.
Specifically, for each future time step i ∈ { 1 , · · · , T } , wepredict the probability p and parameters ( µ x , µ y , σ x , σ y , ρ ) of each Gaussian component as follows
[Equation]
where Z ji ∈ R K× 6 includes K Gaussian components N 1: K ( µ x , σ x ; µ y , σ y ; ρ ) with probability distribution p 1: K .
The predicted distribution of agent’s position at time step i can be formulated as:
[Equation]
(9) where P ji ( o ) is the occurrence probability of the agent at spatial position o ∈ R 2 .
The predictedtrajectories Y j 1: T can be generated by simply extracting the predicted centers of Gaussian components.
3.3 Training Losses
Our model is trained end-to-end with two training losses.
The first auxiliary loss is L 1 regression lossto optimize the outputs of Eq.
For the second Gaussian regression loss, we adopt negative loglikelihood loss according to Eq.
(9) to maximum the likelihood of ground-truth trajectory.
Inspiredby [9, 44], we adopt a hard-assignment strategy that selects one closest motion query pair as positive Gaussian component for optimization, where the selection is implemented by calculating the distancebetween each intention point and the endpoint of GT trajectory.
The Gaussian regression loss isadopted in each decoder layer, and the final loss is the sum of the auxiliary regression loss and all the Gaussian regression loss with equal loss weights. Please refer to appendix for more loss details.
Experiments
4.1 Experimental Setup
Dataset and metrics.
We evaluate our approach on the large-scale Waymo Open Motion Dataset(WOMD) [13], which mines interesting interactions from real-world traffic scenes and is currentlythe most diverse interactive motion dataset.
There are two tasks in WOMD with separate evaluationmetrics: (1) The marginal motion prediction challenge that independently evaluates the predictedmotion of each agent (up to 8 agents per scene).
The joint motion prediction challenge that needsto predict the joint future positions of 2 interacting agents for evaluation.
Both of them provide 1second of history data and aim to predict 6 marginal or joint trajectories of the agents for 8 secondsinto the future.
There are totally 487 k training scenes, and about 44 k validation scenes and 44 k testingscenes for each challenge.
We utilize the official evaluation tool to calculate the evaluation metrics,where the mAP and miss rate are the most important ones as in the official leaderboard[49, 48]. [Table] [Table]
Implementation details.
For the context encoding, we stack 6 transformer encoder layers.
Theroad map is represented as multiple polylines, where each polyline contains up to [-
 [Table] [Table] Val-]20 map points(about 10 m in WOMD).
We select N m = 768 nearest map polylines around the interested agent.
The number of neighbors in encoder’s local self-attention is set to 16.
The hidden feature dimensionis set as D = 256 .
For the decoder modules, we stack 6 decoder layers.
L is set to 128 to collect theclosest map polylines from context encoder for motion refinement.
By default, we utilize 64 motionquery pairs where their intention points are generated by conducting k-means clustering algorithm onthe training set.
To generate 6 future trajectories for evaluation, we use non-maximum suppression(NMS) to select top 6 predictions from 64 predicted trajectories by calculating the distances betweentheir endpoints, and the distance threshold is set as 2 . 5 m .
Please refer to Appendix for more details.
Training details.
Our model is trained in an end-to-end manner by AdamW optimizer with a learningrate of 0.0001 and batch size of 80 scenes.
We train the model for 60 epochs with 8 GPUs (NVDIA
RTX 8000), and the learning rate is decayed by a factor of 0.5 every 5 epochs from epoch 30.
Theweight decay is set as 0.01 and we do not use any data augmentation.
MTR-e2e for end-to-end motion prediction.
We also propose an end-to-end variant of MTR, called
MTR-e2e, where only 6 motion query pairs are adopted so as to remove NMS post processing.
In thetraining process, instead of using static intention points for target assignment as in MTR, MTR-e2eselects positive mixture component by calculating the distances between its 6 predicted trajectoriesand the GT trajectory, since 6 intention points are too sparse to well cover all potential future motions.
4.2 Main Results
Performance comparison for marginal motion prediction.
Table 1 shows our main results formarginal motion prediction, our MTR outperforms previous ensemble-free approaches [18, 33] withremarkable margins, increasing the mAP by +8 .
48% and decreasing the miss rate from 15 .
64% to13 .
In particular, our single-model results of MTR also achieve better mAP than the latest work MultiPath++ [44], where it uses a novel model ensemble strategy that boosts its performance.
Table 1 also shows the comparison of MTR variants.
MTR-e2e achieves better minADE and minFDEby removing NMS post-processing, while MTR achieves better mAP since it learns explicit meaningof each motion query pair that produces more confident intention predictions.
We also propose asimple model ensemble strategy to merge the predictions of MTR and MTR-e2e and utilize NMSto remove redundant predictions (denoted as MTR-ens), and it takes the best of both models andachieves much better mAP. By adopting such ensemble strategy to 7 variants of our framework ( e.g. , [Table]
[Figure] [Table] [Figure] [Table]
memory.
[Table]
more decoder layers, different number of queries, larger hidden dimension), our advanced ensembleresults (denoted as MTR-Advanced-ens) achieve best performance on the test set leaderboard.
Performance comparison for joint motion prediction.
To evaluate our approach for joint motionprediction, we combine the marginal predictions of two interacting agents into joint prediction asin [6, 13, 40], where we take the top 6 joint predictions from 36 combinations of these two agents.
The confidence of each combination is the product of marginal probabilities.
Table 2 shows thatour approach outperforms state-of-the-arts [33, 40] with large margins on all metrics.
Particularly,our MTR boosts the mAP from 12 .
39% to 20 .
37% and decreases the miss rate from 49 .
42% to44 .
The remarkable performance gains demonstrate the effectiveness of MTR for predictingscene-consistent future trajectories.
Besides that, we also provide some qualitative results in Figure 5to show our predictions in complicated interacting scenarios.
As of May 19, 2022, our MTR ranks 1 st on the motion prediction leaderboard of WOMD for bothtwo challenges [49, 48].
The significant improvements manifest the effectiveness of MTR framework.
4.3 Ablation Study
We study the effectiveness of each component in MTR.
For efficiently conducting ablation experiments, we uniformly sampled 20% frames (about 97 k scenes) from the WOMD training set accordingto their default order, and we empirically find that it has similar distribution with the full training set.
All models are evaluated with marginal motion prediction metric on the validation set of WOMD.
Effects of the motion decoder network.
We study the effectiveness of each component in ourdecoder network, including global intention localization, iterative refinement and local movementrefinement.
Table 3 shows that all components contributes remarkably to the final performancein terms of the official ranking metric mAP.
Especially, our proposed static intention queries withintention points achieves much better mAP ( i.e. , +4 .
26% ) than the latent learnable embeddingsthanks to its mode-specific querying strategy, and both the iterative refinement and local movementrefinement strategy continually improve the mAP from 30 .
59% to 32 .
34% by aggregating morefine-grained trajectory features for motion refinement.
Effects of dense future prediction.
Table 3 shows that our proposed dense future prediction modulesignificantly improves the quality of predicted trajectories ( e.g. , +1 .
78% mAP), which verifies thatfuture interactions of the agents’ trajectories are important for motion prediction and our proposedstrategy can learn such interactions to predict more reliable trajectories. [Figure] high [Figure]
trajectory confidence
Effects of local attention for context encoding.
Table 4 shows that by taking the same number ofmap polylines as input, local self-attention in transformer encoder achieves better performance thanglobal attention ( i.e. , +0 . 77 % mAP for 256 polylines and +1 .
12% mAP for 512 polylines), whichverifies that the input local structure is important for motion prediction and introducing such priorknowledge with local attention can benefit the performance.
More importantly, local attention is morememory-efficient and the performance keeps growing when improving the number of map polylinesfrom 256 to 1,024, while global attention will run out of memory due to its quadratic complexity.
Effects of the number of motion query pairs with different training strategies.
As mentionedbefore, during training process, MTR and MTR-e2e adopt two different strategies for assigningpositive mixture component, where MTR depends on static intention points (denoted as α ) while
MTR-e2e utilizes predicted trajectories (denoted as β ).
Figure 4 investigates the effects of the numberof motion query pairs under these two strategies, where we have the following observations: (1)
When increasing the number of motion query pairs, strategy α achieves much better mAP and missrate than strategy β .
Because intention query points can ensure more stable training process sinceeach intention query points is responsible to a specific motion mode.
In contrast, strategy β dependson unstable predictions and the positive component may randomly switch among all components,so a large number of motion query pairs are hard to be optimized with strategy β .
The explicitmeaning of each intention query point also illustrates the reason that strategy α consistently achievesmuch better mAP than strategy β , since it can predict trajectories with more confident scores tobenefit mAP metric.
(3) From another side, when decreasing the number of motion query pairs, themiss rate of strategy α greatly increases, since a limit number of intention query points can not wellcover all potential motions of agents.
Conversely, strategy β works well for a small number of motionquery pairs since its queries are not in charge of specific region and can globally adapt to any region.
Conclusion
In this paper, we present MTR, a novel framework for multimodal motion prediction.
The motionquery pair is defined to model motion prediction as the joint optimization of global intention localization and local movement refinement.
The global intention localization adopts a small set of learnablestatic intention queries to efficiently capture agent’s motion intentions, while the local movement refinement conducts iterative motion refinement by continually probing fine-grained trajectory features.
The experiments on both marginal and joint motion prediction challenges of large-scale WOMDdataset show that our approach achieves state-of-the-art performance.
Limitations.
The proposed framework adopts an agent-centric strategy to predict multimodal 346 future trajectories for one interested agent, which leads to redundant context encoding if there aremultiple interested agents in the same scene.
Although the dense future prediction module partially 348 compensates for this limitation, it can only predict a single future trajectory for each agent. Hence, howto develop a joint motion prediction framework that can simultaneously predict multimodal motion for 350 multiple agents is one important future work. Besides, the rule-based NMS post-processing can resultin suboptimal predictions for minADE and minFDE metrics, and how to develop a learning-based 352 module to produce a required number of future trajectories (e.g., 6 trajectory) from full multimodalpredictions (e.g., 64 predictions) is also worth exploring for a more robust framework.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work
is only for academic research purpose.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
N/A]3.
If you ran experiments...(
a) Did you include the code
, data and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Yes ] Our implementation details are posted on Section 4.1 and the supplemental materials.
The data ispublic dataset and our code will be available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4.1 for training details, and the data splits of ablation [Table] experiments are mentioned in Section 4.3. [Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We fix the random seed for all our experiments.
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1.
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] See our reference.
[Table]
(b) Did you mention the license of the assets?
[No] They are publicly available for
[Table]
academic use.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[Yes] They are publicly available for academic use. [Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[No] We only use the public Waymo Open Motion
[Table]
Dataset.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]