Motion Transformer with Global Intention Localization and Local Movement Refinement
Abstract
Predicting multimodal future behavior of traffic participants is essential for robotic vehicles to make safe decisions.
Existing works explore to directly predict future trajectories based on latent features or utilize dense goal candidates to identify agent’s destinations, where the former strategy converges slowly since all motion modes are derived from the same feature while the latter strategy has efficiency issue since its performance highly relies on the density of goal candidates.
In this paper, we propose the Motion TRansformer (MTR) framework that models motion prediction as the joint optimization of global intention localization and local movement refinement.
Instead of using goal candidates, MTR incorporates spatial intention priors by adopting a small set of learnable motion query pairs.
Each motion query pair takes charge of trajectory prediction and refinement for a specific motion mode, which stabilizes the training process and facilitates better multimodal predictions.
Experiments show that MTR achieves state-of-the-art performance on both the marginal and joint motion prediction challenges, ranking 1 st on the leaderboards of Waymo Open Motion Dataset.
Code will be available at https://github.com/sshaoshuai/MTR .
Introduction
Motion forecasting is a fundamental task of modern autonomous driving systems.
It has been receiving increasing attention in recent years [21, 48, 31, 61, 37] as it is crucial for robotic vehicles to understand driving scenes and make safe decisions.
Motion forecasting requires to predict future behaviors of traffic participants by jointly considering the observed agent states and road maps, which is challenging due to inherently multimodal behaviors of the agent and complex scene environments.
To cover all potential future behaviors of the agent, existing approaches mainly fall into two different lines: the goal-based methods and the direct-regression methods.
The goal-based methods [21, 65] adopt dense goal candidates to cover all possible destinations of the agent, predicting the probability of each candidate being a real destination and then completing the full trajectory for each selected candidate.
Although these goal candidates alleviate the burden of model optimization by reducing trajectory uncertainty, their density largely affects the performance of these methods: fewer candidates will decrease the performance while more candidates will greatly increase computation and memory cost.
Instead of using goal candidates, the direct-regression methods [37, 49] directly predict a set of trajectories based on the encoded agent feature, covering the agent’s future behavior adaptively.
Despite the flexibility in predicting a broad range of agent behaviors, they generally converge slowly as various motion modes are required to be regressed from the same agent feature without utilizing any spatial priors.
They also tend to predict the most frequent modes of training data since these frequent modes dominate the optimization of the agent feature.
In this paper, we present a unified framework, namely Motion TRansformer (MTR) , which takes the best of both types of methods.
In our proposed MTR, we adopt a small set of novel motion query pairs to model motion prediction as the joint optimization of two tasks:
The first global intention localization task aims to roughly identify agent’s intention for achieving higher efficiency, while the second local movement refinement task aims to adaptively refine each intention’s predicted trajectory for achieving better accuracy.
Our approach not only stabilizes the training process without depending on dense goal candidates but also enables flexible and adaptive prediction by enabling local refinement for each motion mode.
Specifically, each motion query pair consists of two components, i.e. , a static intention query and a dynamic searching query.
The static intention queries are introduced for global intention localization, where we formulate them based on a small set of spatially distributed intention points.
Each static intention query is the learnable positional embedding of an intention point for generating trajectory of a specific motion mode, which not only stabilizes the training process by explicitly utilizing different queries for different modes, but also eliminates the dependency on dense goal candidates by requiring each query to take charge of a large region.
The dynamic searching queries are utilized for local movement refinement, where they are also initialized as the learnable embeddings of the intention points but are responsible for retrieving fine-grained local features around each intention point.
For this purpose, the dynamic searching queries are dynamically updated according to the predicted trajectories, which can adaptively gather latest trajectory features from a deformable local region for iterative motion refinement.
These two queries complement each other and have been empirically demonstrated their great effectiveness in predicting multimodal future motion.
Besides that, we also propose a dense future prediction module.
Existing works generally focus on modeling the agent interaction over past trajectories while ignoring the future trajectories’ interaction.
To compensate for such information, we adopt a simple auxiliary regression head to densely predict future trajectory and velocity for each agent, which are encoded as additional future context features to benefit future motion prediction of our interested agent.
The experiments show that this simple auxiliary task works well and remarkably improves the performance of multimodal motion prediction.
Our contributions are three-fold: (1) We propose a novel motion decoder network with a new concept of motion query pair, which adopts two types of queries to model motion prediction as joint optimization of global intention localization and local movement refinement.
It not only stabilizes the training with mode-specific motion query pairs, but also enables adaptive motion refinement by iteratively gathering fine-grained trajectory features.
(2) We present an auxiliary dense future prediction task to enable the future interactions between our interested agent and other agents.
It facilitates our framework to predict more scene-compliant trajectories for the interacting agents.
By adopting these techniques, we propose MTR framework that explores transformer encoder-decoder structure for multimodal motion prediction.
Our approach achieves state-of-the-art performance on both the marginal and joint motion prediction benchmarks of Waymo Open Motion Dataset (WOMD) [15], outperforming previous best ensemble-free approaches with + 8 .
48% mAP gains for marginal motion prediction and +7 .
98% mAP gains for joint motion prediction.
As of 19 May 2022, our approach ranks 1 st on both the marginal and joint motion prediction leaderboards of WOMD.
Related Work
Motion Prediction for Autonomous Driving.
Recently, motion prediction has been extensively studied due to the growing interest in autonomous driving, and it typically takes road map and agent history states as input.
To encode such scene context, early works [38, 33, 6, 13, 64, 4, 9] typically rasterize them into an image so as to be processed with convolutional neural networks (CNNs).
LaneGCN [29] builds a lane graph toscalability capture map topology.
VectorNet [17] is widely adopted by recent works [21, 45, 37, 49] due to its efficiency and scalability, where both road maps and agent trajectories are represented as polylines.
We also adopt this vector representation, but instead of building global graph of polylines, we propose to adopt transformer encoder on local connected graph, which not only better maintains input locality structure but also is more memory-efficient to enable larger map encoding for long-term motion prediction.
Given the encoded scene context features, existing works explore various strategies to model multimodal future motion.
Early works [2, 22, 41, 46, 42] propose to generate a set of trajectory samples to approximate the output distribution.
Some other works [10, 23, 35, 39, 43] parameterize multimodal predictions with Gaussian Mixture Models (GMMs) to generate compact distribution.
HOME series [19, 18] generate trajectories with sampling on a predicted heatmap.
IntentNet [8] considers intention prediction as a classification with 8 high level actions, while [31] proposes a region-based
TransformerEncoder
TransformerDecoderLayer …
DenseFuturePrediction …
DynamicMapCollection
GMMPrediction 𝒦×(2+2) 𝒦×1×𝒦×𝐷 𝒦×(2+2) 𝒦×𝑇×𝒦×𝐷
MotionQueryPair
PolylineEncoder
PredictedTrajectoryInitialization
QueryContentInitialization
QueryUpdating ×𝑁
Agentfeatures
Mapfeature
Predictedtrajectory
Trajectory-specificmapfeature
Interestedagent
Otheragent …
LocalGraph
QueryContentFeature
PredictedTrajectory
Key&Value
QueryPosition
QueryContent （ a ） （ b ） （ c ） training strategy.
Goal-based methods [65, 42, 16, 32] are another kinds of models where they first estimate several goal points of the agents and then complete full trajectory for each goal.
Recently, the large-scale Waymo Open Motion Dataset (WOMD)
[15] is proposed for long-term motion prediction.
To address this challenge, DenseTNT [21] adopts a goal-based strategy to classify endpoint of trajectory from dense goal points.
Other works directly predict the future trajectories based on the encoded agent features [37] or latent anchor embedding [49].
However, the goal-based strategy has the efficiency concern due to a large number of goal candidates, while the directregression strategy converges slowly as the predictions of various motion modes are regressed from the same agent feature.
In contrast, our approach adopts a small set of learnable motion query pairs, which not only eliminate the large number of goal candidates but also alleviate the optimization burden by utilizing mode-specific motion query pairs for predicting different motion modes.
Some very recent works [47, 25, 24] also achieve top performance on WOMD by exploring Mix-andMatch block [47], a variant of MultiPath++ [25] or heterogeneous graph [24].
However, they generally focus on exploring various structures for encoding scene context, while how to design a better motion decoder for multimodal motion prediction is still underexplored.
In contrast, our approach focuses on addressing this challenge with a novel transformer-based motion decoder network.
Transformer.
Transformer [50] has been widely applied in natural language processing [12, 3] and computer vision [14, 52, 5, 51, 62].
Our approach is inspired by DETR [5] and its follow-up works [67, 34, 60, 27, 30, 11, 63], especially DAB-DETR [30], where the object query is considered as the positional embedding of a spatial anchor box.
Motivated by their great success in object detection, we introduce a novel concept of motion query pair to model multimodal motion prediction with prior intention points, where each motion query pair takes charge of predicting a specific motion mode and also enables iterative motion refinement by combining with transformer decoders.
Motion TRansformer (MTR)
We propose Motion TRansformer (MTR), which adopts a novel transformer encoder-decoder structure with iterative motion refinement for predicting multimodal future motion.
The overall structure is illustrated in Figure 1.
In Sec. 3.1, we introduce our encoder network for scene context modeling.
In
Sec. 3.2, we present motion decoder network with a novel concept of motion query pair for predicting multimodal trajectories.
Finally, in Sec. 3.3, we introduce the optimization process of our framework.
3.1 Transformer Encoder for Scene Context Modeling
The future behaviors of the agents highly depend on the agents’ interaction and road map.
To encode such scene context, existing approaches have explored various strategies by building global interacting graph [17, 21] or summarizing map features to agent-wise features [37, 49].
We argue that the locality structure is important for encoding scene context, especially for the road map.
Hence, we propose a transformer encoder network with local self-attention to better maintain such structure information.
Input representation.
We follow the vectorized representation [17] to organize both input trajectories and road map as polylines.
For the motion prediction of a interested agent, we adopt the agent-centric strategy [65, 21, 49] that normalizes all inputs to the coordinate system centered at this agent.
Then, a simple polyline encoder is adopted to encode each polyline as an input token feature for the transformer encoder.
Specifically, we denote the history state of N a agents as A in ∈ N N a × t × C a , where t is the number of history frames, C a is the number of state information ( e.g. , location, heading angle and velocity), and we pad zeros at the positions of missing frames for trajectories that have less than t frames.
The road map is denoted as M in ∈ R N m × n × C m , where N m is the number of map polylines, n is the number of points in each polyline and C m is the number of attributes of each point ( e.g. , location and road type).
Both of them are encoded by a PointNet-like [40] polyline encoder as:
[Equation]
where MLP ( · ) is a multilayer perceptron network, and ϕ is max-pooling to summarize each polyline features as agent features A p ∈ R N a × D and map features M p ∈ R N m × D with feature dimension D .
Scene context encoding with local transformer encoder.
The local structure of scene context is important for motion prediction.
For example, the relation of two parallel lanes is important for modelling the motion of changing lanes, but adopting attention on global connected graph equally considers relation of all lanes.
In contrast, we introduce such prior knowledge to context encoder by adopting local attention, which better maintains the locality structure and are more memory-efficient.
Specifically, the attention module of j -th transformer encoder layer can be formulated as:
[Equation]
where MultiHeadAttn ( · , · , · ) is the multi-head attention layer [50], G
=
[ A p , M p ] ∈ N ( N a + N m ) × D concatenating the features of agents and map, and κ ( · ) denotes k -nearest neighbor algorithm to find k closest polylines for each query polyline.
PE denotes sinusoidal position encoding of input tokens, where we utilize the latest position for each agent and utilize polyline center for each map polyline.
Thanks to such local self-attention, our framework can encode a much larger area of scene context.
The encoder network finally generates both agent features A past ∈ R N a × D and map features M ∈
R N m × D , which are considered as the scene context inputs of the following decoder network.
Dense future prediction for future interactions.
Interactions with other agents heavily affect behaviors of our interested agent, and previous works propose to model the multi-agent interactions with hub-host based network [68], dynamic relational reasoning [28], social spatial-temporal network [59], etc.
However, most existing works generally focus on learning such interactions over past trajectories while ignoring the interactions of future trajectories.
Therefore, considering that the encoded features
A have already learned rich context information of all agents, we propose to densely predict both future trajectories and velocities of all agents by adopting a simple regression head on A :
[Equation]
where S i ∈
R N a × 4 includes future position and velocity of each agent at time step i , and T is the number of future frames to be predicted.
The predicted trajectories S 1: T are encoded by adopting the same polyline encoder as Eq.
(1) to encode the agents’ future states as features A future ∈ R N a × D , which are then utilized to enhance the above features A by using a feature concatenation and three
MLP layers as A =
MLP ([ A past , A future ]) .
This auxiliary task provides additional future context information to the decoder network, facilitating the model to predict more scene-compliant future trajectories for the interested agent.
The experiments in Table 3 demonstrates that this simple and light-weight auxiliary task can effectively improve the performance of multimodal motion prediction.
3.2 Transformer Decoder with Motion Query Pair
Given the scene context features, a transformer-based motion decoder network is adopted for multimodal motion prediction, where we propose motion query pair to model motion prediction as the joint optimization of global intention localization and local movement refinement.
Each motion query pair contains two types of queries, i.e. , static intention query and dynamic searching query, for conducting global intention localization and local movement refinement respectively.
As shown in Figure 2, our motion decoder network contains stacked transformer decoder layers for iteratively refining the predicted trajectories with motion query pairs.
Next, we illustrate the detailed structure.
Global intention localization aims to localize agent’s potential motion intentions in an efficient and effective manner.
We propose static intention query to narrow down the uncertainty of future trajectory by utilizing different intention queries for different motion modes.
Specifically, we generate
[Figure]
[Figure]
by adopting k-means clustering algorithm on the endpoints of ground-truth (GT) trajectories, where each intention point represents an implicit motion mode that considers both motion direction and velocity.
We model each static intention query as the learnable positional embedding of the intention point as:
[Equation]
where PE( · ) is the sinusoidal position encoding, and Q I ∈ R K× D .
Notably, each intention query takes charge of predicting trajectories for a specific motion mode, which stabilizes the training process and facilitates predicting multimodal trajectories since each motion mode has their own learnable embedding.
Thanks to their learnable and adaptive properties, we only need a small number of queries ( e.g. , 64 queries in our setting) for efficient intention localization, instead of using densely-placed goal candidates [65, 21] to cover the destinations of the agents.
Local movement refinement aims to complement with global intention localization by iteratively gathering fine-grained trajectory features for refining the trajectories.
We propose dynamic searching query to adaptively probe trajectory features for each motion mode.
Each dynamic searching query is also the position embedding of a spatial point, which is initialized with its corresponding intention point but will be dynamically updated according to the predicted trajectory in each decoder layer.
Specifically, given the predicted future trajectories Y j 1: T = { Y ji ∈ R K× 2 | i = 1 , · · · , T } in j -th decoder layer, the dynamic searching query of ( j + 1) -th decoder layer is updated as follows:
[Equation]
As shown in Figure 3, for each motion query pair, we propose a dynamic map collection module to extract fine-grained trajectory features by querying map features from a trajectory-aligned local region, which is implemented by collecting L polylines whose centers are closest to the predicted trajectory.
As the agent’s behavior largely depends on road maps, this local movement refinement strategy enables to continually focus on latest local context information for iterative motion refinement.
Attention module with motion query pair.
In each decoder layer, static intention query is utilized to propagate information among different motion intentions, while dynamic searching query is utilized to aggregate trajectory-specific features from scene context features.
Specifically, we utilize static intention query as the position embedding of self-attention module as follows:
[Equation]
where C j − 1 ∈ R K× D is query content features from ( j − 1) -th decoder layer, C 0 is initialized to zeros, and C j sa ∈ R
K× D is the updated query content.
Next, we utilize dynamic searching query as query position embedding of cross attention to probe trajectory-specific features from the outputs of encoder.
Inspired by [34, 30], we concatenate content features and position embedding for both query and key to decouple their contributions to the attention weights.
Two cross-attention modules Motion
[Figure]
Motion Query
Motion Query0.
[Figure]
Prediction of layer 𝑗
[Figure] Intention
[Figure]
Dynamic Searching Query
Predictedtrajectory
Trajectory-specificmapfeatures are adopted separately for aggregating features from both agent features A and map features M as:
C jA =
MultiHeadAttn ( query= [ C j sa , Q jS ] , key= [ A, PE A ] , value= A ) ,
C jM = MultiHeadAttn ( query= [ C j sa , Q jS ] , key= [ α ( M ) , PE α ( M ) ] , value= α ( M )) , (7) [Equation] where [ · , · ] indicates feature concatenation, α ( M ) is the aforementioned dynamic map collection module to collect L trajectory-aligned map features for motion refinement.
Note that for simplicity, in
Eq.
(6) and (7), we omit the residual connection and feed-forward network in transformer layer [50].
Finally, C j ∈ R K× D is the updated query content features for each motion query pair in j -th layer.
Multimodal motion prediction with Gaussian Mixture Model.
For each decoder layer, we append a prediction head to C j for generating future trajectories.
As the behaviors of the agents are highly multimodal, we follow [10, 49] to represent the distribution of predicted trajectories with Gaussian
Mixture Model (GMM) at each time step.
Specifically, for each future time step i ∈ { 1 , · · · , T } , we predict the probability p and parameters ( µ x , µ y , σ x , σ y , ρ ) of each Gaussian component as follows
[Equation]
where Z ji ∈ R K× 6 includes K Gaussian components N 1: K ( µ x , σ x ; µ y , σ y ; ρ ) with probability distribution p 1: K .
The predicted distribution of agent’s position at time step i can be formulated as:
[Equation]
where P ji ( o ) is the occurrence probability of the agent at spatial position o ∈ R 2 .
The predicted trajectories Y j 1: T can be generated by simply extracting the predicted centers of Gaussian components.
3.3 Training Losses
Our model is trained end-to-end with two training losses.
The first auxiliary loss is L 1 regression loss to optimize the outputs of Eq.
For the second Gaussian regression loss, we adopt negative loglikelihood loss according to Eq.
(9) to maximum the likelihood of ground-truth trajectory.
Inspired by [10, 49], we adopt a hard-assignment strategy that selects one closest motion query pair as positive
Gaussian component for optimization, where the selection is implemented by calculating the distance between each intention point and the endpoint of GT trajectory.
The Gaussian regression loss is adopted in each decoder layer, and the final loss is the sum of the auxiliary regression loss and all the
Gaussian regression loss with equal loss weights.
Please refer to appendix for more loss details.
Experiments 4.1 Experimental Setup
Dataset and metrics.
We evaluate our approach on the large-scale Waymo Open Motion Dataset (WOMD) [15], which mines interesting interactions from real-world traffic scenes and is currently [Table] [Table] the most diverse interactive motion dataset.
There are two tasks in WOMD with separate evaluation metrics: (1) The marginal motion prediction challenge that independently evaluates the predicted motion of each agent (up to 8 agents per scene).
The joint motion prediction challenge that needs to predict the joint future positions of 2 interacting agents for evaluation.
Both of them providesecond of history data and aim to predict 6 marginal or joint trajectories of the agents for 8 seconds into the future.
There are totally 487 k training scenes, and about 44 k validation scenes and 44 k testing scenes for each challenge.
We utilize the official evaluation tool to calculate the evaluation metrics, where the mAP and miss rate are the most important ones as in the official leaderboard[55, 54].
Implementation details.
For the context encoding, we stack 6 transformer encoder layers.
The road map is represented as multiple polylines, where each polyline contains up to 20 points (about 10 m in WOMD).
We select N m = 768 nearest map polylines around the interested agent.
The number of neighbors in encoder’s local self-attention is set to 16.
The encoder hidden feature dimension is set as D = 256 .
For the decoder modules, we stack 6 decoder layers.
L is set to 128 to collect the closest map polylines from context encoder for motion refinement.
By default, we utilize 64 motion query pairs where their intention points are generated by conducting k-means clustering algorithm on the training set.
To generate 6 future trajectories for evaluation, we use non-maximum suppression (NMS) to select top 6 predictions from 64 predicted trajectories by calculating the distances between their endpoints, and the distance threshold is set as 2 . 5 m .
Please refer to Appendix for more details.
Training details.
Our model is trained in an end-to-end manner by AdamW optimizer with a learning rate of 0.0001 and batch size of 80 scenes.
We train the model for 30 epochs with 8 GPUs (NVDIA
RTX 8000), and the learning rate is decayed by a factor of 0.5 every 2 epochs from epoch 20.
The weight decay is set as 0.01 and we do not use any data augmentation.
MTR-e2e for end-to-end motion prediction.
We also propose an end-to-end variant of MTR, called
MTR-e2e, where only 6 motion query pairs are adopted so as to remove NMS post processing.
In the training process, instead of using static intention points for target assignment as in MTR, MTR-e2e selects positive mixture component by calculating the distances between its 6 predicted trajectories and the GT trajectory, since 6 intention points are too sparse to well cover all potential future motions.
4.2 Main Results
Performance comparison for marginal motion prediction.
Table 1 shows our main results for marginal motion prediction, our MTR outperforms previous ensemble-free approaches [21, 37] with [Table] remarkable margins, increasing the mAP by +8 .
48% and decreasing the miss rate from 15 .
64% to 13 .
In particular, our single-model results of MTR also achieve better mAP than the latest work
MultiPath++ [49] [Table] where it uses a novel model ensemble strategy that boosts its performance.
Table 1 also shows the comparison of MTR variants.
MTR-e2e achieves better minADE and minFDE by removing NMS post-processing, while MTR achieves better mAP since it learns explicit meaning of each motion query pair that produces more confident intention predictions.
We also propose a simple model ensemble strategy to merge the predictions of MTR and MTR-e2e and utilize NMS to remove redundant predictions (denoted as MTR-ens), and it takes the best of both models and achieves much better mAP.
By adopting such ensemble strategy to 7 variants of our framework ( e.g. , more decoder layers, different number of queries, larger hidden dimension), our advanced ensemble results (denoted as MTR-Advanced-ens) achieve best performance on the test set leaderboard.
Performance comparison for joint motion prediction.
To evaluate our approach for joint motion prediction, we combine the marginal predictions of two interacting agents into joint prediction as in [7, 15, 45], where we take the top 6 joint predictions from 36 combinations of these two agents.
The confidence of each combination is the product of marginal probabilities.
Table 2 shows that our approach outperforms state-of-the-arts [37, 45] with large margins on all metrics.
Particularly, our MTR boosts the mAP from 12 .
39% to 20 .
37% and decreases the miss rate from 49 .
42% to 44 .
The remarkable performance gains demonstrate the effectiveness of MTR for predicting scene-consistent future trajectories.
Besides that, we also provide some qualitative results in Figureto show our predictions in complicated interacting scenarios.
As of May 19, 2022, our MTR ranks 1 st on the motion prediction leaderboard of WOMD for both two challenges [55, 54].
Our approach with more ensembled variants of MTR ( i.e. , MTR-Advacnedens) also won the champion of Motion Prediction Challenge in Waymo Open Dataset Challenge 2022 [56, 44].
The significant improvements manifest the effectiveness of MTR framework.
4.3 Ablation Study
We study the effectiveness of each component in MTR.
For efficiently conducting ablation experiments, we uniformly sampled 20% frames (about 97 k scenes) from the WOMD training set according to their default order, and we empirically find that it has similar distribution with the full training set.
All models are evaluated with marginal motion prediction metric on the validation set of WOMD.
Effects of the motion decoder network.
We study the effectiveness of each component in our decoder network, including global intention localization, iterative refinement and local movement refinement.
Table 3 shows that all components contributes remarkably to the final performance in terms of the official ranking metric mAP.
Especially, our proposed static intention queries with intention points achieves much better mAP ( i.e. , +4 .
26% ) than the latent learnable embeddings thanks to its mode-specific querying strategy, and both the iterative refinement and local movement refinement strategy continually improve the mAP from 30 .
59% to 32 .
34% by aggregating more fine-grained trajectory features for motion refinement.
Effects of dense future prediction.
Table 3 shows that our proposed dense future prediction module significantly improves the quality of predicted trajectories ( e.g. , +1 .
78% mAP), which verifies that future interactions of the agents’ trajectories are important for motion prediction and our proposed strategy can learn such interactions to predict more reliable trajectories.
[Figure]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure] [Figure] Effects of local attention for context encoding.
Table 4 shows that by taking the same number of map polylines as input, local self-attention in transformer encoder achieves better performance than global attention ( i.e. , +0 . 77 % mAP for 256 polylines and +1 .
12% mAP for 512 polylines), which verifies that the input local structure is important for motion prediction and introducing such prior knowledge with local attention can benefit the performance.
More importantly, local attention is more memory-efficient and the performance keeps growing when improving the number of map polylines from 256 to 1,024, while global attention will run out of memory due to its quadratic complexity.
Effects of the number of motion query pairs with different training strategies.
As mentioned before, during training process, MTR and MTR-e2e adopt two different strategies for assigning positive mixture component, where MTR depends on static intention points (denoted as α ) while
MTR-e2e utilizes predicted trajectories (denoted as β ).
Figure 4 investigates the effects of the number of motion query pairs under these two strategies, where we have the following observations: (1)
When increasing the number of motion query pairs, strategy α achieves much better mAP and miss rate than strategy β .
Because intention query points can ensure more stable training process since each intention query points is responsible to a specific motion mode.
In contrast, strategy β depends on unstable predictions and the positive component may randomly switch among all components, so a large number of motion query pairs are hard to be optimized with strategy β .
The explicit meaning of each intention query point also illustrates the reason that strategy α consistently achieves much better mAP than strategy β , since it can predict trajectories with more confident scores to benefit mAP metric.
(3) From another side, when decreasing the number of motion query pairs, the miss rate of strategy α greatly increases, since a limit number of intention query points can not well cover all potential motions of agents.
Conversely, strategy β works well for a small number of motion query pairs since its queries are not in charge of specific region and can globally adapt to any region.
Conclusion
In this paper, we present MTR, a novel framework for multimodal motion prediction.
The motion query pair is defined to model motion prediction as the joint optimization of global intention localization and local movement refinement.
The global intention localization adopts a small set of learnable static intention queries to efficiently capture agent’s motion intentions, while the local movement refinement conducts iterative motion refinement by continually probing fine-grained trajectory features.
The experiments on both marginal and joint motion prediction challenges of large-scale WOMD dataset show that our approach achieves state-of-the-art performance.
Limitations.
The proposed framework adopts an agent-centric strategy to predict multimodal future trajectories for one interested agent, leading redundant context encoding for multiple interested agents in the same scene.
Hence, how to develop a framework that can simultaneously predict multimodal motion for multiple agents is one important future work.
Besides, the rule-based post-processing can result in suboptimal predictions for minADE and minFDE metrics and how to design a better strategy to produce a required number of future trajectories (e g., 6 trajectories) from full multimodal predictions (e g., 64 predictions) is also worth exploring for a more robust framework.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work is only for academic research purpose.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Our implementation details are posted on Section 4.1 and the supplemental materials.
The data is public dataset and our code will be available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.1 for training details, and the data splits of ablation experiments are mentioned in Section 4.3.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We use fixed random seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See our reference.
(b) Did you mention the license of the assets?
[No] They are publicly available for academic use.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] They are publicly available for academic use.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] We only use the public dataset. 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Implementation Details
More architecture details. We train a single model for predicting the future motion of all three categories ( i.e. , Vehicle, Pedestrian, Cyclist), and each category has their own motion query pairs. The input history state A in contains three types of information, including agent history motion state ( i.e. , position, object size, heading angle and velocity), one-hot category mask of each agent and one-hot time embedding of each history time step. The polyline encoder for A in contains a three-layer
MLP with feature dimension 256. The input map feature M in contains three types of information, including the position of each polyline point, the polyline direction at each point, and the type of each polyline. The polyline encoder for M in contains a five-layer MLP with feature dimension 64, where we adopt a smaller feature dimension since the number of map polylines is much larger than the number of agents. Both two polyline encoders are finally projected to 256 feature dimension with another linear layer separately.
For the dense future prediction module, we adopt a three-layer MLP with intermediate feature dimension 512 for predicting future position and velocity of all agents.
For the prediction head in each decoder layer, we adopt a three-layer MLP with intermediate feature dimension 512, and the model weights are not shared across different decoder layers.
The details of Gaussian regression loss.
Given the predicted Gaussian Mixture Models for a specific future time step, we adopt negative log-likelihood loss to maximum the likelihood of the agent’s ground-truth position ( ˆ Y x , ˆ Y y ) at this time step, and the detailed loss can be formulated as:
[Equation]
− log ( p h ) , where d x = ˆ Y x − µ x , d y =
ˆ Y y − µ y , and N h ( µ x , σ x ; µ y , σ y ; ρ ) is the selected positive Gaussian component for optimization.
p h is the predicted probability of this selected positive Gaussian component, and we adopt cross entropy loss in the above equation to maximum the probability of selected positive Gaussian component.
Final training losses.
Given the above definition of Gaussian regression loss, the final training loss of
MTR framework is the sum of all the Gaussian regression loss in each decoder layer and the auxiliary regression loss with equal loss weights.
[Figure]
The distribution of static intention points.
As mentioned in the paper, we adopt k-means clustering algorithm to generate 64 static intention points for each category.
As shown in Figure 6, the generated intention points can well cover most motion intentions by jointly considering both the velocity and direction of future motion.
For example, vehicle category has a series of intention points in the heading direction that actually model the same intent direction with different velocities.
B Per-class Results of MTR Framework
As shown in Table 5 and
[Table]
we report the per-category performance of our approach for both the marginal and joint motion prediction challenges of Waymo Open Motion Dataset [15] for reference.
[Table]
[Table]
C Evaluation of Dense Future Prediction
As introduced in the paper, we propose the dense future prediction module to enable the future interactions among the agents, where the accuracy of the predicted trajectories for all neighboring agents is important for predicting better multimodal trajectories of our interested agent.
As shown in
Table 7, our simple dense future prediction module can predict accurate future trajectory for all agents, where the average miss rate with 6 .
m distance threshold is 21 .
11 % for future motion prediction ofseconds.
It’s worth noting that this miss rate is achieved with a single predicted future trajectory for each agent.
Although the trajectory generated by the dense future prediction module is still not as accurate as that of the decoder network, they can already benefit the multimodal motion prediction of our interested agent by enabling the future interactions of the agents.
D Performance Comparison on Argoverse 2 Dataset
The Argoverse 2 Motion Forecasting Dataset [57] is another large-scale dataset for motion prediction, which contains 250,000 scenarios for training and validation.
The model needs to take the history five seconds of each scenarios as niput, and predict the six-second future trajectories of one interested agent, where HDMap is always available to provide map context information.
To train our model on
[Table]
[Table]
this dataset, we adopt the same hyper-parameters as in Waymo dataset
[Table]
except that the model takes five-second history trajectories as input and needs to predict six-second future trajectories.
We compare out approach with the top-10 submissions on the leaderboard of Argoverse2 dataset [1], where most submissions are tailored for Argoverse2 Motion Forcasting Competition 2022 and are highly competitive.
As shown in Table 8, our MTR framework achieves new state-of-theart performance with remarkable gains on the miss-rate related metrics, demonstrating the great generalizability and robustness of our approach.
[Table]
[Table]
E More Ablation Studies
Effects of the number of neighbors for local self-attention.
As shown in Table 9, using 4 neighbors in local self-attention already achieves good performance in terms of mAP
[Table]
and the mAP metric keeps growing when increasing the number of neighbors from 4 to 16.
However, when we increase the neighbors to 64 for conducting local self-attention, the performance drops a bit ( i.e. , from 34 .
37% to 33 .
It demonstrates that a small number of neighbors can better maintain the local structure of input elements and is easier to be optimized for achieving better performance, and a small number of neighbors is also much more computational- and memory-efficient than a larger number of neighbors for conducting self-attention.
Effects of the number of polylines for dynamic map collection.
Table 10 shows that increasing the number of collected map polylines from 32 to 128 can constantly improve the mAP metric by retrieving trajectory-specific map features with larger receptive field.
However, the performance drops a bit (- 0 .
41% mAP) when collecting 256 map polylines for refining the trajectory, which shows that a larger number of collected map polylines may involve more noise and can not provide accurate trajectory-specific map features refinement.
[Table]
[Table]
Effects of the number of decoder layers.
Table 11 shows that increasing the number of decoder layers can constantly improve the performance, which demonstrates that our stacked transformer decoder layers can iteratively refine the predicted trajectories by continually aggregating more accurate trajectory-specific features.
By default, we adopt 6 decoder layers by considering the trade-off between the performance and the efficiency of MTR framework.
Effects of the distribution of static intention points.
As introduced in the paper, we adopt k-means algorithm to generate 64 static intention points as the basis of our motion query pairs.
We ablate another simple uniform sampling strategy to generate the same number of static intention points for each category, where we uniformly sample 8 × 8 = 64 static intention points by considering the range of trajectory distribution of each category (see Figure 6).
As shown in Table 12, the performance drops significantly when replacing k-means algorithm with the uniform sampling for generating static intention points.
It verifies that our k-means based algorithm can produce better distribution of static intention points, which can capture more accurate and more complete future motion intentions with a small number of static intention points.
Figure 6 also demonstrates the effectiveness of our default static intention points for capturing multimodal motion intentions.
F Qualitative Results
We provide more qualitative results of our MTR framework in Figure 7.
G Notations
As shown in Table 13, we provide a lookup table for notations in the paper.
[Figure]
predictedtoyieldfor
[Figure]
predictedtoslowdownwithhighconfidenceandyieldfortwopedestrianswhoarecrossingtheroad.
[Figure]
in each scene (green rectangle), where our model predicts 6 multimodal future trajectories for each of them.
For other agents (blue rectangle), a single trajectory is predicted by dense future prediction module.
We use gradient color to visualize the trajectory waypoints at different future time step, and trajectory confidence is visualized by setting different transparent.
Abbreviation: Vehicle (V), Pedestrian (P).
[Table]