Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals
Anonymous Author(s)
AfﬁliationAddress email
Abstract:
Accurately predicting the future motion of surrounding vehicles requires reasoning about the inherent uncertainty in driving behavior.
This uncertainty can beloosely decoupled into lateral (e.g., keeping lane, turning) and longitudinal (e.g.,accelerating, braking).
We present a novel method that combines learned discretepolicy rollouts with a focused decoder on subsets of the lane graph.
The policyrollouts explore different goals given current observations, ensuring that the modelcaptures lateral variability.
Longitudinal variability is captured by our latent variable model decoder that is conditioned on various subsets of the lane graph.
Ourmodel achieves state-of-the-art performance on the nuScenes motion predictiondataset, and qualitatively demonstrates excellent scene compliance.
Detailed ablations highlight the importance of the policy rollouts and the decoder architecture.
Introduction
To safely and efﬁciently navigate through complex trafﬁc scenes, autonomous vehicles need the ability to predict the intent and future trajectories of surrounding vehicles.
There is inherent uncertaintyin predicting the future, making trajectory prediction a challenging problem.
However, there’s structure to vehicle motion that can be exploited.
Drivers usually tend to follow trafﬁc rules and followthe direction ascribed to their lanes.
High deﬁnition (HD) maps of driving scenes provide a succinctrepresentation of the road topology and trafﬁc rules, and have thus been a critical component ofrecent trajectory prediction models as well as public autonomous driving datasets.
Early work [1] encodes HD maps using a rasterized bird’s eye view image and convolutional layers.
While this approach exploits the expressive power of modern CNN architectures, rasterization ofthe map can be computationally inefﬁcient, erase information due to occlusions, and require largereceptive ﬁelds to aggregate context.
The recently proposed VectorNet [2] and LaneGCN [3] modelsdirectly encode structured HD maps, representing lane polylines as nodes of a graph.
VectorNet aggregates context using attention [4], while LaneGCN proposes a dilated variant of graph convolution[5] to aggregate context along lanes.
These approaches achieve state-of-the-art performance usingfewer parameters than rasterization-based approaches.
The above methods represent the HD map as a graph and encode the input context into a singlecontext vector as shown in Fig.1.
The context vector is then used by a multimodal prediction header[1, 6] to output multiple plausible future trajectories.
The prediction header thus needs to learna complex mapping, from the entire scene context to multiple future trajectories, often leading topredictions that go off the road or violate trafﬁc rules.
In particular, the prediction header needs toaccount for both lateral or route variability (e.g. will the driver change lane, will they turn rightetc.)
as well as longitudinal variability (e.g. will the driver accelerate, brake, maintain speed).
Thisdecoupling of routes and motion proﬁles for trajectories has been used in path planning [7, 8], andmore recently in prediction [9].
Our core insight is that the graph structure of the scene can additionally be leveraged to explicitlymodel the lateral or route variability in trajectories.
We propose a novel approach for trajectoryprediction termed Prediction via Graph-based Policy (PGP).
Our approach relies on two key ideas.
Predictions conditioned on traversals: We selectively aggregate part of the scene context for eachprediction, by sampling path traversals from a learned behavior cloning policy as shown in Fig.
By more directly selecting the subset of the graph that is used for each prediction, we lessen the representational demands on the output decoder.
Additionally, the probabilistic policy leads to a diverseset of sampled paths and captures the lateral variability of the multimodal trajectory distribution.
Latent variable for longitudinal variability: To account for longitudinal variability of trajectories,we additionally condition our predictions with a sampled latent variable.
This allows our model topredict distinct trajectories even for identical path traversals.
We show through our experiments thatthis translates to greater longitudinal variability of predictions.
We summarize our main contributions on multimodal motion prediction using HD maps:
• A novel method which combines discrete policy roll-outs with a lane-graph subset decoder.
• State-of-the-art performance on the nuScenes motion prediction challenge. • Extensive ablations demonstrating ability to capture lateral and longitudinal motion variations.
Related Work
Graph representation of HD maps: Most self-driving cars have access to HD vector maps, whichinclude detailed geometric information about objects such as lanes, crosswalks, stop signs, and more.
VectorNet [2] encodes the scene context using a hierarchical representation of map objects and agenttrajectories.
Each component is represented as a sequence of vectors, which are then processed bya local graph network.
The resulting features are aggregated globally via a fully-connected graphnetwork.
LaneGCN [3] extracts a lane graph from the HD map, and uses a graph convolutionalnetwork to compute lane features.
These features are combined with both agent and other lanefeatures in a fusion network.
Both methods utilize the entire graph for making predictions, relyingon the header to identify the most relevant features.
Multimodal trajectory prediction: Researchers have proposed a variety of ways to model themultiple possible future trajectories that vehicles may take.
One approach is to model the outputas a probability distribution over trajectories, using either regression [1], ordinal regression [6], orclassiﬁcation [10].
Another approach models the output as a spatial-temporal occupancy grid [11].
Sampling methods use stochastic policy roll outs [12, 13] or latent variable models that map alatent variable sampled from a simple distribution to a predicted trajectory. Latent variable modelsare trained as GANs [14, 15], CVAEs [16, 17], or directly using the winner-takes-all regressionloss [18].
These models must learn a one-to-many mapping from the entire input context (except therandom variable) to multiple trajectories, and can lead to predictions that are not scene compliant.
Goal-conditioned trajectory prediction: Rather than learning a one to many mapping from theentire context to multiple future trajectories, methods such as TnT [19], LaneRCNN [20], and PECNet [21] condition each prediction on goals of the driver.
Conditioning predictions on future goals makes intuitive sense and helps leverage the HD map by restricting goals to be near the lanes.
However, one limitation is that over moderate time horizons, there can be multiple paths that reach agiven goal location.
Additionally, certain plausible goal locations might be unreachable due to constraints in the scene that are not local to the goal location, e.g., a barrier that blocks a turn lane.
Incontrast, our method conditions on paths traversed in a lane graph, which ensures that the inferredgoal is reachable.
Furthermore, the traversed path provides a stronger inductive bias than just thegoal location.
A similar stream of work conditions on candidate lane centerlines as goals (e.g., WIMP [22], GoalNet [9], CXX [23]).
While the lane centerline provides more local context thanjust the goal, accounting for lane changes can be difﬁcult.
Additionally, routes need to be deterministically chosen, with multiple trajectories predicted along the selected route.
Our approach allowsfor probabilistic sampling of both routes and motion proﬁles.
In scenes with just a single plausibleroute, our model can use its prediction budget of K trajectories purely for different plausible motionproﬁles.
Closest to our work is P2T [24].
They predict trajectories conditioned on paths explored byan IRL policy over a grid deﬁned over the scene.
However, they use a rasterized BEV image for thescene, which leads to inefﬁcient encoders and loss of connectivity information due to occlusions.
Additionally, their model cannot generate different motion proﬁles along a sampled path.
Formulation
We predict the future trajectories of vehicles of interest, conditioned on their past trajectory, the pasttrajectories of nearby vehicles and pedestrians, and the HD map of the scene.
We represent the sceneand predict trajectories in the bird’s eye view and use an agent-centric frame of reference alignedalong the agent’s instantaneous direction of motion.
3.1 Trajectory representation
We assume access to past trajectories of agents in the scene obtained from on-board detectors andmulti-object trackers.
We represent the past trajectory of agent i as a sequence of motion statevectors s i − t h :
=
[ s i − t h , ..., s i − 1 , s i 0 ] over the past t h time steps.
Each s it =
[ x it , y it , v it , a it , ω it , I i ] ,where x it , y it are the BEV location co-ordinates, v it , a it and ω it are the speed, acceleration and yawrate of the agent at time t , and I i is an indicator with value 1 for pedestrians and 0 for a vehicles.
We nominally assign the index 0 to the target vehicle, and timestamp 0 to the time of prediction.3.2 Representing
HD maps
as lane graphs
Nodes: We represent the HD map as a directed graph G ( V, E ) . The network of lane centerlinescaptures both, the direction of trafﬁc ﬂow, and the legal routes that each driver can follow.
We seekto use both as inductive biases for our model. We thus use lane centerlines as nodes ( V ) in our graph. We consider all lane centerlines within a ﬁxed area around the target vehicle.
To ensure that eachnode represents a lane segment of a similar length, we divide longer lane centerlines into smallersnippets of a ﬁxed length, and discretize them to a set of N poses.
Each snippet corresponds to anode in our graph, with a node v represented by a sequence of feature vectors f v 1: N =
[ f v 1 , ..., f vN ] .
Here each f vn =
[ x vn , y vn , θ vn , I vn ] , where x vn , y vn and θ vn are the location and yaw of the n th pose of vand
I vn is a 2-D binary vector indicating whether the pose lies on a stop line or crosswalk.
Thus, ournode features capture both the geometry as well as trafﬁc control elements along lane centerlines.
Edges: We constrain edges ( E ) in the lane graph such that any traversed path through the graphcorresponds to a legal route that a vehicle can take in the scene.
We consider two types of edges.
Successor edges ( E suc ) connect nodes to the next node along a lane.
A given node can have multiplesuccessors if a lane branches out at an intersection.
Similarly, multiple nodes can have the samesuccessor if two or more lanes merge.
To account for lane changes, we additionally deﬁne proximaledges ( E prox ) between neighboring lane nodes if they are within a distance threshold of each otherand their directions of motion are within a yaw threshold.
The yaw threshold ensures that proximaledges are not erroneously assigned in intersections where multiple lanes cross each other.
3.3 Output representation
To account for multimodality of the distribution of future trajectories, we output a set of K trajectories [ τ 11: t f , τ 21: t f , ..., τ K 1: t f ] for the target vehicle consisting of future x-y locations over a predictionhorizon of t f time steps.
Each of the K trajectories represents a mode of the predictive distribution,ideally corresponding to different plausible routes or different motion proﬁles along the same route.
Proposed Model
[Figure] Fig. 2 provides an overview of our model.
It consists of three interacting modules trained end-to-end.
The graph encoder (Sec. 4.1) forms the backbone of our model.
It outputs learned representationsfor each node of the lane graph, incorporating the HD map as well as surrounding agent context.
The policy header (Sec. 4.2) outputs a discrete probability distribution over outgoing edges at eachnode, allowing us to sample paths in the graph.
Finally, our attention based trajectory decoder (Sec.4.3) outputs trajectories conditioned on paths traversed by the policy and a sampled latent variable.
4.1 Encoding scene and agent context
Inspired by the simplicity and effectiveness of graph based encoders for trajectory prediction [2, 3],we seek to encode all agent features and map features as node encodings of our lane graph G ( V, E ) .
GRU encoders.
Both, agent trajectories and lane polylines form sequences of features with a welldeﬁned order.
We ﬁrst independently encode both sets of features using gated recurrent unit (GRU)encoders.
We use three GRU encoders for encoding the target vehicle trajectory s 0 − t h :0 , surroundingvehicle trajectories s i − t h :
and node features f v 1: N .
These output the motion encoding h motion , agentencodings h iagent and initial node encodings h vnode respectively.
Agent-node attention.
Drivers co-operate with other drivers and pedestrians to navigate throughtrafﬁc scenes. Thus, surrounding agents serve as a useful cue for trajectory prediction. Of particularinterest are agents that might interact with the target vehicle’s route.
We thus update node encodingswith nearby agent encodings using scaled dot product attention [4].
We only consider agents withina distance threshold of each lane node to update the node encoding.
This allows our trajectorydecoder (Sec 4.3) to selectively focus on agents that might interact with speciﬁc routes that thetarget vehicle might take.
We obtain keys and values by linearly projecting encodings h iagent ofnearby agents, and the query by linearly projecting h vnode .
Finally, the updated node encoding isobtained by concatenating the output of the attention layer with the original node encoding.
GNN layers.
With the node encodings updated with nearby agent features, we exploit the graphstructure to aggregate local context from neighboring nodes using graph neural network (GNN)layers.
We experiment with graph convolution (GCN) [5] and graph attention (GAT) [25] layers.
For the GNN layers, we treat both successor and proximal edges as equivalent and bidirectional.
This allows us to aggregate context along all directions around each node.
The outputs of the GNNlayers serve as the ﬁnal node encodings learned by the graph encoder.
4.2 Discrete policy for graph traversal
Every path in our directed lane graph corresponds to a plausible route for the target vehicle.
However, not every route is equally likely.
For example, the past motion of the target vehicle approachingan intersection might indicate that the driver is preparing to make a turn rather than go straight.
Aslow moving lane make it likelier for the target vehicle to change lane rather than maintain lane.
We seek to learn a policy π route for graph traversal such that sampled roll-outs of the policy correspond to likely routes that the target vehicle would take in the future.
We represent our policy asa discrete probability distribution over outgoing edges at each node.
We additionally include edgesfrom every node to an end state to allow π route to terminate at a goal location.
The edge probabilities are output by the policy header shown in Fig.
The policy header uses an MLP with sharedweights to output a scalar score for each edge ( u, v ) given by,
[Equation]
The scoring function thus takes into account the motion of the target vehicle as well as local sceneand agent context at the speciﬁc edge.
We then normalize the scores using a softmax layer for alloutgoing edges at each node to output the policy for graph traversal,
[Equation]
We train the policy header using behavior cloning.
For each prediction instance, we use the groundtruth future trajectory to determine which nodes were visited by the vehicle.
We can naively assigneach pose in the future trajectory to the closest node in the graph. However, this can lead to erroneousassignment of nodes in intersections, where multiple lanes intersect. We thus only consider lanenodes whose direction of motion is within a yaw threshold of the target agent ’s pose.
An edge( u, v ) is treated as visited if both nodes u and v are visited.
We use negative log likelihood of theedge probabilities for all edges visited by the ground truth trajectory ( E gt ), as the loss function fortraining the graph traversal policy, given by [Equation]
4.3 Decoding trajectories conditioned on traversals
Sampling roll-outs of π route yields plausible future routes for the target vehicle.
We posit that themost relevant context for predicting future trajectories is along these routes and propose a trajectorydecoder that selectively aggregates context along the sampled routes.
Given a sequence of nodes [ v 1 , v 2 , ..., v M ] corresponding to a sampled policy roll-out, our trajectorydecoder uses multi-head scaled dot product attention [4] to aggregate map and agent context overthe node sequence as shown in Fig.
We linearly project the target vehicle’s motion encodingto obtain the query, while we linearly project the node features [ h v 1 node , h v 2 node , ..., h v M node ] to obtainkeys and values for computing attention.
The multi-head attention layer outputs a context vector
C encoding the route.
Each distinct policy roll-out yields a distinct context vector, allowing us topredict trajectories along a diverse set of routes.
Diversity in routes alone does not account for the multimodality of future trajectories.
Drivers canbrake, accelerate and follow different motion proﬁles along a planned route.
To allow the modelto output distinct motion proﬁles, we additionally condition our predictions with a sampled latentvector z . Unlike routes, vehicle velocities and accelerations vary on a continuum. We thus sample zfrom a continuous distribution. We use the multivariate standard normal distribution for simplicity.
[Table] Finally, to sample a trajectory τ k 1: t f from our model, we sample a roll-out of π route and obtain C k ,we sample z k from the latent distribution and concatenate both with h motion and pass them throughan MLP to output τ k 1: t f the future locations over t f timesteps,
[Equation]
The sampling process can often be redundant, yielding similar or repeated trajectories.
However ourlight-weight encoder and decoder heads allows us to sample a large number of trajectories in parallel.
To obtain a ﬁnal set of K modes of the trajectory distribution, we use K-means clustering and outputthe cluster centers as our ﬁnal set of K predictions [ τ 11: t f , τ 21: t f , ..., τ K 1: t f ] .
We train our decoder usingthe winner takes all average displacement error with respect to the ground truth trajectory ( τ gt ) inorder to not penalize the diverse plausible trajectories output by our model,
[Equation]
We train our model end-to-end using a multi-task loss combining losses from Eq. 3 and Eq. 5,
[Equation]
Experiments
Dataset: We evaluate our method on nuScenes [26], a self-driving car dataset collected in Bostonand Singapore.
nuScenes contains 1000 scenes, each 20 seconds, with ground truth annotations and HD maps.
Vehicles have manually-annotated 3D bounding boxes, which are published at [-
-]2 Hz.
The prediction task is to use the past 2 seconds of object history and the map to predict the next 6seconds.
We use the standard split from the nuScenes software kit [27].
Metrics: To evaluate our model, we use the standard metrics on the nuScenes leaderboard [27].
The minimum average displacement error (ADE) over the top K predictions (MinADE K ).
The missrate (MissRate K, 2 ) only penalizes predictions that are further than 2 m from the ground truth.
Theoffroad rate measures the fraction of predictions that are off the road.
Since all examples in nuScenesare on the road, this should be zero.
Additionally, we report metrics measuring sample diversity ofa set of K predictions. To measure lateral diversity, we report the average number of distinct ﬁnallanes reached, and the variance of ﬁnal heading angle of the target vehicle ( σ 2 yaw ) for the set of Ktrajectories. To measure longitudinal diversity, we report the variance of average speeds ( σ 2 speed )and accelerations ( σ 2 acc ) for the set of K trajectories.
Comparison to the state of the art: We report our results on the standard benchmark split of thenuScenes prediction dataset in table 1, comparing with the top performing entries on the nuScenesleaderboard.
We achieve state of the art results on almost all metrics, signiﬁcantly outperforming theprevious best entry P2T [24] on the MinADE K and MissRate metrics, while achieving comparableoff-road rate.
This suggests that our model achieves better coverage of the modes of the trajectorydistribution, while still predicting trajectories that are scene-compliant.
Encoder ablations: We analyze the effects of our graph structure and components of the graphencoder by performing ablations on the graph encoder reported in table 2.
In particular we analyzethe effect of including proximal edges, modeling surrounding agents with agent-node attention and
[Table] [Table]
[Table] [Table] ﬁnally aggregating local context using GCN [5] or GAT [25] layers.
We get improvement across allmetrics by adding proximal edges, and agent-node attention, suggesting the importance of modelinglane changes and agent context.
Somewhat surprisingly, adding GNN layers gives ambiguous resultswith GCN layers achieving slightly worse results and GAT layers performing on par with the encoderwithout GNN layers.
This could be because the multi-head attention layer aggregates context acrossthe entire traversed path, making the GNNs redundant.
Decoder ablations: We next analyze the effect of our traversal and latent variable based decoder.
We compare several decoders, all built on top of our proposed encoder with both types of edges,agent-node attention and 2 GAT layers.
First, we consider the multimodal regression header from MTP [1].
Next we consider ablations of our decoder without the graph traversals and without thelatent variable conditioning.
Finally, we consider a model that conditions predictions on sampledgoals at different node locations, instead of traversals.
Table 3 reports quantitative results while Fig.3 shows qualitative examples comparing the decoders.
We make the following observations.
MTP generally fares worse compared to the other decoders, particularly in terms of offroad rate.
Wenote from Fig. 3 that while it generates a diverse set of trajectories, several veer off-road.
The decoders conditioned purely on the latent variable or purely on traversals both fare worse interms of MinADE and MissRate compared to our decoder conditioned on both.
From the samplediversity metrics (Tables 4 and 5) and qualitative examples (Fig.3) we observe that this is for different reasons.
The ‘LV only’ decoder generates diverse motion proﬁles, but almost always predictstrajectories along a single route, leading to poor lateral diversity of trajectories.
On the other hand,the ’Traversal only’ decoder predicts trajectories over a variety of routes, but lacks diversity in termsof motion proﬁles.
Finally, the ‘Goals + LV’ decoder also fares worse compared to our ‘Traversals + LV’ decoder, again,especially in terms of off-road rate.
Qualitatively, we observe that this is due to two types of errors.
First, it tends to predict spurious goals which aren’t reachable for the target vehicle (Fig.3 3 (cid:13) , 4 (cid:13) ),and second, while it predicts correct goals, it generates trajectories that don’t follow accurate pathsto those goals (Fig.3 2 (cid:13) , 6 (cid:13) ).
Conclusions
We presented a novel method for multimodal trajectory prediction conditioned on paths traversed ina lane graph of the HD map by a discrete policy, and a sampled latent variable. Through experimentalanalysis and ablation studies using the publicly available nuScenes dataset, we showed that
• Selectively conditioning predictions on lane-graph traversals leads to trajectories that are (i) diverse in terms of routes, and (ii) precise and scene compliant with the lowest offroad-rates. • Additionally conditioning predictions on sampled latent variables leads to trajectories that arediverse in terms of motion proﬁles.
• Both put together lead to state of the art results in terms of MinADE and MissRate metrics.