Learning to Predict Vehicle Trajectories with Model-based Planning
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Predicting the future trajectories of on-road vehicles is critical forautonomous driving.
In this paper, we introduce a novel prediction frameworkcalled PRIME, which stands for Prediction with Model-based Planning.
Unlikerecent prediction works that utilize neural networks to model scene context andproduce unconstrained trajectories, PRIME is designed to generate accurate andfeasibility-guaranteed future trajectory predictions.
PRIME guarantees the trajectory feasibility by exploiting a model-based generator to produce future trajectories under explicit constraints and enables accurate multimodal prediction byutilizing a learning-based evaluator to select future trajectories.
We conduct experiments on the large-scale Argoverse Motion Forecasting Benchmark, where
PRIME outperforms the state-of-the-art methods in prediction accuracy, feasibility, and robustness under imperfect tracking.
Introduction
In the architecture of autonomous driving, prediction serves as the bridging module that reasonsfuture states based on the perceived information from upstream detection and tracking and providesthe predicted future states to facilitate the downstream planning.
Therefore, making accurate andreasonable trajectory predictions for on-road vehicles is vital for planning safe, efﬁcient, and comfortable motion for self-driving vehicles (SDVs).
The widely known challenge of trajectory prediction lies in modeling multi-agent interaction andinferring multimodal future states under driving scenarios.
Traditional methods [1, 2, 3, 4, 5] produce motion forecasting by handcrafted rules or models with embedded physical and environmentalfeatures, which are insufﬁcient for modeling interactive agents in complex scenes.
Learning-basedapproaches [6, 7, 8], with deep neural networks to fuse scene context information and generatefuture trajectories, signiﬁcantly promote the prediction accuracy and dominate the recent motionforecasting benchmarks for autonomous driving [9, 10].
Despite achieving steady improvement in accuracy, much less attention has been paid to the feasibility and robustness of learning-based trajectory prediction.
Indeed, most trafﬁc participants operateunder their inherent kinematic constraints (e.g., non-holonomic motion constraints for vehicles)while in compliance with the road structure (e.g., lane connectivity, static obstacles) and semanticinformation (e.g., trafﬁc lights, speed limits).
All these kinematic and environmental constraintsexplicitly regularize the trajectory space.
However, most existing approaches model trafﬁc agents aspoints and generate future trajectories without imposing constraints.
Such constraint-free predictionsmay be incompliant with kinematic or environmental characteristics and thus give rise to massiveuncertainty in the predicted future states.
As a result, the downstream planning module would inevitably undergo some extra burdens, and even the “freezing robot problem” [11].
Furthermore, thetrajectory predictions typically generated by neural network regression have high dependences onlong-term tracking.
For some dense driving scenarios where the target would be momently occludedor suddenly appears within the sensing range, tracking results are discontinuous or not accumulatedenough.
The prediction accuracy would thereby degrade under such imperfect tracking cases.
[Figure]
Toward overcoming these challenges, we propose PRIME, a novel architecture for vehicle trajectory prediction, as illustrated in Fig.
The core idea is to exploit a model-based motion planner asthe prediction generator to produce feasibility-guaranteed future trajectories under explicit physicalconstraints, together with a deep neural network as the prediction evaluator to enable accurate multimodal prediction by learning complex implicit interactions.
To the best of our knowledge, PRIMEis the ﬁrst to incorporate an interpretable motion planner in prediction learning and also the onlymethod that ensures kinematic and environmental feasibility in data-driven trajectory prediction.
We conduct experiments on the large-scale Argoverse motion forecasting benchmark and achievesbetter prediction accuracy over the state-of-the-art.
Furthermore, PRIME shows signiﬁcant superiority in trajectory feasibility guarantee and prediction robustness under imperfect tracking.
Theseattributes would facilitate more ﬂexible and safe motion planning for SDVs.
Related Work
Prediction and Planning are closely intertwined in autonomous driving [12, 13, 14, 15].
Planningis to generate constraint-compliant trajectory candidates and, after considering safety, comfort, pathprogress, etc., select the best trajectory for execution by the SDV (ego agent).
Prediction facilitatesthe trajectory selection in planning by inferring future trajectories of the surrounding vehicles (targetagents).
Their different focuses make the corresponding mainstream solutions diverge.
Model-basedapproaches [16, 17, 18, 19] are preferred in planning due to their interpretability and reliability incomputing safe trajectories under explicit constraints.
Learning-based methods [6, 7, 20, 21], incontrast, prevail in prediction by utilizing its advantage in modeling implicit interactions.
Some learning-based prediction works incorporate the goal-directed idea from planning to inferthe possible goals and then produce goal-conditioned trajectories [22, 23, 24, 25].
Moreover, thenovel planning-prediction-coupled frameworks are introduced to make predictions conditioned onego intentions [26] or motion plans [27, 28].
Although much emphasis on improving the point-levelprediction accuracy, the data-driven frameworks cannot ensure the given constraints are indeed imposed on trajectory generation.
Despite DKM [29] embeds the two-axle vehicle kinematics [30] inthe output layer to ensure kinematic feasibility, yet still no guarantee on environmental compliance.
Inspired by the popular sampling-based paradigm in vehicle motion planning [13, 17], we employ amodel-based planner for providing feasibility-guaranteed trajectory sets, and thereby the learningbased part is reduced to evaluate future trajectories.
With making the most of model-based planningand learning-based prediction, PRIME handles complex agent-map interactions while fulﬁlling environmental and kinematic constraints.
Modeling agent-map interactions is fundamental for capturing information from scene map anddynamic agents.
The rasterized representation [20, 31, 32] is proposed for learning-based methods,which renders trafﬁc entities into images by different colors or intensities and then encodes rasterswith Convolutional Neural Networks.
As an alternative, the vectorized representation [33, 34, 25]vectorizes scene context as nodes to construct a graph, which exploits High Deﬁnition (HD) mapsmore explicitly and improves prediction accuracy.
By contrast, we address the agent-map modeling with a hierarchical structure that incorporates the lane-association ideas from [3] while extendsto learn global scene context.
To be speciﬁc, our prediction generator acts locally in a planningmanner to generate path-conditioned trajectory sets, and the prediction evaluator learns a globalunderstanding of the scene context by aggregating all trajectory and map features.
Generating multimodal trajectories is essential for handling the intrinsic multimodal prediction distributions.
Stochastic models are mostly built upon conditional variational autoencoder [7, 35, 36, 37, 38] and generative adversarial network [39, 40, 41, 42], while sampling with uncontrollable latent variables at inference may impede their deployment on safety-critical driving systems.
Deterministic models are mainly based on multi-mode trajectory regression [43, 44, 20, 34].
To alleviate mode collapse in prediction learning, recent works decompose the task into classiﬁcation over anchor trajectories [45] or goal-conditioned trajectories [25], followed by trajectory offsetregression.
However, no feasibility could be ensured for the regressed results.
CoverNet [32] formulates multimodal prediction by directly classifying on a pre-constructed trajectory set, but still,its predictions may violate the agent kinematics or scene constraints.
For our framework, leveragingmodel-based planning as the prediction generator brings superiorities in 1) maintaining multimodaldistributions by generating trajectories on diverse reachable paths, 2) ensuring trajectory feasibility by imposing real-time constraints, 3) mitigating the high reliance on long-term tracking, and 4)producing trajectories with continuous information rather than just discrete locations.
Overview
Problem formulation.
Assume the self-driving vehicle is equipped with detection and trackingmodules to provide observed states S of on-road agents A and has access to HD map M .
Let s tidenote the state of agent a i ∈ A at frame t , including position, heading, velocity, turning rate andactor type, and s i =
(cid:110)
s − T P
+1 i , s − T P
+2 i , ..., s 0 i (cid:111) denotes the state sequence in the observed period
T P .
Given any agent as the prediction target, we denote it by a tar and its surrounding agents by
A nbrs = { a 1 , a 2 , ..., a m } for differentiation, with their state sequence correspondingly given ass tar and S nbrs = { s 1 , s 2 , ..., s m } .
Accordingly, S = { s tar } ∪ S nbrs and A = { a tar } ∪ A nbrs .
Our objective is to predict multimodal future trajectories T tar =
{T k | k
= 1 , 2 , ..., K } together withcorresponding trajectory probability { p k } , where T k denotes a predicted trajectory for target agenta tar with continuous state information up to the prediction horizon T F , K is the number of predictedtrajectories.
Additionally, it is required to ensure each prediction T k ∈ T tar is feasible with existingconstraints C , which includes environmental constraints C M and the kinematic constraints C tar .
Our framework.
The two-stage architecture of PRIME consists of model-based generator G andlearning-based evaluator E .
Concretely, the generator G : ( s 0 tar , M , C ) (cid:55)→ ( P , T ) is tasked to produce the trajectory space for the target, which is approximated by a ﬁnite set of feasible trajectories
T. This part starts with searching a set of reachable paths P =
{P j | j = 1 , 2 , ..., l } from HD map M ,which provides reference path for trajectory generation.
Then a classical sampling-based planner isutilized to generate trajectory samples under constraints in C , and thus provide the feasible futuretrajectory set T =
(cid:83) l j =1 {T j,k | k
= 1 , 2 , ..., n j } for the target.
T j,k denotes the k -th feasible trajectory generated from path P j , and the total number of trajectories is n = (cid:80) lj
=1 n j .
The model-basedpart is specialized in generating trajectories with feasibility guaranteed but ignores multi-agent interactions.
The evaluator E : ( P , T , S ) (cid:55)→ ( T tar , { p k } ) takes charge of learning implicit interactions,which features with a dual representation for spatial information and with the attention mechanismto process the varying sizes of l reachable paths, m surrounding agents, and n feasible trajectories.
Notably, the evaluator E is reduced to score trajectories and select prediction results T tar ⊂ T ,rather than regressing position or displacement as most learning-based frameworks do.
Model-based Generator
4.1 Path Search
Unlike motion planning, where the reference path for the controllable ego agent is given, the futurepaths of uncontrollable targets in prediction are unknown.
Therefore, we conduct the path searchin advance of trajectory generation such that any prediction target could be associated with a set ofpotential paths P + .
Our path search algorithm G path : ( M , s 0 tar ) (cid:55)→ P + is implemented by Depth-
First-Search on HD map M , with more details described in the supplementary material.
Yielding apotential path P j ∈ P + with the centerline coordinates of each lane segment sequence, we expectall the paths of P + to provide sufﬁcient coverage for the future trajectory space of a tar .
As nodynamic constraint is imposed in this phase, for target with current state s 0 tar , some paths in P +may not be reachable at frame t = T F .
For instance, a high-speed vehicle cannot change to theopposite lane with a U-turn in few seconds.
Such unreachable paths could be recognized in thefollowing trajectory generation phase as no trajectories samples towards them are feasible.
Finally,a set of reachable paths P ⊆ P + would be reserved.
4.2 Trajectory Generation
path centerline trajectory
[Figure]
Given the potential paths in P + as dynamic references, wechoose to generate future trajectories in a planning manner.
For SDV, motion planning typically aims at ﬁnding an optimaltrajectory to connect the current state and a goal state, essentially different from prediction that infers probable trajectoriesfor vehicles with unknown intentions.
Despite this, the modelbased generator in planning, which computes a large numberof trajectory samples for the follow-up selection, could also beexploited in prediction.
We adopt the trajectory generation phase of Fren ´ et planner [17] in our trajectory generator G traj : ( P + , s 0 tar , C ) (cid:55)→
T .
Given a reference path in P + , a dynamic curvilinear frameis given by the tangential vector (cid:126)t r and normal vector (cid:126)n r at acertain point r on the path centerline.
The Cartesian coordinate (cid:126)x = ( x, y ) could be converted to the
Fren ´ et coordinate ( s, d ) , with the relation(cid:126)x (
[Equation]
in which (cid:126)r represents a vector pointing from the path root, s and d denote the covered arc lengthand the perpendicular offset, as illustrated in Fig.
The trajectory generation phase ﬁrst projectsthe current state s 0 tar onto the Fren ´ et frame and obtains the state tuple [ s 0 , ˙ s 0 , ¨ s 0 , d 0 , ˙ d 0 , ¨ d 0 ] .
Thelongitudinal movement s ( t ) and lateral movement d ( t ) within the prediction horizon T F are thengenerated independently by connecting the ﬁxed start state with different end states using parametric curves to cover different driving maneuvers.
Compared with planning, prediction receives lessaccurate state estimation for targets and does not need ﬁne-grained trajectories.
In our trajectory generator, therefore, some high-order state variables are simpliﬁed to zero.
For longitudinal movement,we sample the target velocity ˙
s
( T F ) ← ˙
s i in the range of [ max (0 , ˙ s 0 − δ − T F ) , min (ˆ˙ s, ˙ s
+ δ + T F )]while leaving s ( T F ) unconstrained.
The constants δ − , δ + and ˆ˙ s are given by considering the actortype of a tar and speed limit in M , to control the longitudinal velocity ˙
s i in a reasonable range.
Eachlongitudinal trajectory s i ( t ) is calculated using a quartic polynomials.t.
[ s (0) , ˙ s (0) , ¨ s (0) , ˙ s ( T F ) , ¨ s ( T F )] =
[ s 0 , ˙ s 0 , 0 , ˙
s i , 0] .
For lateral movement, we sample the target offset d ( T F ) ← d j in the range of [ − d lane / 2 , d lane / 2] ,where d lane denotes lane width.
Each lateral trajectory d j ( t ) is calculated using a quintic polynomials.t.
[ d (0) , ˙ d (0) , ¨ d (0) , d ( T F ) , ˙ d ( T F ) , ¨ d ( T F )] =
[ d 0 , ˙ d 0 , 0 , d j , 0 , 0] .
With the resulted longitudinal and lateral trajectory set T lon and T lat , a full trajectory (cid:126)x ( s ( t ) , d ( t ))is formed by every combinations in T lon × T lat .
Next, the trajectories incompliant with given constraints C would be ﬁltered out.
We ﬁrst project the Fren´et coordinates ( s, d ) back to global coordinates ( x, y ) to check if the trajectory collides with static obstacles given in C M .
For collision-freetrajectories, their high-order state variables are then converted by the Fren´et-Cartesian-transfomation
[Equation]
to check if any velocity v , acceleration α or curvature κ exceeds the kinematic constraints given in
C tar .
Finally, each reference path P j ∈ P would generate a set of n j feasibility-guaranteed futuretrajectories {T j,k | k
= 1 , 2 , ..., n j } , and all the trajectories together form an overall trajectory space
T .
Although the constraints are set conservatively with leaving some margin for the learning-basedevaluator, our model-based generator effectively narrows down the trajectory space T by imposingconstraints.
This unique advantage would set our framework with higher accuracy and robustness.
[Figure]
Afterwards, each future trajectory T j,k could query its track tensor X j ( s tar ) from P2T, interaction tensor Y j ( s tar ) from A2A and future tensor Z ( T j,k ) from F2F, and it is scored by feeding the concatenation of these tensors to fully-connected layers.
Finally, the evaluator ranks all feasible future trajectories in T by and set
Learning-based Evaluator
5.1 State Representation
The prediction evaluator aggregates scene context, including observed state sequences S , path set
P , and future trajectory set T .
To make it compatible with most existing trajectory predictiondatasets, state sequence s i is reduced to history track in the learning part.
Before feeding to thenetwork, we discretize each history track
s i and future trajectory T j,k as a location sequence withtime interval ∆ T , and each reference path P
j as a waypoint sequence with distance interval ∆ D .
Since the longitudinal movement s and lateral offset d indicate how an agent moves relative to areference path, they represent the local spatial relationship more straightforwardly.
For this reason,we use the Fren ´ et coordinates ( s, d ) in addition to the Cartesian coordinates ( x, y ) to form a dualspatial representation.
Here, the spatial information ( x, y, s, d ) of future trajectories in T is givenby the generator, while the ( s, d ) coordinates of history tracks in S are obtained by projecting ( x, y )coordinates on the corresponding reference path.
Additionally, we adopt the approach of [34] to adda binary mask b to history track’s representation ( x, y, s, d, b ) to indicate if the location is padded.
5.2 Encoding Scene Context
Prior to capture interrelationships between trafﬁc entities, we ﬁrst encode each kind of entity in thescene.
All encoders are structured with a temporal convolutional layer followed by a long short-termmemory (LSTM) layer.
The track encoder and the future encoder employ a unidirectional LSTMand make the last hidden state h ( · ) as the motion encoding for history track and future trajectory,while the path encoder uses a bidirectional LSTM and provides the sequence of hidden states H ( · )as the path spatial encoding.
Given the scene context description ( S , P , T ) , each reachable path
P j ∈ P is encoded as a H ( P j ) , where j = 1 , 2 , ..., l .
Considering that the Fren´et representationis dependent with the path frame, we encode all history tracks with respect to each reference path
P j , which results in l groups of track encodings { h ( s tar ) , h ( s 1 ) , ..., h ( s m ) } j .
Each future trajectory
T j,k ∈ T is relative to its reference path P j , so all future trajectories are encoded correspondinglyto form l groups of future encodings { h ( T j,k )
| k
= 1 , 2 , ..., n j } .
5.3 Modeling Interactions
Next is to capture the implicit interactions resulted from the static environment and multiple dynamicagents.
To fuse the spatial-temporal information from varying numbers of entities in the scene con- text, the attention mechanism [46] is adopted to construct four modules, namely, path to track (P2T),path to future (P2F), agent to agent (A2A), and future to future (F2F).
They are implemented in thesame way of scaled dot-product attention and use linear layers for mapping key, query and value.
The overall workﬂow is shown in Fig.
In the upper branch, P2T brings the spatial informationof each path encoding P j into the corresponding track encodings { h ( s tar ) , h ( s 1 ) , ..., h ( s m ) } j .
Thetrack encodings are further processed by a self-attention structure in A2A, aiming to capture the interactions between agents in the past time domain.
The lower branch lays emphasis on updating thefeatures contained in future encodings.
P2F brings the spatial information of path encoding H ( P j )into the corresponding future encodings { h ( T j,k )
| k
= 1 , 2 , ..., n j } .
It is followed by F2F that fusesall future encodings (cid:83) lj
=1 { h ( T j,k ) | k
= 1 , 2 , ..., n j } from different paths P j ( j = 1 , 2 , ..., l ) usingself-attention.
In particular, F2F obtains a global understanding of the reachable space given by Pand, by this way, attempts to further perceive the differences between different trajectories in T .
For any future trajectories T j,k ∈ T , the corresponding track tensor X j ( s tar ) , interaction tensor
Y j ( s tar ) and future tensor Z ( T j,k ) could be obtained from P2T, A2A and F2F modules, which arethen concatenated together to form a full description U j,k = Concat ( X j ( s tar ) , Y j ( s tar ) , Z ( T j,k )) .
5.4 Trajectory Scoring, Learning, and Inference
With U j,k as a full description, we score all the n trajectories T j,k using a maximum entropy model:
[Equation]
in which f ( · ) is implemented using a 3-layer MLP at the end of the evaluation network E .
The scorelabel ψ ( T j,k ) is resulted from calculating the accumulated squared distance error Dist ( · ) between thefuture trajectory T j,k and the ground truth trajectory T GT within the prediction horizon T F :
[Equation]
where τ is set as a temperature factor.
The overall network is trained by cross entropy between theevaluated scores and the labeled scores L =
CrossEntropy ( γ ( T j,k ) , ψ ( T j,k )) .
For the inference stagethat requires K predicted trajectories, we adopt the non-maximum suppression (NMS) algorithm toremove near-duplicate trajectories, as did in [25].
According to the predicted scores, this methodgreedily picks trajectories from T and excludes the lower scored trajectory between very closeones.
Finally, K trajectories with descending order of scores form the prediction result T tar =
{T i | k
= 1 , 2 , ..., K } , and the prediction probability p k is derived by p k
= γ ( T k ) / (cid:80)
Kk =1 γ ( T k ) .
Experiments
Dataset.
Argoverse [10] is one of the largest publicly available motion forecasting datasets, whichcontains over 324 K data sequence collected from complex urban driving scenarios.
The training,validation, and test sets are taken from disjoint parts of the cities.
Each sequence lasts for 5 seconds,containing the centroid locations of each tracked agent sampled at 10 Hz, in which one vehicle withrelatively complex motion is marked as the prediction target.
The objective is to predict its locations3 seconds into the future, given an initial 2-second observation.
Metrics.
We follow the Argoverse evaluation criteria under K = 1 and K = 6 .
Minimum Average Displacement Error (minADE K ) is the average L2 distance error of the best predicted trajectory.
Minimum Final Displacement Error (minFDE K ) is the L2 distance error of the best predictedtrajectory at the ﬁnal timestamp.
Miss Rate (MR K ) is the ratio of scenarios that none of K predicted trajectories has less than 2 meters L2 ﬁnal displacement error.
For multimodal prediction,the probability-based metrics p-minADE K and p-minFDE K are calculated by adding − log ( p ) tominADE
K and minFDE K , where p corresponds to the probability of the best predicted trajectory.
In the Argoverse benchmark, best refers to the predicted trajectory with the minimum endpoint error.
Implementation Details.
Our implementation is detailed in the supplementary material.
Among thestate-of-the-art methods, only LaneGCN [34] is open-source.
So we use its ofﬁcial implementationand Argoverse baselines [10] for additional tests about trajectory feasibility and imperfect tracking. [Table]
6.1 Comparison with State-of-the
We compare our proposed PRIME against the Argoverse baselines [10] (CV, LSTM+map, NN+map), the top-3 methods in the Argoverse Motion Forecasting Competition 2020 (Jean [21], Poly, Alibaba-ADLab), and the recently published state-of-the-art, LaneGCN [34] and TNT [25].
The performance comparison under Argoverse test set is shown in Table 1.
It could be noted that PRIME outperforms all other methods on Miss Rate ( K = 6 ), which is the ofﬁcial ranking metricin Argoverse Competition 2020.
It reﬂects that our method produces accurate multimodal predictions consistently in diverse scenarios.
We also achieve the best on the probability-based metric pminFDE 6 , which would be highly beneﬁcial to weigh between multiple predictions in making decisions and motion plans.
From the methods with public details, including LaneGCN [34], TNT [25],and Jean [21], we can ﬁnd they all perform the learning-based paradigm that utilizes neural networksto model trafﬁc entities and generates future trajectories, while PRIME is the only one that integratesa model-based motion generator into prediction learning.
Notably, due to the lack of more detailedon-road information in the Argoverse dataset, such as vehicle types, bounding box, static obstacles,etc., the quantitative result is achieved by imposing general constraints on the model-based generator.
This indicates there exists more space to improve when deploying our framework in a realautonomous driving system.
Furthermore, handling environmental and dynamic constraints in aninterpretable model-based manner and generating trajectories with continuous state information issigniﬁcant for real-world deployment, which could not be reﬂected from the evaluation metrics.
6.2 Ablation Studies
[Table]
We ablate the F2F module and Fren´etrepresentation (denoted by SD) from thecomplete evaluation network to studytheir impacts.
Table 2 reports the results on the Argoverse validation set.
With P2T, P2A, and A2A attention modules capturing the basic interactions between map and agents, the base modelperforms at the same level with TNT(MR 6 = 9% reported in [25]), indicating that these modules are effective in capturing agent-map interactions.
As for the Fren ´ et representation providing the local spatial relationship and the F2F module fusing all feasible trajectories toget a global understanding of the reachable space, they both promote the performance.
By comparison, the inclusion of Fren´et representation is more effective.
Additionally, the complete networkmakes the best performance with only 1 .
02 M parameters, which indicates that separating the function of trajectory generation would reduce the learning burden while achieving high performance.
6.3 Trajectory Feasibility
As a typical non-holonomic motion system, vehicles are constrained by inherent kinematic characteristics.
So we investigate the ratio of infeasible trajectories produced by prediction models.
Sincethe high-order states (velocity, acceleration, or turning rate) cannot be estimated accurately fromdiscrete locations predicted by common learning-based models, we evaluate the trajectory feasibility only using curvature.
By interpolating the predicted positions with pairwise cubic splines, weget the curvature at each point.
A trajectory is labeled as infeasible if the curvature κ > 1 / 3 (i.e.,the minimum turning radius is 3 meters) at any of its points.
The ratio of infeasible trajectories isshown in the last column of Table 1.
Except for the physical baseline Argo-CV (Constant Velocity), the others, as representatives of the unconstrained learning-based models, have at least 16 .
5%infeasible predictions.
Although we only use curvature for judgment and set a fairly conservativethreshold (the minimal turning radius for a regular sedan is around 4 .
5 ∼ 6 . 0 meters), the infeasiblepredictions still take up a considerable proportion, which would cause redundant burdens for SDVsto make decisions and plans.
By contrast, the model-based generator in our framework can handleany kinematic and environmental constraints, thereby ensuring trajectory feasibility.
6.4 Imperfect Tracking
[Figure] While most motion forecasting datasetsprovide tracking results of a certain duration for prediction targets, a self-drivingvehicle would inevitably encounter realworld situations where the target is lostin some timestamps or not tracked longenough yet.
Then the prediction modelis required to robustly handle imperfecttracks rather than being restricted to ﬁxedduration tracking inputs.
To let the models (ours, LaneGCN, and NN+map baseline) be aware of imperfect tracks, we retrained them by randomly dropping outtracked locations.
For processing such inputs while keeping network structures, wepad the locations of dropped timestampswith the nearest tracked location and adda dimension of the binary mask to denotethe padded location.
The drop rate is randomly sampled from 0 ∼ 0 .
6 for each datasequence in training but ﬁxed in testing.
The drop rate is pointwise applied, i.e., 0.6 drop rate maydrop more or less than 60% of locations on a track.
The last timestamp is always kept to ensurethe prediction target could be detected at inference.
Fig.
4 shows how the miss rate varies withtrack drop rate, we observe that our model performs stably, with only 3 . 6% relative increase on
MR 6 , while the relative increase is around 30% ∼ 40% for the others.
The result indicates that thelearning-based prediction models rely on long-term tracked results to regress trajectories, while ourframework design relieves that to a certain extent, thereby improving the prediction robustness.
Conclusion
We present the prediction framework PRIME that learns to predict vehicle trajectories with modelbased planning.
PRIME guarantees the trajectory feasibility by exploiting a model-based generatorto produce future trajectories under explicit constraints.
It makes accurate trajectory predictions byemploying a learning-based evaluator to capture implicit interactions in scene context and selectfuture trajectories by scoring.
With the novel framework design, PRIME outperforms the state-ofthe-art in prediction accuracy, feasibility, and robustness.
Moreover, the advantages of reasonablyregularizing trajectory space, predicting trajectories with continuous state, and the compatibilitywith on-road information would set our framework highly useful in real system deployment.