Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients
Anonymous Author(s)
AfﬁliationAddress email
Abstract:
In this work we are the ﬁrst to present an ofﬂine policy gradient methodfor learning imitative policies for complex urban driving from a large corpus ofreal-world demonstrations.
This is achieved by building a differentiable datadriven simulator on top of perception outputs and high-ﬁdelity HD maps of thearea.
It allows us to synthesize new driving experiences from existing demonstrations using mid-level representations.
Using this simulator we then train apolicy network in closed-loop employing policy gradients.
We train our proposedmethod on 100 hours of expert demonstrations on urban roads and show that itlearns complex driving policies that generalize well and can perform a varietyof driving maneuvers.
We demonstrate this in simulation as well as deploy ourmodel to self-driving vehicles in the real-world.
Our method outperforms previously demonstrated state-of-the-art for urban driving scenarios – all this withoutthe need for complex state perturbations or collecting additional on-policy dataduring training.
We make code and data publicly available.
[Figure]
Introduction
Self-driving has the potential to revolutionize transportation and is a major ﬁeld of AI applications.
Even though already in 1990 there were prototypes capable of driving on highways [1], technologyis still not widespread, especially in the context of urban driving.
In the past decade, the availabilityof large datasets and high-capacity neural networks has enabled signiﬁcant progress in perception[2, 3] and the vehicles’ ability to understand their surrounding environment.
Self-driving decisionmaking, however, has seen very little beneﬁt from machine learning or large datasets.
State-of- the-art planning systems used in industry [4] still heavily rely on trajectory optimisation techniques cost functions.
These cost functions capture desirable properties of the future vehicle path.
However, engineering these cost functions scales poorly with the complexity of driving situations and the long tail of rare events
Due to this, learning a driving policy directly from expert demonstrations is appealing, since performance scales to new domains by adding data rather than via additional human engineering effort.
In this paper we focus speciﬁcally on learning rich driving policies for urban driving from largeamounts of real-world collected data.
Unlike highway driving [5], urban driving requires performing a variety of maneuvers and interactions with, e.g., trafﬁc lights, other cars and pedestrians.
Recently, rich mid-level representations powered by large-scale datasets [6, 7], HD-maps and highperformance perception systems enabled capturing nuances of urban driving.
This led to new methods achieving high performance for motion prediction [8, 9].
Furthermore, [10] demonstrated thatleveraging these representations and behavioral cloning with state perturbations leads to learningrobust driving policies.
While promising, difﬁculty of this approach lies in engineering the perturbation noise mechanism required to avoid covariate shift between training and testing distribution.
Inspired by this approach, we present the ﬁrst results on ofﬂine learning of imitating driving policiesusing mid-level representations, a closed-loop simulator and a policy gradient method.
This formulation has several beneﬁts: it can successfully learn high-complexity maneuvers without the need forperturbations, implicitly avoid the problem of covariate shift, and directly optimize imitation as wellas auxiliary costs.
The proposed simulator is constructed directly from the collected logs of realworld demonstrations and HD maps of the area, and can synthesize new realistic driving episodesfrom past experiences (see Figure 1 for an overview of our method).
Furthermore, for training onlarge datasets reducing the computational complexity is paramount.
We leverage vectorized representations and show how this allows for computing policy gradients quickly using backpropagationthrough time.
We demonstrate how these choices lead to superior performance of our method overthe existing state-of-the-art in imitation learning for real-world self-driving planning in urban areas.
Our contributions are four-fold:
• The ﬁrst demonstration of policy gradient learning of imitative driving policies for complex
urban driving from a large corpus of real-world demonstrations.
We leverage a closed-
loop simulator and rich, mid-level vectorized representations to learn policies capable of
performing a variety of maneuvers.
• A new differentiable simulator that enables efﬁcient closed-loop simulation of realistic
driving experiences based on past demonstrations, and quickly compute policy gradients
by backpropagation through time, allowing fast learning.
• A comprehensive qualitative and quantitative evaluation of the method and its performance
compared to existing state-of-the-art.
We show that our approach, trained purely in simula-
tion can control a real-world self-driving vehicle, outperforms other methods, generalizes
well and can effectively optimize both imitation and auxiliary costs.
• Source code and data are made available to the public.
Related Work
In this section we summarize different approaches for solving self-driving vehicle (SDV) decisionmaking in both academia and industry.
In particular, we focus on both optimisation-based and
ML-based systems.
Furthermore, we discuss the role of representations and datasets as enablers inrecent years, to tackle progressively more complex Autonomous Driving (AD) scenarios.
Trajectory-based optimization is still a dominant approach used in industry for both highway andurban-driving scenarios.
It relies on manually deﬁned costs and reward functions that describe gooddriving behavior.
Such cost can then be optimized using a set of classical optimization techniques(A* [11], RRTs [12], POMDP with solver [13], or dynamic programming [14]).
Appealing properties of these approaches are their interpretability and functional guarantees, which are importantfor AD safety.
These methods, however, are very difﬁcult to scale.
They rely on human engineering rather than on data to specify functionality.
This becomes especially apparent when tacklingcomplex urban driving scenarios, which we address in this work.
Reinforcement learning (RL) [15] removes some complexity of human engineering by providinga reward (cost) signal and uses ML to learn an optimal policy to maximize it.
Directly providingthe reward through real-time disengagements [16], however, is impractical due to a low sampleefﬁciency of RL and the involved risks.
Therefore, most approaches [17] rely on constructing a simulator and explicitly encoding and optimising a reward signal [18].
A limiting factor of theseapproaches is that the simulator often is hand-engineered [19, 20], limiting its ability to capturelong-tail real-world scenarios.
Recent examples of sim-to-real policy transfer (e.g. [21], [22], [23])were not focused on evaluating scenarios typical to urban driving, in particular interacting withother agents.
In our work, we construct the simulator directly from real-world logs through midlevel representations.
This allows training in a variety of real-world scenarios with other agentspresent, while employing efﬁcient policy-gradient learning.
Imitation learning (IL) and Inverse Reinforcement Learning (IRL) [24, 25] are more scalable
ML approaches that leverage expert demonstrations.
Instead of learning from negative events, aimis to directly copy expert behavior or recover the underlying expert costs.
Simple behavioral cloningwas applied already back in 1989 [26] on rural roads, more recently by [27] on highways and [28]in urban driving.
Naive behavioral cloning, however, suffers from covariate shift [24].
This issuehas been successfully tackled for highway lane-following scenarios by reframing the problem asclassiﬁcation task [29] or employing a simple simulator [5], constructed from highway cameras.
Wetake inspiration from these approaches but focus on the signiﬁcantly more complex task of urbandriving.
Theoretically, our work is motivated by [30], as we employ a similar principle of generatingsynthetic corrections to simulate querying an expert.
Due to this, identical proven guarantees holdfor our method, namely the ideal linear regret bound, mitigating the problem of covariate shift.
Adversarial Imitation Learning comprises another important ﬁeld [31, 32, 33], but, to the best of ourknowledge, has seen little application to autonomous driving and no actual SDV deployment yet.
Neural Motion Planners are another approach used for autonomous driving.
In [34] raw sensoryinput and HD-maps are used to estimate cost volumes of the goodness of possible future SDVpositions.
Based on these cost volumes, trajectories can be sampled and the lowest-cost one isselected to be executed.
This was further improved in [35], where the dependency on HD-maps wasdropped.
To the best of our knowledge, these promising methods have not yet been demonstrated todrive a car in the real-world though.
Mid-representations and the availability of large-scale real-world AD datasets [6, 7] have beenmajor enablers in recent years for tackling complex urban scenarios.
Instead of learning policiesdirectly from sensor data, the input of the model comprises the output of the perception system aswell as an HD map of the area.
This representation compactly captures the nuance of urban scenariosand allows large-scale training on hundreds or thousands of hours of real driving situations.
Thisled to new state-of-the-art solutions for motion forecasting [8, 9].
Moreover, [10] demonstrated thatusing mid-representations, large-scale datasets and simple behavioral cloning with perturbations[36] can scale and learn robust planning policies.
The difﬁculty of this approach, however, is inengineering the noise model to overcome the covariate shift problem.
In our work we are inspired bythis approach, but attempt to learn robust policies using policy gradient optimisation [37] featuringunrolling and evaluating the policy during training.
This implicitly avoids the problem of covariateshift and leads to superior results.
This approach is, however, more computationally expensive andrequires a simulator.
To solve this, we show how a fast and powerful simulator can be constructeddirectly from real-world logs enabling scalability of this approach.
Data-driven simulation. A realistic simulator is useful for both training and validation of MLmodels. However, many current simulators (e.g. [19, 38]) depend on heuristics for vehicle controland do not capture the diversity of real world behaviours. Data-driven simulators are designedto alleviate this problem.
[23] created a photo-realistic simulator for training an end-to-end RLpolicy.
[5] simulated a bird’s-eye view of dense trafﬁc on a highway.
Finally, two recent works[39, 40] developed data-driven simulators and showed their usefulness for training and validating
ML planners.
In this work we show that a simpler, differentiable simulator based on replaying logsis effective for training.
Differentiable Trafﬁc Simulator from Real-world Driving Data
In this section we describe a differentiable simulator S that approximates new driving experiencesbased on an experience ¯ τ collected in the real world.
This simulator is used during policy learningfor the closed-loop evaluation of the current policy’s performance and computing the policy gradient.
As shown in Section 5, differentiability is an important building block for achieving good results,especially when employing auxiliary costs. [Figure]
We represent the real-world experience ¯ τ as a sequence of state observations ¯ s t around the vehicleover time:
[Equation]
We use a vectorized representation based on [8], in which each state observation ¯ s t consists of acollection of static and dynamic elements e 1 t , e 2 t , ..., e Kt around the vehicle pose ¯ p t ∈ SE 2 , with
SE 2 denoting the special Euclidean group.
Static elements include trafﬁc lanes, stop signs andpedestrian crossings.
These elements are extracted from the underlying HD semantic map of thearea using the localisation system.
The dynamic elements include trafﬁc lights status and trafﬁcparticipants (other cars, buses, pedestrians and cyclists).
These are detected in real-time using theon-board perception system.
Each element e jt includes a pose q jt ∈ SE 2 relative to the SDV posep t , as well as additional features, such as the element type, time of observation, and other optionalattributes, e.g. the color of associated trafﬁc lights, recent history of moving vehicles, etc.
The fulldetails of this representation are provided in Appendix C.
Goal of the simulation is to iteratively generate a sequence of state observations τ ={ s 1 , s 2 , . . .
, s T } that corresponds to a different sequence of driver actions a 1 , a 2 , ..., a N in the scenario.
This is done by ﬁrst computing the corresponding SDV trajectory p 1 , p 2 , ..., p N and thenlocally transforming states ¯ s 1 , ¯ s 2 , . .
, ¯ s N .
Updated poses of the SDV are determined by a kinematic model p t +1 =
f ( p t , a t ) , which is assumedto be differentiable.
The state observation s t is then obtained by computing the new position q jt foreach state element e jt using a transformation along the differences of the original and updated pose:
[Equation]
See Figure 2 for an illustrative example.
It is worth noting that this approximation is effective if thedistance between the original and generated SDV pose is not too large.
We denote performing these steps in sequence with the step-by-step simulation transition functions t +1 =
S ( s t , a t ) .
Moreover, since both Equation (2) and vehicle dynamics f are fully differentiable,we can compute gradients with respect to both the state ( S s ) and action ( S a ).
This is critical for theefﬁcient computation of policy gradients using backpropagation through time as described in thenext section.
Imitation Learning Using a Differentiable Simulator
In this part, we detail how we use the simulator S described in the previous section to learn adeterministic policy π to drive a car using closed-loop policy learning.
We frame the imitation learning problem as minimisation of the L1 pose distance L ( s t , a t ) =(cid:107) ¯ p t − p t (cid:107) 1 between the expert and learner on a sequence of collected real-world demonstrations¯ τ 1 , ¯ τ 2 , ..., ¯ τ N ∼ π E .
Note that with a slight abuse of notation we use the poses ¯ p t , p t here to referto 3D vectors ( x, y, θ ) , instead of roto-translation matrices in SE 2 – yielding the common L1 norm [Figure] and loss.
This can be expressed as a discounted cumulative expected loss [41] on the set of collectedexpert scenarios: [Figure]
[Equation]
Optimising this objective pushes the trajectory taken by the learned policy as close as possible to theone of the expert, as well as limiting the trajectory to the region where the approximation given bythe simulator is effective.
In Appendix B we further extend this to include auxiliary cost functionswith the aim of optimising additional objectives.
We can use any policy optimisation method [42, 43] to optimize Equation (3).
However, given thatthe transition S ( s t , a t ) is differentiable, we can exploit it for a more effective training that doesnot require a separate estimation of a value function.
As shown in [31, 37, 44], this results into anorder of magnitude more efﬁcient training.
The optimisation process consists of repeatedly samplingpairs of expert and policy trajectories ¯ τ i , τ i and computing the policy gradient J θ for these samplesto minimize Equation (3).
We describe both steps in detail in the following subsections.
4.1 Sampling from a Policy Distribution π
Input: Expert policy samples ¯ τ 1 , ¯ τ 2 , ..., ¯ τ N ∼ π E
Output: Learned policy π π = random ; for ¯ τ ∼ π E do for t = 1 to T do a t
= π ( s t ) ; s t +1 = S
( s t , a t ) ; end
J T +1 s = 0 ;
J T +1 θ = 0 ; for t =
T downto K do
J ts =
L s + L a π s + γJ t +1 θ ( S s + S a π s ) ;
J tθ =
L a π θ + γ
( J t +1 s
S a π θ + J t +1 θ ) ; end π = gradient_update ( π, J Kθ ) ; end
In this subsection we detail sampling pairs ofexpert ( ¯ τ ) and corresponding policy trajectory( τ ) drawn from policy π . Sampling expert trajectories ¯ τ consists of simply sampling from the collected dataset of expert demonstrations. To generate the policysample τ we acquire an expert state ¯ s 1 ∈ ¯ τ ,and then unroll the current policy π for T stepsusing the simulator S . This naive method, however, introduces bias,as the initial state of the trajectory is alwaysdrawn from the expert π E and not from thepolicy distribution π .
As shown in Appendix
B, this results in the under-performance of themethod. To remove this bias we discard the ﬁrst K timesteps from both trajectories and use onlythe remaining T − K timesteps to estimate thepolicy gradient J θ as described next (see Figure3 for a visualization). 4.2 Computing Policy Gradient J θ
Here we describe the computation of the policy gradient J θ around the rollout trajectory τ =s 1 , a 1 , s 2 , a 2 , . . .
, s T , a T given by the current policy.
This gradient can be computed for deterministic policies π using backpropagation through time leveraging the differentiability of the simulator
S .
Note that we denote partial differentiation with subscripts, i.e. g x (cid:44) ∂g ( x, . . . ) /∂
( x ) .
We followthe formulation in [37] and express the gradient by a pair of recursive formulas:
[Equation]
The resulting algorithm is outlined in Algorithm 1 and illustrated in Figure 3.
It can be implementedsimply as one forward pass of length T and one backward pass of length T − K .
To compute thepolicy gradient we use equations (4) and (5) recursively from t =
T to t =
K and use it to updatepolicy parameters θ .
Experiments
In this section we evaluate our proposed method and benchmark it against existing state-of-the-artsystems.
In particular, we are interested in: its ability to learn robust policies dealing with varioussituations observed in the real world; its ability to tailor performance using auxiliary costs; thesensitivity of key hyper-parameters; and the impact on performance with increasing amounts oftraining data.
Additional results can be found in the appendix and the accompanied video.
5.1 Dataset
For training and testing our models we use the Lyft Motion Prediction Dataset [6].
This datasetwas recorded by a ﬂeet of Autonomous Vehicles and contains samples of real-world driving on acomplex, urban route in Palo Alto, California.
The dataset captures various real-world situations,such as driving in multi-lane trafﬁc, taking turns, interactions with vehicles at intersections, etc.
Datawas preprocessed by a perception system, yielding the precise position of nearby vehicles, cyclistsand pedestrians over time.
In addition, a high-deﬁnition map provides locations of lane markings,crosswalks and trafﬁc lights.
All models are trained on a 100h subset, and tested on 25h.
Thetraining dataset is identical to the publicly available one, whereas for the sake of execution speed fortesting we use a random, but ﬁxed, subset of the listed test dataset, which is roughly 14 in size.
5.2 Baselines
We compare our proposed algorithm against three state-of-the-art baselines:
• Naive Behavioral Cloning ( BC ): we implement standard behavioral cloning using our vectorized backbone architecture.
We do not use the SDV’s history as an input to the model to
avoid causal confusion (compare [10]).
• Behavioral Cloning + Perturbations ( BC-perturb ): we re-implement a vectorized versionof ChauffeurNet [10] using our backbone network.
As in the original paper, we add noisein the form of perturbations during training, but do not employ any auxiliary losses.
We
test two versions: without the SDV’s history, and using the SDV’s history equipped withhistory dropout.
• Multi-step Prediction ( MS Prediction ): we apply the meta-learning framework proposedin [30] to train our vectorized network.
We observe that a version of this algorithm can
conveniently be expressed within our framework; we obtain it by explicitly detaching gradients between steps (i
e. ignoring the full differentiability of our simulation environment).
Differently from
the original work [30],
we do not save past unrolls
as
new dataset samples
over time.
5.3 Implementation
Inspired by [8, 45], we use a graph neural network for parametrizing our policy.
It combines a
PointNet-like architecture for local inputs processing followed by an attention mechanism for global
[Table]
reasoning.
In contrast to [8], we use points instead of vectors.
Given the set of points corresponding to each input element, we employ 3 PointNet layers to calculate a 128-dimensional featuredescriptor.
Subsequently, a single layer of scaled dot-product attention performs global feature aggregation, yielding the predicted trajectory.
We found K = 20 and T = 32 to work well, i.e. weuse 20 timesteps for the initial sampling and effectively predict 12 trajectory steps.
γ is set to 0.8.
In total, our model contains around 3.5 million trainable parameters, and training takes 30h on
Tesla V100 GPUs.
For more details we refer to Appendix C.
For the vehicle kinematics model f we use an unconstrained model p t +1 =
p t + a t with a t ∈ SE 2 .
This allows for a fair comparisons with the baselines as both BC-perturb and MS Prediction assumethe possibility of arbitrary pose corrections.
Other kinematics models, such as unicycle or bicyclemodels, could be used with our method as well.
All baseline methods share the same network backbone as ours, with model speciﬁc differences asdescribed above – and BC and BC-perturb predicting a full T-step trajectory with a single forward,while MS Prediction and ours are calling the model T times.
To ensure a fair comparison, also for
MS Prediction we use our proposed sampling procedure, i.e. use the ﬁrst K steps for sampling only.
We train all models for 61 epochs with a learning rate of 10 − 4 , and drop it to 10 − 5 after 54 epochs.
We note that we achieve best results for our proposed method by disabling dropout, and hypothesizethis is related to similar issues observed for RNNs [46].
We refer the reader to Appendix B for ablations on the inﬂuence on data and sampling.
5.4 Metrics
We implement the metrics describe below to evaluate the planning performance.
These capturekey imitation performance, safety and comfort.
In particular, we report the following, which arenormalized – if applicable – per 1000 miles driven by the respective planner:
• L2 : L2 distance to the underlying expert position in the driving log in meters.
• Off-road events : we report a failure if the planner deviates more than 2m laterally fromthe reference trajectory – this captures events such as running off-road and into opposingtrafﬁc.
• Collisions : collisions of the SDV with any other agent, broken down into front, side andrear collisions w.r.t.
the SDV. • Comfort : we monitor the absolute value of acceleration, and raise a failure should thisexceed 3 m/s 2 .
• I1K : we accumulate safety-critical failures (collisions and off-road events) into one keymetric for ease of comparison, namely Interventions per 1000 Miles (I1K).
5.5 Results for Pure Imitation
We evaluate our method and all the baselines by unrolling the policy on 3600 sequences of 25seconds length from the test set and measure the above metrics.
Table 1 reports performance when all methods are trained to optimize the imitation loss alone.
Behavioral cloning yields a high number of trajectory errors and collisions.
This is expected, as thisapproach is known to suffer from the issue of covariate shift [24].
Including perturbation duringtraining dramatically improves performance as it forces the method to learn how to recover from drifting.
We further observe that MS Prediction yields comparable results for many categories,while yielding less rear collisions.
We attribute this to the further reduction of covariate shift whencompared to the previous methods: the training distribution is generated on-policy instead of beingsynthesized by adding noise.
Finally, our method yields best results overall.
It is worth notingthat all models share a high number of comfort failures, due to the fact that they are all trained forimitation performance alone, which does not optimize for comfort, but only positional accuracy ofthe driven vehicle – which we address in the appendix.
5.6 In-car Testing
In addition to above stated simulation results, we further deployed our planner on SDVs in the real.
For this, a Ford Fusion equipped with 7 camera, 3 LiDAR and 11 Radar sensors was employed.
The sensor setup thus equals the one used for data collection, and during road-testing our perceptionand data-processing stack is run in real-time to generate the desired scene representation on theﬂy.
For this, vehicles are equipped with 8 Nvidia 2080 TIs.
Experiments were conducted on aprivate test track, including other trafﬁc participants and reproducing challenging driving scenarios.
Furthermore, this track was never shown to the network before, and thus offers valuable insights intogeneralization performance.
Figure 5 shows our model successfully crossing a signaled intersection,for more results we refer to the appendix and our supplementary video.
Conclusion
In this work we have introduced a method for learning an autonomous driving policy in an urbansetting, using closed-loop training, mid-level representations with a data-driven simulator and a largecorpus of real world demonstrations.
We show this yields good generalization and performance forcomplex, urban driving.
In particular, it can control a real-world self-driving vehicle, yielding betterdriving performance than other state-of-the-art ML methods.
We believe this approach can be further extended towards production-grade real-world driving requirements of L4 and L5 systems – in particular, for improving performance in novel or rarely seenscenarios and to increase sample efﬁciency, allowing further scaling to millions of hours of driving.