DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles
Abstract: Autonomous vehicle (AV) stacks are typically built in a modular fashion, with explicit components performing detection, tracking, prediction, planning, control, etc.
While modularity improves reusability, interpretability, and generalizability, it also suffers from compounding errors, information bottlenecks, and integration challenges.
To overcome these challenges, a prominent approach is to convert the AV stack into an end-to-end neural network and train it with data.
While such approaches have achieved impressive results, they typically lack interpretability and reusability, and they eschew principled analytical components, such as planning and control, in favor of deep neural networks.
To enable the joint optimization of AV stacks while retaining modularity, we present DiffStack, a differentiable and modular stack for prediction, planning, and control.
Crucially, our model-based planning and control algorithms leverage recent advancements in differentiable optimization to produce gradients, enabling optimization of upstream components, such as prediction, via backpropagation through planning and control.
Our results on the nuScenes dataset indicate that end-to-end training with DiffStack yields substantial improvements in open-loop and closed-loop planning metrics by, e.g., learning to make fewer prediction errors that would affect planning.
Beyond these immediate beneﬁts, DiffStack opens up new opportunities for fully data-driven yet modular and interpretable AV architectures. Project website: https://sites.google.com/view/diffstack
Introduction
Intelligent robotic systems, such as autonomous vehicles (AVs), are typically architected in a modular fashion and comprised of modules performing detection, tracking, prediction, planning, and control, among others [1, 2, 3, 4, 5, 6, 7, 8].
Modular architectures are generally desirable because of their veriﬁability, interpretability and generalization performance; however, they also suffer from compounding errors, information bottlenecks, and integration challenges.
A promising line of work tackling these issues focuses on making AV stacks more integrated (by relaxing inter-module interfaces) and data-driven (by optimizing modules jointly with respect to their downstream task).
For example, in the context of AV perception, recent work has achieved substantial performance gains by jointly training tracking models with detection [9] and prediction models [10, 11].
To extend such a joint, data-driven approach to decision making, existing approaches replace hand-engineered components, e.g., planning and control algorithms, with deep neural networks [12, 13, 14].
As neural networks are differentiable, they can be optimized end-to-end for a ﬁnal control objective; however, they offer weaker generalization, little to no interpretability or safety guarantees.
We introduce DiffStack , a differentiable AV stack with modules for prediction, planning, and control that combines the beneﬁts of modular and data-driven architectures (Fig. 1).
The prediction module in DiffStack is a learned neural network that predicts the future motion of agents; the planning and control modules are principled, hand-engineered algorithms that produce AV actions given the current world state and motion predictions.
Importantly, our hand-engineered planning and control algorithms are differentiable , enabling the training of the upstream prediction module for a downstream control objective by backpropagating gradients through the algorithms.
In doing so, DiffStack can jointly [Figure]
P r e d i c t i o n
P l a nn i n g
C o n t r o l
P e r c e p t i o n
P r e d i c t i o n
P l a nn i n g C o n t r o l
P e r c e p t i o n optimize the entire stack for the control
, as , however it also maintains the interpretability and formal guarantees of standard modular AV architectures.
To make the planner and controller differentiable, we build upon the growing body of work on differentiable algorithm networks [15].
In particular, we leverage the differentiable Model Predictive
Control (MPC) algorithm [16] for our controller.
While differentiable algorithms show great promise, most prior works consider relatively simple control problems, e.g., pendulum and cartpole in [16].
To the best of our knowledge, this is the ﬁrst work to demonstrate that, through careful design choices and integration, differentiable algorithms can be composed into stacks and trained with real-world data for AV prediction, planning, and control.
We evaluate DiffStack in both open-loop and closed-loop simulation settings using the large-scale, real-world nuScenes dataset [17].
Our results show some immediate beneﬁts of differentiable stacks: by training a prediction model with respect to the ﬁnal control objective, DiffStack increases the effectiveness of predictions for decision making by up to 15 % over a large number of diverse scenarios.
DiffStack achieves this by, e.g., learning to make fewer prediction errors that would negatively affect planning.
Further, DiffStack outperforms alternative handcrafted planning-aware prediction losses; compensates for artiﬁcially introduced integration errors; and tunes an interpretable cost function to make resulting plans more similar to human driving.
Beyond these immediate beneﬁts, our work demonstrates the feasibility and potential of differentiable AV decision making stacks, making an important step towards a new class of data-driven yet modular AV architectures.
Related work
Modular AV architectures.
Many state-of-the-art methods for AV perception have achieved substantial gains by relaxing bottlenecks between learned neural network modules and training multiple modules together.
Examples include works that more tightly integrate object detection and tracking [9], as well as prediction with object detection [11], classiﬁcation [18], and tracking [19, 20, 10].
While successful, these works focus on AV modules where neural networks are already commonly employed and thus differentiable, i.e., perception and prediction.
Looking downstream, there are many works that focus on more tightly integrating prediction within planning [21, 22, 23, 24, 25], however, these works only use predictions in planning rather than improve the prediction module with information from planning.
Recently, works such as [26, 27, 28] present potential avenues for designing upstream modules in a planning-aware fashion.
In particular, Ivanovic and Pavone [27] propose an evaluation metric for motion prediction that considers the impact of predictions on downstream planning.
While this metric can be used to train prediction with information from planning, we will show in Section 4 that the resulting improvement is less than with DiffStack.
Data-driven AV architectures.
Another line of work eschews modularity entirely, and learns a policy network directly from data [12, 29, 30].
The approach beneﬁts from the removal of information bottlenecks and performance scaling with increasing dataset sizes.
However, fully end-to-end approaches lack interpretability, important for debugging, veriﬁcation, and safety guarantees; it is also unclear how well purely data-driven methods can generalize to unseen or rare-but-critical scenarios.
Accordingly, recent works have focused on reintroducing structure and interpretability to learned policy networks. Most notably, the Neural Motion Planner [13] and follow-up methods [31, 32, 14] use object detection and agent behavior prediction as interpretable intermediate auxiliary objectives while training a joint backbone network from sensors directly to decision making. An important
[Table]
[Table]
u
[Table]
<latexit sha1_base64="RNBHhwt1Y9dVimVZ51WUVInlZ9Q
[Table]
[Table] [Table]
[Figure]
Importantly, this means that gradients can propagate backwards all the way from the ﬁnal planning objective, allowing upstream predictions to be optimized with respect to downstream decision making.
downside of this approach is that all components are neural networks, preventing the incorporation of principled planning and control algorithms, which are otherwise common in practice.
Differentiable algorithms.
The idea of making existing model-based algorithms differentiable opens up new design paradigms that blend data-driven deep learning with classic model-based algorithms [15].
When both neural network models and ﬁxed algorithmic computation can be expressed in the same “neural programming" framework, one can perform joint end-to-end learning even in structured modular architectures.
Differentiable algorithms have been explored in various robotic domains including localization [33, 34], mapping [35, 36], navigation [37, 38, 15], and robot control [39, 16, 40, 41].
Our work is motivated by the initial successes of these approaches in rather simple robotic domains.
To the best of our knowledge, this is the ﬁrst work to apply differentiable algorithms in the context of autonomous driving stacks including prediction, planning, and control.
DiffStack: A Differentiable and Modular Autonomous Vehicle Stack
DiffStack is an autonomous driving stack with modules for prediction, planning, and control (Fig. 2).
At a high level, DiffStack produces a trajectory for the AV to follow given a goal state, the tracked states of other agents, and a lane graph.
To do so, it predicts where non-AV agents could move in the future; uses the predictions to select a safe, low-cost, and dynamically-feasible motion plan from a set of candidates; and ﬁnally optimizes the plan to yield a continuous and smooth ego-trajectory.
The key feature of DiffStack is that it is differentiable , i.e., we can compute @ L j / @✓ i , the gradient of the training objective L j of downstream module j with respect to the parameters ✓ i of upstream module i .
Differentiability enables training all parameters jointly for the overall objective of the stack with a gradient-based optimizer.
We can also interpret DiffStack as a neural network with interpretable internal representations and ﬁxed computational blocks for planning and control.
Section 3.1 will describe how these components are made differentiable.
The main use case considered in this paper is to train the prediction module with respect to a ﬁnal control objective.
Different agents’ predictions have different importance to the ego-vehicle (see examples in Appendix B).
Standard metrics used to train prediction models are agnostic to the consequences of downstream planning, treating harmless and dangerous errors equally [26, 27].
By training directly for the end-objective of ego control, we expect DiffStack’s prediction module to not only be accurate, but also synergize with planning and yield better overall ego controls.
We consider two open-loop training settings.
In reinforcement learning (RL) experiments we optimize a hindsight cost that measures the quality of an ego trajectory in hindsight , i.e., after observing the future trajectory of other agents in the scene.
The hindsight cost can be viewed as a negative reward for
RL.
In imitation learning (IL) experiments we compare the planned ego trajectory to the ego-vehicle’s enacted motion in the data through a mean-squared-error (MSE) loss.
We additionally explore using
DiffStack to tune the hand-speciﬁed planning and control cost functions.
In this section, we describe each module in DiffStack and their respective training setups, with more details in Appendix C.
Nomenclature.
Ego refers to the AV and agent is a non-AV vehicle or pedestrian.
States, s , consist of 2D position, heading, and longitudinal velocity.
Control variables, u , are heading rate and longitudinal acceleration.
A trajectory is a sequence of states, or states and controls, depending on the context.
Distance between states is measured by the 2D Euclidean distance, whereas root-mean-squared state distance over time is used for trajectories.
We denote trainable parameters of the prediction module by ✓ , and trainable parameters of the planning and control cost by w . We perform two types of optimization: training (also called learning ) optimizes ✓ and w to minimize a loss L over some data; planning and control optimizes the ego trajectory to minimize a cost function with ✓ and w ﬁxed.
3.1 DiffStack modules
Prediction.
We employ Trajectron++ [42], a state-of-the art CVAE that takes H seconds of state history for all agents as input, and outputs multimodal trajectory predictions for one agent a 2 A ,
[Equation]
where k 2 K is the mode of the output distribution.
We will use ˆ s a = ˆ s (1: T ) a ( ✓ ) for brevity.
The encoder of the CVAE processes agent state histories with recurrent LSTM networks and models inter-agent interactions using graph-based attention.
The decoder is a GRU that outputs a Gaussian Mixture Model (GMM) for each future timestep.
The GMM modes correspond to the CVAE’s K = discrete latent states.
To ensure predictions are dynamically-feasible, GMMs are deﬁned over controls and then integrated through a known (differentiable) dynamics function to produce a trajectory.
We use the default model conﬁguration without map and ego conditioning.
We augment the input states with an ego-indicator variable to allow for ego-agent relation reasoning.
The raw prediction training objective is the InfoVAE loss, L pred =
L InfoVAE [Equation] , the same as for the original Trajectron++.
Planning.
The planner is a sampling-based algorithm that generates a set of N dynamically-feasible ego trajectory candidates, P = { s n , u n } n 2 N , and selects the candidate with the lowest cost.
Namely,
[Equation]
where C is the cost function, ˆ s a 2 A are multimodal predictions for all agents from the prediction module, g is a given goal, and m is a lane graph.
To generate trajectory candidates we sample a set of lane-centric terminal states, ﬁt a cubic spline from the current state to the terminal state, and reject dynamically-infeasible trajectories.
The cost function is a weighted sum of handcrafted terms,
[Equation]
penalizing collisions, distance to the goal, lateral lane deviation, lane heading deviation, and control effort, respectively.
Most notably, the collision term incorporates predictions into planning,
[Equation]
, where ˆ s a,k is the k -th mode of the predicted trajectory distribution for agent a , ⇡ k is the probability of the k -th mode, ' is a Gaussian radial basis function, and || · || is the Euclidean norm.
Without loss of generality, in experiments we only do prediction for the vehicle closest to ego, and use GT futures for other agents.
The remaining terms of (3) are deﬁned in Appendix C.
We make the planner differentiable by relaxing the arg min operator of (2) into sampling from a categorical distribution, and choosing a cross entropy loss for L plan , effectively treating the planner as a classiﬁer during training. Speciﬁcally, we deﬁne the probability p n of choosing candidate n as p n =exp(   C ( s n , u n ; · ; w )) /Z , where  is a temperature parameter and Z is the normalization constant.
The planning loss is then L plan =
L CE =
P n 2 N 1 ( s n = s ⇤ ) log p n , where s ⇤ is a target trajectory chosen depending on the training setup (see Section 3.2). We can now compute  [Equation]  w
[Equation] C  w ; and similarly,  L [Equation] ✓
[Equation] , where all terms exist. Control. The control module performs MPC over a ﬁnite horizon using an iterative box-constrained linear quadratic regulator (LQR) algorithm [43].
Formally, we aim to solve s ctr , u ctr =
arg min s,u
[Equation]
where C denotes the cost function, f d the dynamics, s init the current ego state, and u, u the control limits.
We use the cost deﬁned in (3) for C and the dynamically-extended unicycle [44] for f d .
We initialize the trajectory with u plan from the planner.
The algorithm then iteratively forms and solves a quadratic LQR approximation of (4) around the current solution s ( i ) , u ( i ) for iteration i , using ﬁrstand second-order Taylor approximations of f d and C , respectively.
The trajectory is updated to be close to the LQR optimal control while also decreasing the original non-quadratic cost.
We stop iterations upon convergence or a ﬁxed limit.
To make the control algorithm differentiable we leverage Amos et al.
The iLQR optimal trajectory, s ctr , can be differentiated wrt. C and f d by implicitly differentiating the underlying KKT conditions of the last LQR approximation.
The gradients can be analytically computed by one additional backward pass of a modiﬁed iterative LQR solver.
If iLQR fails to converge, we do not backpropagate gradients.
In our setting f d is ﬁxed. We compute gradients wrt. cost parameters,  L ctr  w = [Equation]
L
[Equation]
s ctr  s ctr  C  C  w , and further wrt. prediction model parameters, [Equation] [Table]
, where L ctr is the control loss, which we will discuss next.
[Table]
3.2 End-to-end training
An important question for data-driven AV stacks is the training objective and data.
Learning in the real world is prohibitively expensive [Table] and building a simulator with realistic trafﬁc agent behavior is an open challenge.
Accordingly, standard practice is to perform open-loop training with human driving data [45, 13, 31, 23, 24, 46, 47].
We consider two common types of open-loop training settings: reinforcement learning (RL) and imitation learning (IL). In the RL setting
[Table]
we aim to minimize the (hindsight) cost of output ego trajectories, s ctr , u ctr , over a dataset, L ctr = L HC = C H  s [Equation]
, u [Equation] ; s [Equation]
[Equation] [Equation] ; w  . The hindsight cost C H captures the quality of a trajectory in hindsight , i.e., after knowing the future trajectory of non-ego agents s gt a 2 A , similar to the concept of rewards in RL.
We choose C H identical to the control cost C deﬁned in (3), but with GT future trajectory inputs instead of predictions, and ﬁxed w .
Note that our open-loop training setup does not account for the effect of ego actions on other agents; nevertheless, we treat recorded trajectories as GT futures, as in the prediction literature [48]. In this setting we only train the prediction model.
Given GT futures [Equation] where both terms exists.
For the planner’s target we choose the trajectory candidate with the lowest hindsight cost, s ⇤ =
arg min s n ,u n 2 P C H ( s n , u n ; · ) .
In the IL setting we aim to minimize the MSE between the output ego trajectory and the GT in the dataset, L ctr =
[Equation]
For the planner’s target we choose s ⇤ = s gt . The total loss for training DiffStack is L = ↵ 1 L pred + ↵ 2 L plan + ↵ 3 L ctr .
We experiment with different ↵ i values, including setting each ↵ i to zero.
In the following we omit ↵ i for brevity.
Experiments
We ask the following questions: 1) can DiffStack learn predictions that lead to better plans? 2) how does DiffStack compare to alternative planning-aware training techniques? 3) can DiffStack correct integration errors? 4) can DiffStack learn a cost from imitation? 5) do our results translate to closed-loop simulation?
We begin with open-loop experiments (1–4), and then present closed-loop results (5) in Section 4.3.
4.1 Experimental setup
Dataset.
We use the nuScenes dataset [17], comprised of state annotations for vehicles and pedestrians in 1000 scenes across Boston and Singapore.
For training and open-loop evaluation we sample suitable planning scenarios from the dataset with H =
s history and T =3 s future data.
For each scenario, we choose one vehicle to act as the ego.
Details including dataset splits are in Appendix D.
Metrics.
We evaluate DiffStack’s prediction module via standard prediction metrics: average displacement error (ADE) for the most-likely prediction and negative log-likelihood (NLL) for its full distributional output.
To evaluate planning and control, we use the cross-entropy planning loss L plan =
L CE , and control loss L ctr .
We report the hindsight cost for reinforcement learning (
L ctr = L HC ) and MSE for imitation learning experiments ( L ctr =
L MSE ).
Since control is an AV stack’s end-goal, L ctr reﬂects the overall performance of the stack.
To make these metric values more interpretable, we report them relative to a No prediction baseline and a GT prediction -based oracle.
The baseline ego plans without predictions (ignoring the predicted agent), providing a performance lower bound.
The GT oracle plans with GT futures in place of predictions, providing a notion of a [Figure] performance upper bound
Baselines.
We compare DiffStack with standard AV stacks. For a fair comparison, we use the same set of modules as in DiffStack, but without making the planner and controller differentiable. We only train the prediction model, using increasingly planning-aware training objectives.
Standard trains the prediction model with only the prediction loss L pred , unaware of downstream planning.
Next, we re-weight prediction losses (in each batch) based on a handcrafted measure of relevance for planning.
Distance weighted weights losses with the inverse distance between the ego and agent GT futures.
r Cost weighted uses the magnitude of the control cost gradient with respect to the GT future trajectory of the agent, proposed as a planning-aware prediction metric in [27].
Finally, DiffStack backpropagates gradients of the ﬁnal loss to the prediction module.
DiffStack (no L pred ) uses
L = L plan + L ctr ; DiffStack uses L =
L pred + L plan + L ctr .
Implementation details.
We implement DiffStack in PyTorch [49] and build on the open-source code of Trajectron++ [42] and Differentiable MPC [16].
We use H =4 s, T =3 s, and  t =0 .
5 s. We train all models for 20 epochs using 4 NVIDIA Tesla V100 GPUs, taking 10–20 hours.
Additional details are in Appendix C.
The code is
4.2 Open-loop results
End-to-end training is useful.
Our main results are summarized in Table 1.
The key observation is that DiffStack improves the effectiveness of predictions on ego planning by up to 15.5% compared to standard training ( L ctr =  1 . 86 vs.  1 . 61 ); and reduces the gap to the cost attainable with a GT-informed oracle by 39.1% (  1 .
86 + 2 . 25 vs.  1 .
68 + 2 . 25 ).
DiffStack also improves planning without signiﬁcantly impacting raw prediction accuracy (ADE= 1 . 27 vs. 1 .
32 , within standard error).
Alternative methods that re-weight the prediction loss in a planning-aware manner (rows 2 and 3) also help, but less than DiffStack.
Surprisingly, we can even recover comparable prediction performance in terms of ADE when training solely for planning and control objectives (row 4).
The poor distribution ﬁt (high NLL) is due to our control cost being agnostic to the variance of the predicted GMMs.
DiffStack makes fewer prediction errors that affect planning.
Qualitatively analysing predictions and plans shows that the main source of improvement from end-to-end training is the reduction of spurious and/or unrealistic predictions that lead to plans with large (unnecessary) deviations from the lane center or the goal position.
Fig.
3 shows two particular examples.
In both cases, DiffStack’s predictions are more realistic and accurate, yielding much more reasonable downstream plans.
DiffStack can compensate for system integration errors.
Integrating independently developed modules into an AV stack can be challenging due to, e.g., misaligned module interfaces.
Tableshows results for an experiment that explores this issue.
We introduce an artiﬁcial interface mismatch between prediction and planning by adding a ﬁxed 1m offset to all GT prediction targets.
As a result, the standard model yields very poor plans (note the positive value in row 1, indicating worse performance than baseline planning).
However, DiffStack can compensate for interface mismatch and substantially improves the overall plans, suggesting that it can learn to correct erroneous predictions that affect planning.
More broadly, these results demonstrate the potential for differentiable AV stacks
[Table]
[Table] to reduce various development/engineering costs associated with developing AVs, e [Table] g., by replacing tedious manual parameter tuning with end-to-end data-driven optimization.
We further explore this topic in the cost tuning experiments below.
DiffStack can imitate humans better.
Imitation learning results are summarized in Table 3.
The overall performance of the stack is now measured by L ctr =
L MSE , the MSE between planned and human expert trajectories relative to the MSE for the baseline planner (without predictions).
We observe a similar trend as before; however, the results also reveal some limitations.
The standard stack performs worse in terms of MSE than the baseline stack which neglects predictions entirely (note the positive sign for relative L MSE in row 1).
While DiffStack improves MSE signiﬁcantly, the absolute difference is small.
We hypothesize this is caused by our cost function not being rich enough to fully capture human driving behavior; and because agent-agent interactions that affect planning are rare in nominal driving data (comprised mainly of lane- and speed-keeping).
These limitations could be addressed by a richer cost function and alternative goal deﬁnitions, which we leave to future work.
[Table]
DiffStack can learn a better control cost.
While our main use case in this paper is to learn planningaware prediction, DiffStack opens up various other opportunities for data-driven optimization of various components of the AV stack.
For example, an important challenge in practice is to design a cost function for planning and control.
In this experiment, we explore DiffStack’s potential to learn interpretable control costs that optimize the quality of output plans from data.
Table 4 reports results in the imitation learning setting, where we ﬁrst train a prediction module (as before), then we train for an additional 20 epochs allowing DiffStack to update the weights w i of the control cost (3) by backpropagating gradients from the ﬁnal control loss L ctr .
Compared to the default hand-tuned weights (row 1), DiffStack signiﬁcantly decreases plan MSEs (row 2).
The resulting learned weights w i are lower for the control effort term, and higher for the goal and lane keeping terms (see Appendix A).
To ensure safety, we ﬁx the weights for collision avoidance.
We leave comparison with alternative techniques for learning control costs to future work.
We expect DiffStack to be more effective compared to, e.g., Bayesian optimization, where the number of learned parameters is large.
Ablations.
Results of an ablation study can be found in Appendix A.
In short, improvements from
DiffStack are consistent with our observations when training for L
=
L pred + L plan ; L =
L pred + L ctr ; when changing the relative scale of loss components; with higher time resolution  t =0 . 1 for planning and control; and when only using one of the planning or control modules.
4.3 Closed-loop evaluation
Simulation.
We perform closed-loop simulation using a simple log-replay setup, where ego states are unrolled based on the planned control outputs and known dynamics, while non-ego agents follow their ﬁxed trajectories recorded in the dataset. The evaluation scenarios are T sim = 10 s long. The goal
[Table]
and lane inputs for planning are updated in each simulation step based on the logged ego trajectory.
We compute the following metrics.
Trajectory Cost captures closed-loop performance: it evaluates the cost function C on unrolled simulation trajectories, 1 /T sim
[Equation]
where s sim and u sim are the unrolled ego state and control.
We exclude the goal cost term because the goal is updated in each simulation step.
Open-loop Cost is the average of hindsight costs calculated open-loop at each simulation step, L HC =
C ( s ctr , u ctr ; s gt a 2 A , g, m ) .
Collision Cost is the collision term in the trajectory cost, w 1 C coll ( s ( t ) sim , s gt , ( t ) a 2 A ) .
Lane Cost is the sum of lane keeping terms, w 3 C ` ?
+ w 4 C ` ] .
Control Effort is the control effort term, w 5 C u .
Deviation is the average distance between unrolled and logged ego trajectories from the data, || s ( t ) sim
s gt , ( t ) || .
Results.
Simulation results are in Table 5.
Most importantly, we observe similar relative improvement for DiffStack in terms of closed loop trajectory cost as in terms of open-loop cost.
DiffStack performs substantially better that the no prediction baseline, but somewhat surprisingly the standard stack performs worse, both in terms of closed-loop trajectory cost and open-loop cost.
Analysing the cost components sheds light on possible reasons.
As expected, incorporating predictions into planning results in lower collision costs, but higher lane keeping costs.
The contribution of these two terms to the average cost depends on the data distribution, e.g., the frequency of close interactions where predictions are useful.
As we saw earlier in Fig.
3, poor predictions in the standard stack frequently lead to unnecessary lane deviations in the planner.
This is reﬂected in the substantially higher lane cost and control effort cost for the standard stack compared to DiffStack in Table 5.
Limitations. One limitation of our work is the open-loop training and log-replay based simulation setup. In lieu of a real-world AV or a simulator with strong behavioral realism, this is standard practice; however, recent efforts on accurate behavior simulation could be leveraged in the future [50, 51, 52, 53, 54].
Our implementation of DiffStack also has limitations.
First, it is not differentiable wrt. all possible parameters, e.g., no gradients ﬂow from the control loss to the planner’s trajectory candidate generator.
Future work may develop more sophisticated differentiable algorithms and explore ideas for gradient approximation for non-differentiable components [55].
Second, individual modules could be improved, e.g., by adding a more sophisticated prediction model, improving candidate sampling in the planner, and adding trust-region constraints to the controller.
Finally, even though hand-engineered components, modularity, and intermediate training objectives in differentiable stacks remedy challenges of learning AV policies end-to-end, other challenges naturally remain.
For instance, designing an overall performance metric, or reducing the scarcity of and cost to obtain (interesting) driving data remain open problems, each of which are impactful areas of future work.
Conclusions.
In this paper we take a step towards fully-differentiable and modular AV stacks by introducing key differentiable components for prediction, planning, and control.
Our experimental results show the potential beneﬁts of jointly training modules of AV stacks for downstream performance.
For motion prediction in particular, our results indicate that there is value in moving from purely prediction-oriented evaluation metrics towards downstream task-oriented metrics, in line with arguments in recent work [26, 27, 28].
While our experiments focused on learning planningaware predictions, DiffStack opens up various exciting opportunities for task-oriented learning in modular stacks.
For example, we may learn a rich neural network as part of the control cost to capture hard-to-engineer concepts, e.g., reasoning with occlusions or accounting for uncertainties.
Eventually, we envision having differentiable modules for the entire AV stack, allowing any subset of modules to be learned and optimized for a downstream task.
Overall, this would relax information bottlenecks and enable uncertainty to more easily propagate through the stack without needing to forgo interpretability, modularity, and veriﬁabilty of the various components. Acknowledgments
We thank Yuxiao Chen and Edward Schmerling for generous help with our planner implementation and closed-loop evaluation experiment; Christopher Maes for fruitful discussions on differentiable
MPC and cost learning experiments; and Nikolai Smolyanskiy for valuable high-level feedback.