Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning
Abstract
Learning-based behavior prediction methods are increasingly being deployed in real-world autonomous systems, e.g., in fleets of self-driving vehicles, which are beginning to commercially operate in major cities across the world.
Despite their advancements, however, the vast majority of prediction systems are specialized to a set of well-explored geographic regions or operational design domains, complicating deployment to additional cities, countries, or continents.
Towards this end, we present a novel method for efficiently adapting behavior prediction models to new environments.
Our approach leverages recent advances in meta-learning, specifically Bayesian regression, to augment existing behavior prediction models with an adaptive layer that enables efficient domain transfer via offline fine-tuning, online adaptation, or both.
Experiments across multiple real-world datasets demonstrate that our method can efficiently adapt to a variety of unseen environments.
All of our code, models, and data are available at https://github.com/NVlabs/adaptive-prediction .
Introduction
Learning-based behavior prediction methods are becoming a staple of the modern robotic autonomy stack, with nearly every major autonomous vehicle organization incorporating state-of-the-art behavior prediction models in their software stacks [1–7].
In order to deploy such systems safely and reliably alongside humans, organizations extensively train and test models in their specific operational domains.
While this improves system accuracy and safety within the targeted domain, it does so at the cost of generalization.
Specifically, deploying autonomous vehicles to different cities or countries remains challenging due to their different social standards, laws, agent types and road geometries.
Contributions.
Our key contributions are threefold.
First, we combine ALPaCA-based [8] last layer adaptation with recurrent models and show that they work synergistically to enable both broad generalization and efficient transfer.
Second, we present a novel trajectory forecasting algorithm based on
Trajectron++ [9] that naturally pairs with the ALPaCA adaptive last layer.
In particular, we introduce a within-episode aleatoric uncertainty prediction scheme that enables state-dependent multimodality and modulates last layer adaptation.
Finally, we show experimentally that this architecture broadly extends the deployment envelope of trajectory forecasting algorithms, enabling efficient transfer to problem settings substantially different from those on which they were trained.
Related Work
Adaptive Behavior Prediction.
There have been a plethora of trajectory forecasting datasets released in recent years, with thousands of hours of data now publicly available [10].
Accordingly, there have been many methods tackling the problem of domain adaptation for trajectory forecasting.
They can broadly be categorized into the following three groups: (1) Memory-based approaches generally tackle domain shifts by first computing past and future trajectory embeddings (e.g., with recurrent neural networks) and then leveraging an associative external memory to store and retrieve the embeddings at test-time [11, 12].
One key drawback, however, is that constantly-growing extra memory may not be feasible for platforms with fixed memory and compute limits.
In contrast, our method does not require any extra memory.
(2) Architectural methods generally employ neural networks whose intermediate representations or structures are generalizable and can transfer to different domains [13, 14].
A key drawback of such approaches is that they either require labeled data from the target domain during training [13] or are not inherently online-adaptive [14].
(3) Finally, least-squares-based methods typically employ recursive least squares to adapt models to data observed online.
Similar to our approach, RLS-PAA [15, 16] updates the last layer of a trained neural network via iterative least squares.
However, since RLS-PAA is not performed during training, it is generally only useful for adapting to local agent behaviors within the same dataset (rather than domain transfer).
Few-Shot Learning by Meta-Learning.
Meta-learning exists as a particular approach to few-shot learning, in which transfer to a new domain is enabled by “learning to learn" across a set of training tasks [17–24].
By learning update rules on many tasks, an agent may learn to learn effectively, and thus efficiently, in a new task.
While there are several ways to design meta-learning algorithms, two are of note for this work.
First, black-box meta-learning exploits sequence-processing neural network models such as recurrent networks [20, 24, 25].
While expressive, these models have no particular inductive bias toward learning, and thus they are practically inefficient and require huge amounts of training data.
To address this issue, optimization-based meta-learners [17, 8, 26, 22], which directly leverage optimization in the inner learning loop, have been a major research focus.
Our work can be viewed as a combined black-box and optimization meta-learner.
In particular, it was previously shown that recurrence alone is not sufficient for effective transfer.
We address this by showing that recurrence-based adaptation can be effectively paired with optimization-based meta-learning.
Generalizing Prediction Models Through Adaptive Meta-Learning
Problem Formulation.
Our core problem formulation follows that of trajectory forecasting.
Namely, we wish to generate plausible distributions p ( y t | x t , I ( t ) ) of a time-varying number N ( t ) of agents’ future trajectories y t
= s ( t +1: t + T ) 1 ,...,N ( t ) given their past state histories x t = s ( t − H : t ) 1 ,...,N ( t ) and (optional) scene context I ( t ) .
At train time, we assume access to a dataset D S
= { x j , y j } collected from a set of source environments S , and aim to obtain a model p ( y | x ) that can be effectively deployed to a target environment T .
Since this work tackles methods for adaptation, we focus on the following two problem settings: Online , where methods must adapt to streaming data (i.e., new inputs are observed at every timestep t ), and Offline , where prediction methods have access to a small amount of already-collected data D ⊂ D T from the target environment T on which they can finetune.
Architecture.
At a high-level, our method leverages an adaptive formulation of meta-learning, an overview of which can be found in Appendix A.1.
Our architecture takes an encoder-decoder structure, similar to Trajectron++ [9].
Inputs (e.g., agent histories and any encoded scene context) are mapped to an overall scene encoding vector v via an attentional graph-structured recurrent network. v is then fed to the recurrent decoder, which maps v , the current hidden state h t , and input states of the observed agents in the environment s ( t ) 1 ,...,N ( t ) to a distribution over actions taken by the predicted agent.
This action is then sampled and passed through the chosen dynamics model to generate the next state, which can then be fed back into the decoder at the next timestep.
Whereas Trajectron++ models multimodality via discrete latent variables [9], our model accounts for multimodality via controllable aleatoric uncertainty, sampling, and recurrent network dynamics.
We parameterize both output features Φ t =
Φ( h t ) and the predictive noise covariance Σ t = Σ ε ( h t ) as a function of the decoder’s hidden state h t .
This parameterization as a function of history is important: it captures irreducible aleatoric uncertainty as a function of state (or state history).
Multi-step Prediction.
For multi-step prediction, we begin by sampling from the last layer, via w i t +1 | t ∼ N ( ¯ w t +1 | t , S t +1 | t ) for samples i = 1 , . . .
, N .
We then set τ = 0 and repeatedly sample u it + τ | t ∼ N (Φ it + τ ¯ w it + τ | t , Σ it + τ ) , compute the next state s it
+ τ +1 | t
= f
( s it + τ | t , u it + τ | t ) , set h it + τ +1 ← DecoderRNNCell
( s it + τ +
| t , h t + τ ) , sample the next ¯ w it + τ +1 | t ∼ N ( ¯ w it + τ | t , Σ ν ) , and set τ ← τ + 1 for the next iteration.
Looping this procedure for T steps gives N samples of T -length forecasts.
Gradients are computed through the sampling via the reparameterization trick [27]. [Table] [Figure]
Loss.
We propose to approximate the predictive density via kernel density estimation [33], maximizing 1 T (cid:80) Tτ
= t log 1 N (cid:80) Ni
=1 N ( s t + τ +1 ; s it + τ +
| t , V it + τ +1 ) , where s t + τ +1 is the true state at time t and V it
+ τ +1 corresponds to the covariance matrix used at time t + τ + 1 for particle i .
Note that the choice of V it
+ τ +1 is important; If V it + τ +1 is fixed across time, later samples may induce extremely large variance in our gradient estimation.
Accordingly, we propose to integrate the uncertainty via
V it
+ τ +1 = (cid:80) t
+ τ +1 k
= t Σ ik .
In the case where the dynamics are a single integrator (a common choice in
Trajectron++ [9]), this captures the state uncertainty induced by aleatoric control uncertainty exactly .
Adaptation.
In the online setting, we make a probabilistic prediction, the mean and variance of which are used to update the last layer using the update equations described in Appendix A.1.
Critically, adaptation in this setting is done per agent (as each agent has only local information) and with temporally-correlated data.
In the offline setting, a small dataset from the target environment is assumed to be provided.
This dataset, which consists of several different agents’ state information, is then used to adapt the last layer via the same adaptation mechanism.
Combining Last Layer Adaptation with Gradient-Based Finetuning.
We can also combine our approach with gradient-based finetuning for even more efficient and effective domain transfer in the offline setting.
In particular, we first perform M steps of last layer adaptation and then switch to gradient-based finetuning for future updates.
This sequential update scheme leverages the fast initial adaptation of our last layer exact inference, while exploiting the high capacity and strong performance of gradient-based fine-tuning [34].
Experiments
Datasets.
We evaluate our method on the ETH [35] and UCY [36] pedestrian datasets, as well as the nuScenes [37] and Lyft Level 5 [38] autonomous driving datasets.
The ETH and UCY datasets are a standard benchmark in the field, comprised of pedestrian trajectories captured at 2.5 Hz in Zurich and
Cyprus, respectively.
As in prior work [13], up to 3 .
2 s of history are observed and the next 4 .
8 s are predicted.
nuScenes is a large-scale autonomous driving dataset comprised of 1000 scenes annotated at 2 Hz collected in Boston and Singapore.
Lyft Level 5 is comprised of 170K scenes annotated at[-
-]
Hz collected in Palo Alto.
As in the nuScenes prediction challenge, up to 2 s of history are observed and the next 6 s are predicted.
Details about our experimental protocol can be found in Appendix A.2.
Ablations, Oracle, and Baselines.
We compare against the following five ablations: (1) Base is the original Trajectron++ [9] model without any of our adaptive architecture; (2) Finetune is
Base combined with gradient-based finetuning for adaptation.
(3) K 0 is our method without any adaptation, i.e., only using the prior model; (4) K
+Finetune is the K 0 baseline combined with
[Figure] [Figure]
gen t
H i s t
o r y
A gen t
F u t u r e
A gen t
C u rr en t
N e i ghbo r H i
[Figure]
t [Figure] r
[Figure]
N e i ghbo r F u t u r e
[Figure]
e i ghbo
[Figure]
gradient-based finetuning. Note, only the last layer is finetuned (the rest of the model is frozen).
In essence, this ablation compares using gradient descent for adaptation instead of Bayesian updates and (5) Ours+Finetune is our combined gradient M Finally we also include an oracle i.e Trajectron trained data target environment representing ideal transfer performance.
Metrics.
We evaluate prediction accuracy with a variety of deterministic and probabilistic metrics:
Average/Final Displacement Error (ADE/FDE) : mean/final ℓ 2 distance between the ground truth (GT) and predicted trajectories; minADE 5 / 10 : ADE between the GT and best of 5 or 10 samples, respectively; and Negative Log-Likelihood (NLL) : mean NLL of the GT trajectory under the predicted distribution.
We also evaluate methods’ predictive calibration by computing their Expected Calibration Error (ECE) [39], comparing a model’s predictive uncertainty with its empirical uncertainty.
Direct Transfer.
In Table 1, we train on a source scene X and directly transfer to a target scene Y (denoting the pair as “X2Y"), without any online or offline adaptation to fairly compare with an assortment of state-of-the-art methods [28–32, 13].
As can be seen, our non-adaptive ablation ( K 0 ) significantly outperforms other methods on the vast majority of transfer settings, showing that our prior and its analytical propagation of uncertainty already yield strong performance.
Offline Calibration.
Fig.
1 shows that our method learns a well-calibrated prior for pedestrians, and maintains its calibration with more updates.
Results on Lyft further highlight our method’s improvement, and show that switching to gradient-based finetuning after 100 last-layer adaptation steps significantly accelerates improvement, even yielding a better calibrated model than the oracle.
Online Adaptation.
Focusing on the transfer from UCY-Zara1 to ETH-Hotel (D2B), Fig. 2 (left) shows that our method’s median ADE reduction rapidly approaches the oracle as more of an agent’s trajectory is observed online.
This performance is replicated in transferring from nuScenes to Lyft, where our approach tracks closely to whole-model finetuning while still being real-time feasible.
Qualitative Results.
Fig.
3 shows our model’s adaptation visually.
After observing only 0 .
5 s of data, our model has adapted to the significant domain shift between nuScenes and Lyft (data frequency mismatch and map format differences).
Fig.
3 (right) shows that our proposed sampling scheme yields diverse, multimodal predictions, even in an unusual intersection in Palo Alto.
Conclusion
In this work, we present a model that combines the strength of recurrent models with adaptive, optimization-based meta-learning.
Our approach has shown particularly strong results for the one-toone transfer setting.
One strength of the approach presented in this paper—which we do not address due to space constraints—is the ability to perform effective many-to-one transfer, in which many training datasets are available.
This setting is reflective of deploying in a new geographical location, where the full training dataset of many other cities is available for pre-training.
We anticipate that further training data diversity will result in monotonic improvement to performance, yielding both better calibrated priors and more expressive learned features
Acknowledgments and Disclosure of Funding
We thank Nikolai Smolyanskiy and the rest of the NVIDIA AV Prediction team, John Willes and
Brian Ichter for helpful feedback on this paper
A Appendix A.1 Background: Adaptive Meta-Learning
Our framework leverages an adaptive formulation of meta-learning.
In contrast to the more common episodic meta-learning (e.g., MAML [17]), the adaptive formulation allows tasks to vary over time within an episode, similar to meta-learning for continual learning [40].
In particular, our approach builds on an extension to ALPaCA [8] developed in [41].
Instead of Bayesian linear regression (as in
ALPaCA)
, adaptation is done via Kalman filtering on the last layer of the network, allowing for task drift over time [42].
Inputs at time t (e.g., a history of data) are written x t and outputs are written as y t .
Our objective is to infer the probability of y t conditional on x t .
We write our predictive model as
[Equation]
) where Φ θ ( · ) is a matrix of neural network features parameterized by θ , w t is a (time-varying) vector last layer, and ε t is a zero mean Gaussian noise instantiation at time t with covariance Σ ε (assumed independent across time).
This differs from standard neural network regression in several ways.
First, instead of a vector of features and a matrix last layer, we have a vector last layer and a matrix of features; this yields simpler inference within the model.
Second, the last layer is assumed time-varying; we choose parameter dynamics
[Equation]
to enable tractability of inference.
We typically assume simple A, b in practice, such as the identity matrix for A .
The term ν t is a noise term, with variance Σ ν .
There are several alternate choices that can be made in this framework while retaining inferential tractability, discussed in depth in [41].
Inference within this model is as follows.
Similar to Kalman filtering for state estimation, filtering consists of a prediction step—where the dynamics are applied to predict how the parameters change forward in time—and correction, where a measurement is used to update the estimate.
The prediction step is
[Equation]
where the subscript t + 1 | t denotes the prediction for the value of the parameter at time t + 1 made at time t .
In the above, A t is shorthand for A ( x t ) , which we will use for other quantities.
The terms ¯ w and S correspond to the mean and variance of the Gaussian prediction.
The correction step is
[Equation]
From this framework, the predictive loss is the log likelihood of the prediction using mean and variance ¯ w t +1 | t , S t +1 | t .
Critical to the meta-learning formulation, the parameters of the feature matrix, (possibly) the parameter dynamics matrices, and the noise covariances are trained by backpropagating through the iterated inference and prediction, using the log-likelihood over a segment of an episode as a loss.
The model thus learns features that are capable of adapting to novel environments.
Computational Complexity.
There are several considerations that are important for achieving efficient performance in the model.
First, in practice, we fix a set of parameters for each output dimension, which are adapted independently.
This corresponds to choosing a diagonal noise and parameter covariance, fixing parameter dynamics ( A ) diagonal (in practice we simply choose this to be identity), and fixing independent priors for parameters of each output dimension.
Parameterizing each output dimension independently substantially reduces computational complexity, as discussed in [43].
Furthermore, we may implement the model such that complexity is at most quadratic in the parameter dimension.
First, A must be chosen appropriately; identity is sufficient but other representations are
[Table]
[Figure]
possible. In addition, the multiplication of K t +1 Φ t +1 S t +1 | t should be done with the last two terms first, before multiplying by the gain matrix K t +1 .
This enables better representational capacity for adaptation by enabling choice of a larger number of features.
Finally, we note that prediction in the model is sequential in time due to the unrolling of the recurrent model.
Thus, at run time, parallelizing over a large number of predictions is straightforward.
Indeed, for hardware such as GPUs or other specialized cores for neural networks, parallelized matrix multiplications are extremely efficient, and thus many samples can be parallelized across.
A.2 Experimental Setup
Our approach was implemented in PyTorch [44] and all datasets were interfaced through trajdata [45], a recently-released unified interface to trajectory forecasting datasets.
ETH/UCY Evaluation Protocol.
As in recent prior work [13], we treat each of the 5 scenes in the ETH and UCY datasets as individual source domains S ∈ { ETH , Hotel , Univ , Zara1 , and
Zara2 } , and use the other 4 scenes as our target domains T ̸ = S , yielding 20 cross-domain pairings.
To ensure that we are purely evaluating adaptation, we restrict models’ training sets to only be their source domain’s training split D S , train (leaving the source domain’s validation set D S , val for hyperparameter tuning) and evaluate their performance on the entire target dataset D T .
This is a notable difference from the evaluation protocol proposed in [13], where prediction methods train on the source domain’s training split D S , train as well as the target domain’s validation split D T , val .
This complicates measuring a model’s capability for domain adaptation, as it has already seen data from the target environment during training. nuScenes → Lyft Evaluation Protocol.
We treat nuScenes as the source domain and Lyft Level 5 as the target domain.
In particular, we train models on the nuScenes prediction challenge train split, tune hyperparameters on the train_val split, and evaluate methods’ capability to adapt to the entire Lyft Level 5 sample split.
This is a much more challenging domain transfer problem than in the ETH/UCY datasets, since the nuScenes and Lyft datasets feature different underlying map annotations (e.g., nuScenes does not annotate lanes through intersections whereas Lyft does) and data frequencies (2 Hz vs 10 Hz), in addition to being collected in diverse cities with unique road geometries. [Figure] t [Figure]
T P
[Figure]
i n t
[Figure]
[Figure]
R
[Figure]
d u c t
i
o n
[Figure]
A.3 Additional Pedestrian Evaluations
Table 2 shows the same analyses as in Table 1, but with the FDE evaluation metric.
Fig.
4 shows the same analyses as Fig. 2, but with the FDE and NLL evaluation metrics.
Lastly, Fig.
5 shows detailed calibration plots which underlie the ECE computations in Fig.
Broadly, the results of these additional evaluations confirm and reinforce the performance of our approach over other baselines and ablations.
A.4 Additional Autonomous Driving Evaluations
As can be seen in Fig.
6, our method rapidly reduces NLL, outperforming Finetune after receiving only 200 updates.
Ours+Finetune further improves upon our method, showing that gradient-based finetuning is a complementary and performant addition to our last layer adaptation scheme for offline transfer.
[Figure]
[Figure]
[Figure]
Fig.
7 shows detailed calibration plots which underlie the ECE computations in Fig.
Finally, Fig.shows the same analyses as in Fig.
2, but on the other four evaluation metrics.
Overall, the results of these additional evaluations confirm and reinforce the performance of our approach over other baselines and ablations.