Agent Forecasting at Flexible Horizons using ODE Flows
Abstract
In this work we describe OMEN, a neural ODE based normalizing ﬂow for the prediction of marginal distributions at ﬂexible evaluation horizons, and apply it to agent position forecasting.
OMEN’s architecture embeds an assumption that marginal distributions of a given agent moving forward in time are related, allowing for an efﬁcient representation of marginal distributions through time and allowing for reliable interpolation between prediction horizons seen in training.
Experiments on a popular agent forecasting dataset demonstrate signiﬁcant improvements over most baseline approaches, and comparable performance to the state of the art while providing the new functionality of reliable interpolation of predicted marginal distributions between prediction horizons as demonstrated with synthetic data.
1. Introduction
Autonomous driving has beneﬁted tremendously from recent progress in deep learning and computer vision (Grigorescu et al., 2019).
The capability of recognizing trafﬁc signs (Arcos-Garc ´ ıa et al., 2018; Zhou et al., 2020), localizing pedestrians (Mao et al., 2017; Liu et al., 2019), etc. makes it possible for autonomous vehicles to “see” the world (Zhao et al., 2019).
However, one critical component for safe and efﬁcient planning in autonomous vehicles is an accurate prediction of the future position of such agents (such as pedestrians or moving vehicles) in the environment (Mozaffari et al., 2019; Rudenko et al., 2020).
Despite the importance of the position prediction problem, the performance on this task is still far from satisfactory because of the following challenging requirements: (1) predictions must be conditioned on the environment, as contextual clues are essential for an accurate prediction (an example is given in Fig. 1a); and (2) predictions are required to be highly multi-modal (shown in Fig.
1b)
as the real-world environment often exhibits junctions where an agent has several distinct possible future trajectories, and mode collapse in these moments could lead to disastrous planning outcomes.
It is common to frame the agent forecasting task as learning marginal distributions over potential agent positions (Makansi et al., 2019;
Oh & Valois, 2019; Zieba et al., 2020), also known as “occupancy maps”, a popular representation in planning for robotics and autonomous vehicles (Grigorescu et al., 2019; Mozaffari et al., 2019).
By predicting the marginal distribution at a speciﬁc point in time, these methods are often superior at capturing the complex multi-modal nature of the data, avoiding the challenges of generating diverse trajectories (Ma et al., 2020).
In addition, while the underlying process of an agent’s trajectory is continuous, most popular forecasting models operate on a discretized representation of time chosen during training (Whittle, 1951;
Rhinehart et al., 2018; Mozaffari et al., 2019; Makansi et al., 2019; Salinas et al., 2019; Tang &
Salakhutdinov, 2019; Rhinehart et al., 2019; Oh & Valois, 2019; Zieba et al., 2020).
The granularity of time-steps used in training constrains the resolution and utility of these approaches.
Please refer to the Appendix for a detailed discussion on related works.
Recently, Deng et al.
(2020) demonstrated a conditional temporal process which can produce marginals and trajectories fully continuous in time.
However, the expressiveness of this approach is ultimately bounded by the formulation as a stochastic process, taken in their paper to be a differential deformation of the Wiener process.
Building upon this approach, we propose a novel normalizing ﬂow based architecture motivated by the assumption of modelling a continuous temporal process, where our model deﬁnes a new temporal process rather than deforming an existing one.
The described model is shown in Fig. 1c.
The main contributions of this work are summarized as following
: (1) An expressive, multi-modal conditional normalizing ﬂow based model for predicting agent positions.
(2) A model capable of predicting at ﬂexible horizons, including those not seen in training.
(3) A ﬂow architecture that embeds assumptions that, for a continuous process, pre-
[Figure]
dicted marginal distributions deform smoothly in time.
Demonstrations on both synthetic data, and an important agent forecasting dataset.
2. Method
In this section, we present our model and its optimization.
We consider the task of predicting marginal distributions over future vehicle positions based on asynchronous conditioning information.
Speciﬁcally, given 2D positional data x :=
{ x ( t (cid:48) i ) } i for an agent at asynchronous times t (cid:48)
i ∈ T (cid:48) , we are interested in the marginal distributions p ( { x ( t i ) } i ) , with T (cid:51) t i > max( T (cid:48) ) , where T is a set of target horizons.
In practice, we may also have image-based auxiliary information a = { a ( t (cid:48) i ) } i , such as Lidar scans, and write φ := { x , a } to summarize all available information up to time t 0 :
=
max( T (cid:48) ) .
Due to the nature of the data we work with, we will principally refer to timepoints (e.g., t i , t (cid:48) i ), however, our model is continuous in time, and as such it will at times be necessary to refer to the continuous axis of time t which those observations lie on.
Further the positional data x ( t i ) is taken to be the discrete vectorized observations of a function x ( t ) .
Our approach builds upon previous work on normalizing ﬂows (NFs) and its continuous counterparts.
We refer the reader to (Rezende & Mohamed, 2016; Chen et al., 2018;
Grathwohl et al., 2018; Papamakarios et al., 2019; Kobyzev et al., 2020) for additional details.
2.1. Normalizing Flows with Informative Base Distributions
In the normalizing ﬂow literature, it is usually assumed that a sufﬁciently expressive ﬂow makes the choice of base distribution irrelevant (Papamakarios et al., 2019; Kobyzev et al., 2020), and is therefore commonly chosen as a simple Gaussian distribution.
However, recent works (Deng et al., 2020;
Jaini et al., 2020; Mahajan et al., 2020) have started exploring constructions where the choice of base distribution embeds information about the target distribution, allowing good approximations of the target distribution with simpler ﬂow transforms.
For example, Jaini et al.
(2020) demonstrated that for a target distribution with heavy tails, choosing a base distribution with similar heavy tails can be more effective than a wide variety of modern complex NF transforms in capturing the target distribution accurately.
Inspired by the aforementioned discussion, we suggest that to model the distribution of p t ( x ( t ) | φ ) for a range of values of t > t 0 , a desired property of the model would be that the distributions of p t ( x ( t ) | φ ) and p t + (cid:15) ( x ( t + (cid:15) ) | φ ) should be similar for small (cid:15) and identical when (cid:15) → 0 .
In other words, p t ( x ( t )) can be served as an informative base distribution of p t + (cid:15) ( x ( t + (cid:15) )) .
This can be realized by incrementally transforming distributions as time progresses.
Therefore, we can formulate the proposed model as follows: at any target time in the future, we can describe the target distribution p t + (cid:15) ( x ( t + (cid:15) )) as a transform f (taken to be a normalizing ﬂow) forward in time from the previous time-step p t ( x ( t )) ,
[Equation]
In addition, we can take advantage of the fact that the series of ﬂow transforms at any point in a sequence building out from the base distribution represents a valid normalizing ﬂow.
Therefore, we can implement a network with multiple outputs, with each output further from the base distribution n m = t = t = t = t = t = t = t = t = t =
t = n m = n m =
* * * * learning to predict a point further into the future.
This formulation, inspired by recent progress on informative base distributions for NFs (Deng et al., 2020; Jaini et al., 2020;
Mahajan et al., 2020), motivates our proposed architecture.
2.2. Representation Through a Continuous Conditional Normalizing Flow
Built upon the discrete model described above, we realise the proposed NF architecture by adopting a neural ODE representation.
With this approach, we ﬁnd our model can, with minimal regularization (Finlay et al., 2020), learn reasonable interpolations between evaluation points at training phase, allowing us to produce valid marginal distributions at arbitrary target times.
The proposed model utilizes the abovediscussed “prior” intuition when constructing marginal distributions by taking marginals at earlier time-steps as informative base distributions.
A illustration outlining this approach is available in the Appendix.
To facilitate asynchronous conditioning when predicting conditional marginal distributions, a vector of conditioning information from an encoder model is passed to the neural
ODE.
Speciﬁcally, as an extension to (Chen et al., 2018;
Grathwohl et al., 2018), this information is concatenated to the input of a fully-connected neural network f described by the neural ODE transform ∂z ( t ) ∂t , such that for some parameters θ and conditioning information φ we have
[Equation]
Following (Chen et al., 2018; Grathwohl et al., 2018), setting z ( t i ) to match the an observation x ( t i ) , we can solve the initial value problem to ﬁnd the equivalent point in the base distribution z (0) :
[Equation]
Calculating likelihood estimates at multiple horizons of interest simply requires solving the initial value problem for each different choice of t , where here the temporal axis of the ODE is explicitly aligned with the axis of time in the dataset of interest.
A ‘trajectory’ can be generated by ﬁrst sampling from the base distribution and then solving the ODE for the sampled point at t = 0 , however unlike a true trajectory the only source of stochasticity is the initial sample from the base distribution.
Training.
The proposed model can be optimized by minimizing the mean negative log-likelihood of distributions at | T | target horizons.
Therefore, our optimization objective can be formulated as:
[Equation]
Note that although the model is trained on a ﬁnite selection of time-steps, inference (evaluation) can be conducted at any time.
3. Evaluation
In this section we demonstrate the ability of the model to generate realistic position estimates for an agent at a future time in both synthetic datasets and a complex autonomous driving environment.
3.1. Position Estimation on Synthetic 2D Data
In order to explore our model’s ability to interpolate and extrapolate through time we created a synthetic multi-modal temporal process dataset.
This process consists of radially growing angular distribution bands.
The bands have 3 different modes.
The modes control the angular division of distributional bands.
At each time-step the radial distance of the band grows with step length drawn from a normal distribution.
Conditioning information on the number of modes n m ∈ { 1 , 3 , 8 } is encoded using an MLP before concatenated to every layer of the neural ODE ﬂow in place of φ .
Our model was trained on a speciﬁc subset of time [Table] [Table] points t ∈ { 10 , 20 , 40 , 50 , 60 , 70 } , then evaluated at a variety times never seen in training, including examples of both interpolation and extrapolation.
Performance on loglikelihood estimation are comparable to a model trained explicitly on held out times.
Full results are show in Table 1, qualitative results are shown in Fig.
3.2. Agent Forecasting Experiments
Baselines and Ablations.
Results from our model are compared to several leading approaches for likelihood estimation on agent forecasting.
Minor modiﬁcations to the CTFP model (Deng et al., 2020), a discrete ablation of OMEN, and an ablation of OMEN without conditioning information are described in the appendix.
While all baselines are capable of producing likelihood estimates for times seen in training, only the full OMEN model, its ablation without conditioning information, and the CTFP model (Deng et al., 2020) are able to produce likelihood estimates for unseen time points.
Metrics.
Following Rhinehart et al.
(2019), results are presented here using the extra nats metric ˆ e , which provides a normalized and bounded likelihood metric, ˆ e :=
H ( p (cid:48) , q ) − H ( η ) / ( | T | · N D ) , where H ( p (cid:48) , q ) is the crossentropy between the true distribution p (cid:48) perturbed by some noise η (taken here as η = N ( 0 , 0 . 01 2 · I to match Rhinehart et al. and our models prediction q , N D is the number of dimensions in the position data, and H ( η ) can be calculated analytically.
Following Oh & Valois (2019) we combine our marginal predictions at separate horizons to form a joint prediction to allow direct comparison to
Rhinehart et al.
PRECOG Carla Dataset.
The PRECOG Carla dataset (Rhinehart et al., 2019) is comprised of the complex simulated trajectories of an autopilot and four other agents in the Carla trafﬁc simulation (Dosovitskiy et al., 2017), and includes additional Lidar data centred on the main autopilot agent.
Here train, validation, and test data subsets were chosen to match Rhinehart et al.
OMEN and its ablations were trained to minimize the NLL of PRECOG
Carla’s autopilot for all future time-steps available in the dataset.
Results are presented in Table 2, and plots showing example predictions are available in the Appendix.
We also refer the readers to the Appendix for further implementation details.
4. Conclusion
We presented a normalizing ﬂow based architecture with a structure motivated by the assumption of modelling a continuous temporal process.
Experimental evidence suggested that the constraints that allow for the smooth interpolation of likelihood estimates did cause some degradation in performance, however novel new capabilities are demonstrated in comparison to other leading approaches for likelihood estimation on agent forecasting.
Speciﬁcally we demonstrated the ability to conditionally model complex processes, and to both interpolate and extrapolate those results through time.
Further, performance on the important and challenging task of agent forecasting is explored, and comparable performance to the state-of-the-art is achieved.
In future work the authors plan to extend this approach to the important task of multi-agent forecasting, where a normalizing ﬂow formulation is expected to be particularly useful for capturing the complex high dimensional distributions.
A. Related Works
The proposed framework is intrinsically related with two broad literature families: (1) ode based time-series forecasting models, and (2) distribution based forecasting models.
In this section, we review statistical models from relevant literature, and discuss the difference between the proposed method with previous works.
Neural ODEs for Time Series Forecasting.
Much recent work has explored embedding neural ODEs in models designed to process sequential data, like Recurrent Neural
Networks (RNNs), replacing the hidden state with a neural
ODE which evolves as a function of time (Rubanova et al., 2019; Brouwer et al., 2019; Voelker et al., 2019).
These works are principally pre-occupied with solving the problem of encoding asynchronous time series data, in contrast we instead focus predicting the evolution of a probability distribution in what is assumed to be a continuous process.
Other recent works have used neural ODE based ﬂows to connected multiple distributions (Li et al., 2020;
Rempe et al., 2020).
As in our architecture this models leverage a neural ODE ﬂow to smoothly interpolate between multiple complex distributions.
However unlike our model this transformation is not aligned with the temporal axis of the observed data.
Similar to our proposed architecture,
Tong et al.
(2020) uses a neural ODE ﬂow to connect predictions at several horizons, aligning ODE ’time’ with the time of observations.
However their model uses no conditional information, and generates plausible trajectories between observed data rather than attempting to forecast future marginal distributions.
In Deng et al.
(2020) the model learns a distribution through time by ﬂowing from the target distribution to a Wiener process.
Similar to the work presented here this approach allows for an efﬁcient estimation of the marginal distribution at any target horizon of interest.
The key distinction is in their method the continuous prediction as a function of prediction horizon comes from the choice of a Wiener base distribution, separate from the choice of ﬂow model.
In our work the continuous behaviour is instead a direct result of the ﬂow architecture used, deﬁning a new temporal process rather than deforming an existing one.
Concurrent and closely related to our work is Chen et al.
(2021), which explores a similar architecture for the related problem of point processes, and also utilizes a continuous normalizing ﬂow to describe a marginal distribution across predicted event features as a function of target time.
However their approach differs from our own as they are principally concerned with conditioning on the features and timing of past events, to predict the timing and features of discrete future events, where our model is concerned with the smoothly interpolated prediction of an underlying continuous process (e.g. the path of a vehicle) using a synthesis of extremely high dimensional conditioning information (lidar, cameras etc.).
Practically this means that the way conditioning information is passed to the continuous ﬂow model is quite distinct in the two approaches.
Speciﬁcally in their model an attention mechanism allows sharp changes in the conditional distribution as a function of time, consistent with modelling a discontinuous point process.
In our model a single vector of conditioning information in used across all time, consistent with modeling a continuous temporal process, and allowing for the smooth interpolation of marginals through time- a core functionality our model provides in contrast to other leading approaches
. Distribution-Based Forecasting Models. Auto-
regressive forecasting models provide a way to generate trajectories of any length(Whittle, 1951), with modern models allowing for the prediction of expressive distributions which in can capture complex multi modal behavior (Salinas et al., 2019;
Qiu et al., 2020) with a number of approaches utilizing normalizing ﬂows in some way (Kumar et al., 2019; Shchur et al., 2019;
Mehrasa et al., 2019;
Bhattacharyya et al., 2019; Rasul et al., 2021)
However in order to infer the statistics of a marginal distribution beyond the next time-step extensive sampling is required, and in these works a ﬁxed discrete sampling in time is assumed.
Jain et al.
(2019) proposes an architecture which, similar to our approach, explicitly relates marginal distributions in time.
However their model is discrete in both time and agent position, and doesn’t use the formalism of Normalizing
Flows.
Instead learning direct transforms on an discretized representation of the marginal distribution or an “occupancy grid” (Grigorescu et al., 2019; Mozaffari et al., 2019).
Rhinehart et al.
(2019) describes a model which uses a series of afﬁne transforms to learn a conditional joint distribution over a selection of agents and horizons.
This formulation is similar to a discrete version of our model with a much less expressive choice of Normalizing Flow, and unlike our model is limited to only predict times seen in training.
Most similar to our model is Oh & Valois (2019) a conditional auto-regressive ﬂow for marginal prediction at ﬂexible horizons.
Here however the ﬂow model is a series of discrete layers, speciﬁcally a conditional extension of Neural Autoregressive Flows (Huang et al., 2018b) with the predicted horizon passed as an explicit conditioning variable.
B. Method
Figure 3 shows an overview of our models computational graph.
B.1. Mapping Data Time to ODE Time
One clear exception to the assumptions outlined earlier in the paper, is in the earliest possible predicted marginal, at some time t , with t > t 0 .
This distribution can be arbitrarily distinct from the base distribution.
To solve this problem a “warm-up” period is introduced between the base distribution and the ﬁrst evaluation point, with the length of the warm-up period optimized as a parameter in training.
With this formulation, the translation from time in the target space t i to time in the ODE space τ i , given the warm-up period set by the parameter α is given simply as τ i = α + t i .
Following (Oh & Valois, 2019), we explore an extension of the synthetic Gaussian experiment from (Huang et al., 2018a), where a single model conditionally represents one
[Equation]
Data
[Equation]
Encoder
[Equation]
Time
Sampling z
[Equation]
Predictions Observations
[Equation]
LL
[Equation]
of three multi-modal conﬁgurations.
For OMEN, conditioning information n m ∈ 0 , 1 , 2 is encoded using an MLP before concatenated to every layer of the neural ODE ﬂow in place of φ .
Results are shown in table 3, performance is comparable to the HCNAF approach and demonstrates that our choice of a conditional neural ODE based normalizing ﬂow is capable of conditionally representing complex multi modal data.
[Table]
C.2. Precog Carla Dataset, Example Results
For Precog Carla dataset, an encoder network which is a partial re-implementation of that in (Oh & Valois, 2019), is used.
LSTM modules encode the past trajectories of agents in the environment, and a residual CNN encodes
Lidar information from a single main agent.
Speciﬁcally two seconds of historical position data at a sampling of 5hz, or 10 historical points in time, are provided to the LSTM.
The encoded trajectory and Lidar information is combined in a MLP and concatenated to every layer of a Neural ODE describing a normalizing ﬂow, as outlined in Section 3.3.
The model is trained and evaluated on the future position data of the main agent over four seconds at a sampling of 5hz, or 20 future time points.
In addition to Table 2, we also provide qualitative results.
Figures 4, 5, and 6 show example predicted conditional marginal distributions for four of the twenty horizons in the Precog Carla Dataset.
All examples are taken from the precog carla town01 test set.
C.2.1. B ASELINES AND A BLATIONS
Minor extensions are made to the CTFP (Deng et al., 2020) model to provide a functional baseline.
Speciﬁcally additional encoding information was concatenated with the output of the ODE-RNN, and an extra loss on extrapolating the predicted process into the future was added in training.
OMEN-discrete has a separate ODE ﬂow transform between each inference time point in training.
In this way it resembles a model following Eq.
1 where (cid:15) in the delta between forecast time points in the training set, and each neural ODE transform represents a separate but sequential normalizing ﬂow transform.
This ablation is expected to have superior expressive power as the representation no longer is constrained to be fully continuous in time, and each separate
ODE transform can learn its own ODE stop time, allowing for expressive power between time steps to vary.
This highlights a small degradation in performance from using a single Neural ODE to represent all time points, and suggests future approaches would beneﬁt from a learned mapping of data time t to ODE time τ .
However it does not allow for continuous interpolation of marginals in time.
OMEN-nocon has no conditioning information φ appended to the neural ODE.
This ablation is expected to have signiﬁcantly worse overall performance as the model only learns a distribution over all points observed in the training set, and we expect the task of predicting agent locations to be strongly conditional on the available environmental information.
This highlights the importance the extension to (Chen et al., 2018; Grathwohl et al., 2018) presented in this paper to include conditioning information.
[Figure]
L o g L i
k e li h oo d [Figure]
L o g L i
k e li h oo d [Figure]
L o g L i
k e li
h
oo d