Graph Conditional Variational Models: Too Complex for Multiagent Trajectories?
Abstract
Recent advances in modeling multiagent trajectories combine graph architectures such as graph neural networks (GNNs) with conditional variational models (CVMs) such as variational RNNs (VRNNs).
Originally, CVMs have been proposed to facilitate learning with multi-modal and structured data and thus seem to perfectly match the requirements of multi-modal multiagent trajectories with their structured output spaces.
Empirical results of VRNNs on trajectory data support this assumption.
In this paper, we revisit experiments and proposed architectures with additional rigour, ablation runs and baselines.
In contrast to common belief, we show that prior results with CVMs on trajectory data might be misleading.
Given a neural network with a graph architecture and/or structured output function, variational autoencoding does not seem to contribute statistically signiﬁcantly to empirical performance.
Instead, we show that well-known emission functions do contribute, while coming with less complexity, engineering and computation time.
Introduction
Modeling the whereabouts of agents over space and time is an emerging research topic and important in many areas including the prediction of pedestrian movements (e g. Alahi et al., 2016; Gupta et al., 2018), sports analytics (e.g. Le et al., 2017) and autonomous driving (e g. Casas et al., 2020).
Trajectory data over multiple agents is usually highly multi-modal and comes with a structured output space:
Firstly, assuming stochastic actions of each agent, there are multiple modes with respect to the density of future trajectories.
Intuitively, one can think of different futures being all likely given a current setting.
Secondly, the output space is naturally structured, since multiple trajectories have to be estimated per time step, while individual trajectories have to account for spatial dimensionality.
Graph architectures such as graph neural networks (GNNs, Scarselli et al., 2008), or graph networks in general (cf.
Battaglia et al., 2018), are providing convincing architectural beneﬁts for learning tasks on multiagent trajectories which come as unordered sets of individual trajectories.
This is of special importance in learning scenarios where the number of agents are not ﬁxed and/or the order of the agents is not predetermined, such as predicting the movements of pedestrians or players in team sports (e.g. Alahi et al., 2016; Gupta et al., 2018; Le et al., 2017; Felsen et al., 2018).
Conditional variational models (CVMs) such as conditional variational autoencoders (CVAEs, Sohn et al., 2015) and variational recurrent neural networks (VRNNs, Chung et al., 2015) have both been proposed to learn representations that help with structured prediction.
Being an extension to variational autoencoders (VAEs, Kingma and Welling, 2014; Rezende et al., 2014), both architectures involve autoencoding an output variable, while conditioning on some input variable.
In the case of
CVAEs, the concept of the input variable is generic.
In the case of VRNNs, a speciﬁc decomposition of sequential data and assumed latent variables result in time-step-wise conditioning with respect to the past.
The inherent stochasticity of the models is suggested to help with “modeling variability” (Sohn et al., 2015) and making “diverse predictions” (Chung et al., 2015).
Recent research towards modeling multiagent trajectories proposes neural network architectures that combine components operating on graph structured data with conditional variational models.
In this paper, we mainly focus on two examples: Ivanovic and Pavone (2019), who leverage spatiotemporal graphs with dynamic edges for neighboring agents with per agent CVAE and recurrent decoder for pedestrian prediction, and Yeh et al.
(2019), who model interaction in team sports and propose to extend the VRNN model with GNNs.
Both papers report empirical results which support the assumption that the employed CVMs are important for the performance of the architectures.
By contrast, in this paper we provide evidence that a statistically signiﬁcant contribution of the conditional variational model components to the empirical results for either architecture cannot be established.
Our approach is motivated by the observation that the structure of the output space of an individual trajectory in two dimensions is not overly complex.
Revising an experiment on modeling handwriting data conducted in Chung et al.
(2015), we show that mixture density networks (MDNs, Bishop, 1994), with Gaussian mixture models (GMMs) as output distributions, are indeed sufﬁcient too capture the multi-modality and structure of two-dimensional trajectories.
Lifting these results to interactions in multiagent trajectories with graph networks shows that the output space decomposes due to permutation equivariance and thus preserves the low-dimensional structure locally.
We provide experimental evidence, that variational autoencoding does not result in higher likelihoods with respect to modeling multiagent trajectories when added to neural networks of certain structure.
We thus argue, that conditional variational autoencoding (as considered in this paper) is not the right choice for modeling multiagent trajectories.
By contrast, structured emission functions such as MDNs, perform on par or even better and seem to render variational autoencoding unnecessary.
Additionally, MDNs come with less complexity, engineering and compute.
Speciﬁcally, our contributions are: (i)
We revise an experiment from Chung et al.
(Section 5.2) and thus show, that MDN emission functions render the VRNN architecture unnecessary for modeling single two-dimensional trajectories.
(ii)
We provide a new ablation study with respect to the trajectron model (Ivanovic and Pavone, 2019)
(Section 5.3) and new baseline experiments with graph VRNNs (Yeh et al., 2019)
(Section 5.4).
We thus also show, that MDN emission functions are more signiﬁcant than variational autoencoding for modeling multiagent trajectories with graph architectures.
With this paper we hope to correct expectations towards CVMs for modeling trajectories to an appropriate level and mark the proper inclusion of CVM components for such problems as open research.
Our results are intended to encourage practitioners (interested in fast returns) to focus on well understood and possibly simpler results, before incorporating components with increasing complexity, when modeling multiagent trajectories.
Related work
Mixture density networks (MDNs) have been introduced in Bishop (1994) and are by now well established.
This holds especially for Gaussian component densities and thus GMM output distributions (cf. Bishop, 2006, for a textbook introduction). Combining RNNs with GMM emission for spatiotemporal tasks speciﬁcally, has been popularized by Graves (2013), who employed this architecture to model handwriting data from the IAM-OnDB dataset (cf. Marti and Bunke, 2002).
The IAM-OnDB dataset was subsequently used in Chung et al. and is thus also used in our experiment in Section 5.2. Ha and Eck (2018) model structurally similar data (i.e. vector images) with an (unconditional) VAE paired with a mixture density RNN decoder and GMM emission.
Regarding multiagent trajectories, especially prior work on modeling team sports usually involves heuristics to account for player ordering (Le et al., 2017; Zhan et al., 2019).
While there are arguments supporting this approach (Hobbs et al., 2019), the use of graph networks (Battaglia et al., 2018) or related attention mechanisms (Xu et al., 2015; Bahdanau et al., 2015) to model agent interaction is widespread.
Notable early work includes Sukhbaatar et al.
(2016) in general and Hoshen (2017) as well as Kipf et al.
(2018) for multiagent trajectories in sports.
Variational autoencoding approaches towards sports trajectories include Felsen et al.
(2018) and Zhan et al.
We consider SanchezGonzalez et al.
one of the earliest works on explict graph RNNs (GRNNs).
With respect to Ivanovic and Pavone (2019), follow-up work (Salzmann et al., 2020) uses the same overall design, but proposes to improve the latent variable component via the InfoVAE objective (Zhao et al., 2019).
Preliminary ablation studies regarding the combination of CVAE and GMM emission in the architecture have been conducted in Schmerling et al.
and Ivanovic et al.
While they suggest the CVAE to provide a small beneﬁt, the signiﬁcance of the results is not clear.
Casas et al.
(2020) is another recent example for the combination of CVAEs and graph networks.
Different from the architectures discussed in this paper however, a non-recurrent and non-probabilistic decoder is employed.
Conditional variational models 3.1 Conditional variational autoencoders
Variational autoencoders (VAEs, Kingma and Welling, 2014; Rezende et al., 2014) assume a generative model p ( x , z )
=
p ( x | z ) p ( z ) , where a latent variable z is introduced to better model a data point x .
Speciﬁcally, VAEs come with a stochastic optimization method for a lower bound (the evidence lower bound , or ELBO) of the marginal log-likelihood log p ( x ) .
Distributions p θ and q φ involved in the ELBO formulation are modeled with neural networks.
Parameters θ and φ can be learned via stochastic gradient ascent.
In this framework, q φ is considered a variational approximation to the true but unknown posterior of the latent variable.
Transferring variational autoencoding to conditional density estimation of some target variable y ,
Sohn et al.
(2015) proposes to optimize the ELBO of log p ( y | x ) instead:
[Equation]
Here, D KL ( q || p ) denotes the Kullback-Leibler (KL) divergence of two distributions q and p .
When the prior p θ ( z | x ) and the variational posterior q φ ( z | x , y ) are modeled as Gaussians, the KL divergence can be calculated in closed form.
Optimization of the expected log-likelihood of the target variable y with respect to the variational parameters φ is done via reparameterization (cf.
Mohamed et al., 2019).
Latent variable instances z are thus reparameterized as z = g
( x , y , (cid:15) ) , with g being a deterministic function with respect to x and y , while (cid:15) is an auxiliary noise variable, usually taken to be componentwise standard normal, i.e. (cid:15) i ∼ N (0 , 1) .
Besides CVAEs, Sohn et al.
(2015) proposes Gaussian stochastic neural networks (GSNNs) as a simpler alternative, where sampling stays consistent between learning and test time: GSNNs are obtained from CVAEs by setting the variational posterior equal to the prior, thereby reducing the model to straightforward density estimation with (learnable) noise injected during training.
As such, GSNNs make it natural to discuss two properties of CVAEs from an architectural perspective:
Firstly, the target variable y is fed into lower parts of the model.
Secondly, learnable noise is injected, enabling the network to bridge the gap between the approximate posterior (having access to the target) and the prior.
We conjecture, that autoencoding the target variable via CVMs is not necessary for some tasks, on which the noise nevertheless helps with convergence.
In a ﬁrst hypothesis, which we will address in the experiments in Section 5, we state this even stronger:
Hypothesis 1: Conditional autoencoding does not contribute towards density estimation for lowdimensional trajectories, at least if we account for its regularization properties via noise injection.
3.2 Variational recurrent neural networks
The variational RNN (Chung et al., 2015) is a conditional variational model for sequential data, which assumes a speciﬁc decomposition of a latent variable z and a data sequence x
= ( x 1 , x 2 , . .
, x T ) of length T .
This decomposition leads to the following (empirical) lower bound on a sequence:
[Equation]
where z ≤ t ∼ q φ ( z ≤ t | x ≤ t ) is sampled via ancestral sampling.
Modeling the joined distribution p θ ( x <t , z <t ) with a recurrent neural network f , such that hidden representations are h t = f
( x t , z t , h t − 1 ) , makes the ELBO eligible to optimization even for sequences with variable length.
From an architectural standpoint, the VRNN features time-step-wise CVAEs, with x t as output, and conditioning with respect to the past (of trajectory data as well as of latent variables).
Inter alia, Chung et al.
(2015) experimented on density estimation for two-dimensional handwriting.
Notably, the paper compared RNN and VRNN models with single Gaussian and GMM emission functions.
Their results suggest that the VRNN-GMM model is best for the task.
However, the experiment featured one run with train and validation splits only.
In Section 5.2, we revisit this experiment with additional rigour, questioning the assumption that variational autoencoding and structured emission functions are indeed orthogonal.
Speciﬁcally, we test our own conjecture, which we phrase as a second hypothesis:
Hypothesis 2:
Given an MDN with GMM emission function, conditional variational models are not signiﬁcant components anymore regarding the task of modeling single two-dimensional trajectories.
Modeling multiagent trajectories with graph architectures 4.1 Permutation equivariance via graph networks
Graph architectures such as graph networks (Battaglia et al., 2018) allow neural networks to process a set of inputs and aligned outputs independently of their permutation.
Also, the input set does not have to be of ﬁxed size.
Speciﬁcally, graph networks assume a graph over the data, representing inputs, outputs and hidden states as nodes or agents and (directed) edges or relations.
Disregarding global attributes, each graph network layer consists of update and aggregation functions with respect to node and edge representations.
Edge representations are updated with respect to their afﬁliated nodes.
Node attributes are updated via aggregation functions which operate on incoming set of edges.
When the aggregation functions are permutation invariant with respect to their input, the resulting node representations are permutation equivariant.
Elementwise operations, such as sum or mean, are permutation invariant.
Considering graph networks operating on a set of K partial trajectories x 1: K<t = { x 1 <t , x 2 <t , . .
, x K<t } , e.g. to learn parameter estimates for densities of individual trajectories, thus leads to a third hypothesis, which we test experimentally in Section 5.4:
Hypothesis 3: Results with respect to conditional variational models for density estimation over single trajectories do translate to multiagent trajectories, if computations are permutation equivariant.
4.2 Graph conditional variational models
Speciﬁcally, we are looking at two recently proposed graph architectures for trajectory modeling, which feature either a CVAE with RNN decoder or VRNN components respectively:
Ivanovic and Pavone (2019) propose the trajectron, an architecture for modeling trajectories of pedestrians, given their past.
Inputs to the model are position, velocity and acceleration.
Agents are considered typed nodes, while edges are created heuristically when agents are close in space.
Inﬂuences from these dynamic edges are modeled with separate RNNs in the spirit of structuralRNNs (Jain et al., 2016).
Node and edge types are accounted for by parameter sharing over involved functions.
Output of the model is a density estimation over velocities, which equal the differences in consecutive positions: ∆ x t = x t − x t − 1 .
Results can readily be used to calculate a density over future positions via integration.
While the full trajectron architecture features several more components, notable architectural choices (with respect to our research question) are the use of (structural-)RNN encoders for past trajectories, trajectory-wise CVAEs with discrete latent variables, a bidirectional RNN encoder for future trajectories and an RNN decoder with GMM emission functions. We refer to the original paper for more details.
Yeh et al.
(2019) propose the graph VRNN (GVRNN) to model interactions in multiagent trajectories.
Within the VRNN framework, this architecture models the parameterization of Gaussian prior and variational posterior, as well as the parameterization of a Gaussian decoder with GNNs which operate over individual trajectories.
Featuring recurrence over agents, i.e. h kt = f
( x kt , z kt , h kt − 1 ) , the model thus assumes a factorization of the latent variables over all ( K ) agents.
With µ denoting a mean vector, σ denoting a vector of standard deviations relating to a diagonal covariance matrix and [ · , · ] indicating concatenation, the architecture amounts to the following variational autoencoder structure (abstracting from implementation details):
[Equation]
The input to the model is positional data, while output densities are predicted over velocities also.
Skip connections are employed and node and edge types are encoded via embeddings, which are passed to the GNNs.
Notably, GMM emission functions were not experimented with.
Results on density estimation as well as metrics regarding prediction performance reported in the two papers, suggest that it is beneﬁcial to combine graph architectures with CVMs.
We argue however, that the signiﬁcance of these results was not established.
Thus, we conduct a further ablation study with respect to the trajectron in Section 5.3 and run experiments regarding GVRNN architectures with and without GMM emission functions on synthetic data in Section 5.4.
We thereby gather evidence regarding CVMs in speciﬁc graph architectures, aiming at a fourth (and last) hypothesis:
Hypothesis 4: Graph architectures modeling multiagent trajectories with CVM components do not generally result in better log-likelihoods than alike models without the CVM component.
Experiments 5.1 About evaluation
We evaluate the modeling capacity of trajectory models via log-likelihood estimation on unseen test data D test .
For models with Gaussian or GMM emission not involving stochastic auxiliary variables, this is done analytically by calculating E D test (cid:104)(cid:80)
Tt = t 0 log p ( x t | x <t ) (cid:105) .
{+In our experiments on synthetic data in Section 5.4 we set t 
+}
= 2 , while following the evaluation scheme of the reference experiments in Section 5.2 and 5.3.
For variational models, we report the expectation of the sum over ELBOs per time step instead.
For the experiment on modeling handwriting, we further report an importance sampled log-likelihood estimate, as has been done in the original experiment (and as originally proposed in Rezende et al., 2014).
For the models with prior noise, i.e. GSNNs, we simply approximate the log-likelihood via heuristically injecting mean noise (alike the usual procedure for dropout, cf.
Srivastava et al., 2014).
5.2 IAM-OnDB: modeling single trajectories with a VRNN model
In this section, we revise the experiment with respect to modeling handwriting from the IAM-OnDB dataset (cf.
Marti and Bunke, 2002), as conducted by Chung et al.
While the data comes with an additional binary indicator relating to pen lifting (which is modeled alongside), we consider the
IAM-OnDB dataset an example of two-dimensional single trajectory data.
As Chung et al.
(2015) used the experimental setting of Graves (2013), the data originally was split into training and validation sets only.
Further, the experiment only consisted of one data split.
This introduced the possibility of overﬁtting on the validation set, especially as early stopping was employed.
For our reimplementation of the experiment, we considered 5 data splits, always training on 3 / 5 of the data, validating on 1 / 5 and testing the models with the best validation results on the last/ 5 of the data.
We split the trajectories over writer IDs, as was done in the data splits used by Chung et al.
We conducted our experiment faithfully following the model description in the paper, as well as according to code provided by the authors.
1 Speciﬁcally, we applied differencing to the time series and conducted normalization with respect to the training statistics.
An LSTM cell (Hochreiter and
Schmidhuber, 1997) with tanh activation functions is used in the RNN.
For the variational methods, no further regularization is conducted with respect to the KL divergence term.
Feature extractors (one layer neural networks) for inputs x and latent variables z are learned, as they have been considered crucial by Chung et al.
Our reimplementation was done in PyTorch (Paszke et al., 2019).
Different from Chung et al. , we did not choose to model Gaussians within the emission function with full covariance matrices.
Recent work suggests this to be hindering optimization (cf.
Rybkin et al., 2020).
We used diagonal Gaussians instead for both, Gaussian as well as GMM emission.
Further, we bounded the variances from below (e.g. as in Goyal et al., 2017).
This allowed us to forgo further weight normalization or gradient clipping as well as to use a higher learning rate ( 0 . 001 , using Adam, Kingma and Ba, 2014).
We further used a larger batch size ( 128 instead of 64 ), to make use of modern hardware.
Next to the models from the original experiment, we included two GSNN variants (one with Gaussian and one with GMM emission), as well as a VRNN-I model with GMM emission.
The VRNN-I is a model, in which the VRNN prior is input- independent .
It has been proposed by Chung et al.
(2015), to test whether conditional priors are necessary.
Running our models on the original data splits, showed comparable results to the original experiment, although our implementation results in slightly better log-likelihoods overall.
However, we did not observe the reported performance gap between the VRNN and RNN models with GMM emission functions.
Test results on our data splits can be seen in Figure 1.
The results do conﬁrm these preliminary ﬁndings.
While a paired t -test shows that differences between VRNN-Gauss and RNNGMM are highly signiﬁcant ( p -value < 0 . 0001 ) a paired t -test regarding the RNN-GMM and
VRNN-GMM log-likelihoods, indicates that there is no signiﬁcant difference between models.
At the same time, mean KL divergences of the VRNN models with GMM emissions are considerably smaller than the KL divergences of the VRNN models with Gaussian emission (see Table 1).
Note, that the effect of importance sampling for the variational GMM models is thus negligible and can only be seen in the ﬁgure at higher zoom factors.
The performance of GSNNs on this task is comparable to that of RNNs.
Noise alone does not seem to increase log-likelihoods even for Gaussian emission.
[Table]
5.3 The trajectron: an ablation study with respect to its CVAE component
In this section, we report an ablation study on the full trajectron model as proposed in Ivanovic and
Pavone (2019).
The paper experiments with respect to pedestrian trajectory datasets ETH (Pellegrini et al., 2009) and UCY (Lerner et al., 2007), which have been preprocessed into coordinates and split into train, validation and test sets by Gupta et al.
In total, there are splits for 5 different scenes.
The code of the model, as well as the training and evaluation procedures including all hyperparameters, has been made available by the authors.
This made it possibly to readily conduct an ablation study with respect to the density modeling contribution of the CVAE component in the full model.
Speciﬁcally, we disabled the CVAE component of the trajectron to obtain two model variants: For the ﬁrst model variant, we interchanged the CVAE with a GSNN component.
For the second model variant, we removed the latent variable component altogether.
We did not conduct any additional hyperparameter tuning, but rather trained our models with the parameters chosen for the trajectron.
[Table]
Test and validation log-likelihood results of the fully trained models are reported in Table 2.
Here, all log-likelihood expectations are exact, due to integration over the low-dimensional categorical latent variables.
We further report the runtime of our model variants compared to the full trajectron (in percent).
A paired t -test shows no signiﬁcant difference in the model variants with respect to their mean log-likelihoods over all ﬁve datasets.
However, the runtimes of the simpler models are signiﬁcantly lower when compared to the trajectron.
Notably, removing the CVAE altogether reduces the mean runtime by over 50% without any further optimization.
5.4 GVRNNs: modeling synthetic multiagent trajectories
In this section, we report an experiment regarding the modeling capacity of GVRNNs (Yeh et al., 2019) with respect to small scale synthetic multiagent trajectories.
Speciﬁcally, we compare the density estimation capabilities of simpler GRNN models (cf.
Sanchez-Gonzalez et al., 2018)
to
GVRNNs, again including a version in which we interchanged the CVM component with a GSNN.
Other than Yeh et al.
(2019), we experiment with both, Gaussian as well as GMM emission functions.
We sampled the dataset from trajectories of interacting reinforcement learning (RL) agents, trained within a conﬁned two-dimensional space.
Speciﬁcally, we chose a continuous hunter–prey scenario (referred to as simple tag ) from the multi-agent particle environment (Mordatch and Abbeel, 2018).
Within the scenario, we constrained the entities to two hunters and one prey agent only.
We deem this one of the simplest settings for the agents to show coordinated behaviour.
All agents were trained with the MADDPG (Lowe et al., 2017) RL algorithm.
To obtain more sensible trajectories, we introduced reward shaping via the L 2 distance between agents and added a further penalty to soft-force the agents within a square area.
Even though the MADDPG algorithm is in principle deterministic, data generation was conducted with training noise.
The dataset thus contains stochasticity.
Training, validation and test episodes were sampled with 25 time steps each.
We constructed 5 data splits, such that we trained models on 1000 episodes, validated on 2000 episodes and tested on 20000 episodes each.
See Figure 2 for an example of trajectories and interaction.
Like Yeh et al.
(2019), we used positional data as input and modeled the density of next step velocities.
We further used feature extractors on inputs and latent variables as in Chung et al. (2015).
For graph networks, we choose to use interaction networks (Battaglia et al., 2016), with learned embeddings for node and edge types.
We however used elementwise differences between nodes for the edge updates, inspired by the importance of the relative positions of agents.
This resulted in better performance during preliminary experiments.
We further use a GRU cell (Cho et al., 2014) for recurrence.
The overall architectures of our models were designed such that all models are comparable in parameter number to the trajectory models employed by Chung et al.
That is, models have around 8 million parameters, with variational models granted a bit more each.
For optimization we resorted to Adam and introduced gradient norm clipping (Pascanu et al., 2013).
A small learning rate decay (gamma=0.999) was added for stability and convergence, after an initial parameter sweep.
We further conducted a grid search over initial learning rates and weight decay parameters on the ﬁrst of our data splits, resulting in higher weight decay parameters for the models with Gaussian emission.
Log-likelihood results on test splits are shown in Figure 3a.
Again, all models with GMM emission resulted in much better density estimates than their Gaussian counterparts.
Indeed, only the differences with respect to emission functions are signiﬁcant according to paired t -tests.
Even though a KL scaling parameter was annealed (cf.
Sønderby et al., 2016)
from 0 to 1 over the ﬁrst 200 epochs for the
GVRNN models , the divergence remained very small compared to the log likelihood smaller than 10 − 5 and 10 − 7 respectively for Gaussian and GMM emission.
This led us to forgo importance sampling for these results as increases in the bound are deemed negligible.
[Figure]
Figure 3b shows mean runtimes per epoch for the different models.
As in the trajectron ablation study in Section 5.3, the models without conditional variational components are considerably faster.
GSNN models do not seem to provide any beneﬁt on the synthetic trajectories.
However, different from the reimplementation of the IAM-OnDB experiment in Section 5.2, we do not observe a performance gain with CVMs for Gaussian emission on this task either.
The density estimation capabilities of our models seem to only depend on the choice of emission function.
Discussion
In this section, we summarize our ﬁndings and relate the empirical results from Section 5 to the four hypotheses established in Sections 3 and 4.
Connections to further related literature are pointed out.
Regarding Hypothesis 1, which states that conditional autoencoding does not contribute towards density estimation for low-dimensional trajectories, we make the following observations: Statistical evidence gathered with respect to the contribution of the VRNN components for models with Gaussian emission on single trajectories allows us to reject the hypothesis in its broad scope.
With regards to this, our results support the ﬁndings in Chung et al.
Nevertheless, models without a conditional prior might be considered instead of the full VRNN model, as the performance of the
VRNN-I model suggests.
Noise regularization however, does not seem to be crucial for trajectory data, as GSNN models perform on par with deterministic models over all considered experiments.
With regards to Hypothesis 2, the IAM-OnDB experiment provides further evidence which implies that MDN emission dominates the contribution of conditional variational models for single trajectories:
Modeling the emission via GMMs seems to render the contribution of CVMs for single trajectory modeling statistically insigniﬁcant.
As such, this leads us to reject Hypothesis 1 only in the case of a weak decoder with single Gaussian emission.
These ﬁndings are in line with literature on VAEs having problems with autoregressive or/and powerful decoders (cf.
Chen et al., 2017; Zhao et al., 2019).
See Bowman et al.
(2015) and Fraccaro et al.
(2016) for early results on sequential data.
Evidence with respect to Hypothesis 3 is mixed: The above results on single trajectories seem to translate only in part to multiagent trajectories.
On the one hand, we do not observe any signiﬁcant contribution of CVMs in case of our experiments with GVRNNs with Gaussian emission.
With respect to that, the hypothesis seems false.
However, as this partial result is only supported on one dataset, it might not be conclusive.
Regarding Gaussian emission, Yeh et al.
(2019) reports better log-likelihood results on one of two datasets for comparable variational models (i.e. typed GVRNNs modeling interactions).
On the other hand, the signiﬁcant contribution of GMM emission does translate to graph architectures in our experiments.
For model performance, this is the important part.
Lastly, both, the experiments with trajectron variants as well as the experiment with GVRNNs on synthetic data do not allow us to reject Hypothesis 4.
It thus stays likely, that graph architectures modeling multiagent trajectories with CVM components do not generally result in better log-likelihoods than alike models without a CVM component.
Given the empirical evidence, we conjecture this to be especially true in the case of architectures which make use of MDN emission functions.
Broader Impact
This paper revises possibly misleading results in prior work with respect to conditional variational models such as CVAEs and VRNNs used for (multiagent) trajectory modeling.
Both CVAEs and VRNNs are highly inﬂuential models (measured in citation), while multiagent trajectory data receives a great deal of attention in current research.
At the same time, proposed models are becoming increasingly complex, which renders attribution of model performance to individual components more difﬁcult.
This paper corrects expectations towards CVMs for trajectory modeling, identiﬁes the proper inclusion of CVM components for trajectory modeling as open research question and encourages practitioners to rather run models without variational components for the time being.
We thus believe that, both, researchers and practitioners may beneﬁt from our contribution.
Acknowledgments and Disclosure of Funding
We would like to thank Boris Ivanovic for his helpful insights and comments with respect to our ablation study on the trajectron model.
Further, we would like to thank Johannes Höhne for being essential in driving forward the research collaboration between SAP SE and the Leuphana University of Lüneburg, of which this paper is one result.
In addition, Yannick Rudolph would like to thank
Johannes Höhne for his fundamental supervision at SAP SE, as well as Marius Lehne and Christian
Reisswig at SAP SE for their help and support while writing this paper.