Learning interacting dynamical systems with latent Gaussian process ODEs
Abstract
We study uncertainty-aware modeling of continuous-time dynamics of interacting objects.
We introduce a new model that decomposes independent dynamics of single objects accurately from their interactions.
By employing latent Gaussian process ordinary differential equations, our model infers both independent dynamics and their interactions with reliable uncertainty estimates.
In our formulation, each object is represented as a graph node and interactions are modeled by accumulating the messages coming from neighboring objects.
We show that efﬁcient inference of such a complex network of variables is possible with modern variational sparse Gaussian process inference techniques.
We empirically demonstrate that our model improves the reliability of long-term predictions over neural network based alternatives and it successfully handles missing dynamic or static information.
Furthermore, we observe that only our model can successfully encapsulate independent dynamics and interaction information in distinct functions and show the beneﬁt from this disentanglement in extrapolation scenarios.
Introduction
A broad spectrum of dynamical systems consists of multiple interacting objects.
Since their interplay is typically a priori unknown, learning interaction dynamics of objects from data has become an emerging ﬁeld in dynamical systems [1, 2, 3].
The ever-growing interest in interaction modeling is due to the diversity of real-world applications such as autonomous driving [4], physical simulators [5] and human-robot interactions [6].
Standard time-series algorithms or deep learning approaches (e g. recurrent neural networks), that have been designed for single-object systems, do not scale to a large number of interacting objects since they do not exploit the structural information of the data.
In recent years, graph neural networks (GNNs) have emerged as a promising tool for interactive systems, where objects are represented as graph nodes.
State-of-the-art methods learn interactions by sending messages between objects in form of multi-layer perceptrons [1] or attention modules [7].
These methods yield highly ﬂexible function approximators that achieve accurate predictions when trained on large-scale datasets.
However, their predictions come without calibrated uncertainties, hindering their reliable implementation for uncertainty-aware applications.
In contrast, Gaussian processes (GPs) are well-known for providing calibrated uncertainty estimates.
They have been successfully employed on discrete time-series data [8, 9, 10] and, more recently, to continuous-time generalizations of these methods [11, 12, 13].
Importantly, none of the these works adresses dynamical models for interacting systems.
While it is possible to study each object in isolation, ignoring the interaction effects might lead to inaccurate predictions.
In this work, we address the shortcomings of both model families by presenting an uncertainty-aware continuous-time dynamical model for interacting objects.
Our formulation decomposes the dynamics into independent (autonomous) and interaction dynamics.
While the former governs the motion of an object in isolation, the latter describes the effects that result from interactions with neighboring objects.
For successful uncertainty characterization, we propose to infer the unknown independent and interaction dynamics by two distinct GPs.
We demonstrate that having a function-level GP prior on the individual dynamics components is the key to successfully disentangling these dynamics, which in turn allows for interpretable predictions and leads to improved extrapolation behavior.
We employ latent Gaussian process ordinary differential equations (GP-ODEs) for dynamics learning, allowing to learn complex relationships between interacting objects without the need of having access to fully observed systems.
Thanks to recently proposed decoupled sampling scheme [14], the computational complexity of our model scales linearly with the number of time points at which the
ODE system is evaluated.
As a result, our algorithm scales gracefully to datasets with thousands of sequences.
To demonstrate the beneﬁts of our framework, we exhaustively test our method on a wide range of scenarios varying in function complexity, signal-to-noise ratio, and system observability.
Our model consistently outperforms non-interacting dynamical systems and alternative function approximators such as deterministic/Bayesian neural networks.
Background
In this section, we give background on continuous-time systems and Gaussian processes.
Both together form the backbone of our uncertainty-aware framework for interactive dynamical systems.
2.1 Continuous-time Dynamical Systems
Continuous-time dynamical systems are often expressed using differential functions ˙ x ( t ) ≡ ddt x ( t ) ≡ f ( x ( t )) , where x ( t ) ∈ R D represents the state of an ODE system at time t and and f : R D (cid:55)→ R D is the time differential function governing the dynamics evolution.
The state solution x ( t 1 ) at an arbitrary time t 1 is characterized by the initial value at time point t 0 and the differential function:
[Equation]
Existing work aims to approximate the unknown differential by Gaussian processes [15, 11] or neural networks [16].
These methods have shown to accurately capture the dynamics and outperform their discrete-time counterparts in a wide range of applications such as time series forecasting [17], classiﬁcation [18] or reinforcement learning [19].
Furthermore, ODE models allow to easily inject domain knowledge into the system, enabling interpretable and ﬂexible hybrid models [20, 21, 22].
2.2 Gaussian Processes
Gaussian processes (GPs) deﬁne priors over functions [23]:
[Equation]
where f : R D → R maps D -dimensional inputs into one-dimensional outputs.
GPs are fully speciﬁed in terms of their mean and their covariance:
[Equation]
where µ : R D → R is the mean and k : R D × R D
→ R is the kernel function.
GPs can be treated as an extension of a multivariate normal distribution to inﬁnitely many dimensions, where any ﬁxed set of inputs X ∈ R N × D follows the Gaussian distribution
[Equation]
with µ X being the mean function evaluated at inputs X , and K XX the kernel function evaluated at all input pairs in X .
While GPs provide a natural mechanism to handle uncertainties, their computational complexity grows cubically with the number of input points.
This problem is often tackled by sparse
GPs , which rely on augmenting the GP with M inducing inputs Z =
[ z T 1 , . . .
, z TM ] T ∈ R M × D and corresponding output variables u
= [ u 1 , . .
, u M ] T ∈ R M × 1 with u m ≡ f ( z m ) [24, 25].
Assuming a zero-mean prior, as it is common in the literature, the conditional distribution over f ( X ) follows the Gaussian process:
[Equation]
where K ZZ is the covariance between all inducing points Z , and K XZ between the input points X and the inducing points Z .
The inducing points can thereby be interpreted as a compressed version of the training data in which the number of inducing points M acts as a trade-off parameter between the goodness of the approximation and scalability.
In this work, we employ the squared exponential kernel k ( [Equation] [Equation]
=
[Equation] [Table]
where x d denotes the d -th entry of the input x , σ 2 is the output variance and (cid:96) d is the dimension-wise lengthscale parameter.
Interacting Dynamical Systems with Latent Gaussian Process ODEs
In Sec. 3.1, we describe our continuous-time formulation for systems of interacting objects.
It decomposes the dynamics into independent kinematics and an interaction component that takes the interactions to neighboring objects into account.
Placing a GP prior over the individual components is essential in order to arrive at (i) calibrated uncertainty estimates and (ii) disentangled representations as we later on also verify in our experiments.
In Sec. 3.2, we embed the GP dynamics into a latent space that can accomodate missing static or dynamic information.
Both together allows the application of our continuous-time formulation to a wide range of scenarios and allows for learning interpretable dynamics.
We conclude this section by our variational inference framework (Sec 3.3) based on sampling functions from GP posteriors.
3.1 Interacting Dynamical Systems
We assume a dataset of P sequences Y
=
{ Y 1 , . . .
, Y P } , where each sequence Y ≡ Y 1: N ≡ y 1: A 1: N is composed of measurements of A objects at time points T
= { t 1 , . . t N } .
Without loss of generality, we assume that the measurement y a ( t n ) ≡ y an ∈ R O is related to the physical properties of object a , such as position and velocity, which can routinely be measured by standard sensors.
The dynamic state of object a at any arbitrary time t is denoted by a latent vector
h a ( t ) ∈
R D , which does not necessarily live in the same space as the observations.
We furthermore assume that each object a is associated with a global feature vector c ∈ R C , which corresponds to the static attributes that remain constant over time.
Finally, we denote the concatenation of all states by
H ( t ) =
[ h 1 ( t ) , . . .
h A ( t )]
∈ R
A × D and all globals by C
=
[ c 1 , . . .
c A ] ∈ R A × C .
In the following, we propose to disentangle the complex continuous-time dynamics into independent kinematics and interaction differentials.
More concretely [Equation] we introduce the following dynamics:
[Equation]
where N a denotes the set of neighbors of object a in a given graph.
The ﬁrst function f s : R D + C
(cid:55)→ R D models the independent (autonomous) effects, which speciﬁes how the object would behave without any interactions.
The second function f b : R 2 D +2 C (cid:55)→ R D models the interactions by accumulating messages coming from all neighboring objects.
Since message accumulation is the de-facto choice in interaction modeling [2, 26], the additive form of the differential function is a very generic inductive bias.
Furthermore, our formulation models interactions between pairs of objects explicitly via the differential equation (Eq.
Higher-order interactions are taken into account via the continuous-time formulation that allows information to propagate through the complete graph via local messages over time such that the state of the object h an can also depend on objects that are not directly connected in the graph.
In contrast to discrete formulations, for which the message passing speed is limited by the sampling rate, our continuous-time formulation enjoys instant propagation of information across objects.
Finally, please see Section for an investigation of our interaction component under a kernel perspective.
RemarkIn Sec A.1 we demonstrate two straightforward extensions of our formulation with non-linear message accumulation, which we empirically show to have no improvement over our proposed, simpler formulation.
Remark-
Previous GP-based ODE methods [15, 11, 12] assume a black-box approximation to the unknown system ddt H ( t ) =
f ( H ( t )) whereas our state representation gracefully scales to a varying number of objects and also include global features.
3.2 Probabilistic Generative Model
Real-world data of interactive systems necessitates embedding the dynamics into a latent space in order to allow for missing information; the observations may contain only partial estimates of the states and the globals C might not be observed at all.
We account for those circumstances by treating the states H and the globals C as latent variables, leading to following generative model (see Figure 1):
[Equation]
where we introduced a standard Gaussian prior over the initial latent state, and assumed that the data likelihood decomposes across time and objects.
We furthermore model unknown functions f s and f b under independent vector-valued GP priors.
In our experiments, we further set p ( y an | h an ) =
N ( y an | Bh an , diag ( σ 2 e )) , where B ∈ R O × D maps from the latent to the observational space and σ 2 e ∈ R O + is the noise variance.
We further ﬁx
B =
[ I , 0 ] where I ∈
R O × O , 0 ∈ R O,D − O , in order to arrive at an interpretable latent space in which the ﬁrst dimensions correspond to the observables.
This assumption is fairly standard in the GP state-space model literature since more complex emission models can be subsumed in the transition model without reducing the model complexity [27].
Modeling partially observed systems often leads to non-identiﬁability issues that hamper optimization and ultimately lead to deteriorated generalization performance.
One way to counteract this behavior is to inject prior physical knowledge into the system by decomposing the state space of each object h a ( t ) ≡ [ s a ( t ) , v a ( t )] into position s a ( t ) and velocity v a ( t ) components [28].
Using elementary physics, the differential function has then the form of d dt h a ( t ) = (cid:2) v a ( t ) , d dt v a ( t ) (cid:3) with ddt v a ( t ) = f s ( h a ( t ) , c a ) + (cid:80)
[Table]
Remark Unlike previous work, our formulation incorporates global features c that modulate the dynamics.
In many applications such as control engineering and reinforcement learning, the dynamics are modulated by external control signals [29] which can also be incorporated into our framework.
The solid curves are the groundtruth trajectories and the shaded regions denote the predicted 95% conﬁdence intervals.
I-GPODE (ours) yields better calibrated long-term predictions than I-NODE.
Additional results can be found in Figure 5.
3.3 Variational Inference
Next, we derive an efﬁcient approximate inference scheme that provides a high level of accuracy and model ﬂexibility.
In the above formulation, the model unknowns are the initial values H 1 ≡ h 1: A 1 , global variables C
= c 1: A and the differentials f s and f b .
Since the exact posterior p ( f s , f b , H 1 C |Y ) in non-linear ODE models is intractable, we opt for stochastic variational inference [30].
We ﬁrst describe the form of the approximate posterior and then discuss how to optimize its parameters.
Variational family
Similarly to previous work [16], we resort to amortized inference for initial values and global variables, H 1 ∼ q φ ( H 1 | Y 1: N ) and , C ∼ q ρ ( C | Y 1: N ) , where φ is the parameters of a neural network encoder that outputs a Gaussian distribution with diagonal covariance.
For the unknown time differentials f = { f s , f b } , we follow the standard sparse GP approximation in which each output dimension d ∈ [1 , D ] has its own independent set of inducing values U s,d , U b,d ∈ R D and kernel output variances σ 2 s,d , σ 2 b,d ∈ R + .
Inducing locations { Z s , Z d } and lengthscales { (cid:96) s , (cid:96) b } are shared across output dimensions.
Denoting the collection of inducing points by U
= { U s , U b } , we arrive at the mean-ﬁeld variational posterior:
[Equation]
where the means { m s,d , m b,d } Dd =1 and the covariances { S s,d , S b,d } Dd =1 are free variational parame- ters.
Putting everything together, our variational approximation becomes as follows [31]:
[Equation]
where p ( f s | U s ) and p ( f b | U b ) follow Eq.
Our variational family makes two assumptions that are fairly standard in the (deep) GP literature (e.g. [32]): (i) we apply the same independence assumptions in the approximate posterior as in the prior resulting in a mean-ﬁeld solution, and (ii) we assume that the inducing outputs U capture the sufﬁcient statistics of the training data allowing the use of the prior p ( f | U ) in the approximate posterior.
Variational bound
We then seek to optimize the parameters of the approximate posterior q by maximizing a lower bound to the evidence [30]:
[Equation]
In the following, we detail its computation for a single data instance Y 1: N , omitting its generalization to multiple sequences for the sake of better readability,
[Equation] [Equation]
where KL denotes the Kullback–Leibler divergence.
Likelihood computation via decoupled sampling from GP posteriors
Computing the conditional log-likelihood log p
( Y 1: N | H 1 , C , f , U ) entails a forward pass in time (Eq.
(5)) which can be done with any standard ODE solver.
The difﬁculty lies in marginalizing over the approximate posterior of the initial latent states q ( H 1 ) , global variables q ( C ) , and the GP functions q ( f , U ) .
Each marginalization step alone is already analytically intractable, let alone their combination.
We therefore opt for Monte Carlo integration which gives us an unbiased estimate of the expected log-likelihood.
We start by drawing L samples from the approximate posteriors
[Equation]
where l denotes the sample index and f ( l ) ( · ) is a function drawn from the sparse GP posterior.
Sampling from the GP posterior naively, scales cubically with the number of data points.
Moreover, since we do not know a-priori on which points the ODE solver evaluates the function, we would have to sequentially draw points from the posterior.
While this can still be done cubically in time by performing low-rank updates, it often leads to numerical instabilities for small step sizes.
To overcome this challenge, we resort to the decoupled sampling scheme proposed in [14], where we ﬁrst draw the prior samples from a set of random Fourier features and then update them using Matheron’s rule to obtain posterior samples.
After having sampled the quadruple via Eq.
(7), we can compute the trajectory H ( l ) 1: N deterministically by forward integration of Eq.
(5) and the Monte Carlo estimate of the log-likelihood becomes
[Equation]
where the log-likelihood term decomposes between objects and between time points, enabling doubly stochastic variational inference [33].
KL regularizer
The prior distributions over the inducing variables follow the Gaussian process
[Equation]
) with p ( U s ) and p ( U b ) following the equivalent of Eq.
Since we furthermore assumed a standard Gaussian prior p ( H 1 ) =
N ( 0 , I ) and p ( C 1) = N ( 0 , I ) with suitable dimensions for the initial values, all KL terms can be computed in closed form. Computational complexity Evaluating the differential function costs O
( M 2 ( A + I )) for a graph with A objects, I interactions and M inducing points. The term O ( M 2 A ) stems from the independent kinematics, the term O ( M 2 I ) from the interaction effects. As a consequence, forward integration using RK4 solver takes O ( TM 2 ( A + I )) time, where T is the number of time points.
Our approach enhances the capabilities of the GPODE model family in the following three aspects: Explicit modeling of interactions Standard GPODEs model interactions by allowing the time differential to take the whole state vector H ( t ) as input and to learn one independent GP for each latent dimension.
This shared latent space assumption entails three major drawbacks: (i) obligation to ﬁx the object count as a model hyperparameter, (ii) dependency of the learned model on a predeﬁned ordering of the objects in the scene, (iii) inevitable growth of the latent dimensionality proportional to the object count.
The latter sets a severe bottleneck especially for GP modeling as the performance of many kernel functions in widespread use are highly sensitive to input dimensionality.
(For example, on the bouncing ball dataset with N a balls and D latent states per object, GPODE needs to learn a latent function with N a D -dimensional inputs and outputs.)
In contrast, I-GPODE needs only to learn two functions, the independent kinematics f s and the interaction function f b , whose input sizes scale independently of N a .
Our Table 2 indicates that learning interaction dynamics without the strong inductive bias of our model is difﬁcult and the GPODE model chooses to stay at the prior instead leading to deteriorated MSEs and ELLs.
Disentangled representation We infer object-speciﬁc latent variables that modulate the dynamics, which allows our model to disentangle the dynamics evolution (time differential function) from static object properties (e.g. charge information).
The interpretability and likely physical correspondence of the disentangled factors have the potential to facilitate the use of our approach in transfer learning, as well as explainable AI applications.
Inference of latent state dynamics
We perform the learning in a latent space, where the initial value of a trajectory is given by an encoder, leading to the following two advantages: First, the dynamical system and the data points may live in different spaces, which facilitates learning from
[Table]
high-dimensional sequences.
Second, Bayesian modeling of state dynamics on a latent space enables reliable quantiﬁcation and principled treatment of sources of uncertainty, such as imprecision of modeling assumptions, approximation error, and measurement noise.
GPs for modeling ODE systems have been studied in a number of publications (e.g. [11, 12, 13, 15, 36]).
With the notable exception of [13], they only consider systems in which the dynamics are deﬁned in the data space.
The work that is closest to ours from a technical perspective is [11, 12] that also employ decoupled sampling in order to compute consistent trajectories during inference.
We are not aware that interacting dynamical systems under a GP prior have been studied previously, either in the continuous or in the discrete time setting.
NODEs for dynamical data
Since the debut of Neural ODEs [16], much progress has been made on how to apply neural networks for modeling sequential data in the continuous time domain.
These works [17, 37] typically assume a black-box approximation to unknown ODE systems ddt H ( t ) =
f ( H ( t )) , where f is a deterministic neural network, and the latent space is separated from the observational space using a neural network decoder.
A subtle difference to our approach is that we decided to use a linear mapping instead.
However, in case that the outputs are high-dimensional, e.g. video data, this can be easily changed.
Few works have integrated function uncertainty into Neural
ODEs by putting a prior over the neural network weights [28, 38].
To the best of our knowledge, none of these works addressed interacting systems.
However, we still compare against an interactive adaptation of these approaches in our experiments.
Modeling interacting dynamics
Interacting dynamical systems have ﬁrst been considered for discrete time-step models and for the deterministic setting [1, 26, 39].
Many of these discrete formulations can be transferred to the continuous case as shown in [3, 34, 40].
This also holds true for our approach, for which the discretized version of the dynamics (Eq.
(3), (4)) can be easily cast into one of the existing frameworks (e.g [39]).
These works have also been extended to the probabilistic context using a variational auto-encoder [2, 35].
The hidden variables are thereby used to either encode the initial latent states or static information.
In contrast to our work, none of these approaches allow for function uncertainty in the dynamics.
Finally, [41] proposes a symbolic physics framework, differing from our method in its search-based ﬁxed grammar describing the dynamics.
Finally, we provide a summary of related techniques and derived comparison partners for our experiments in Table 1.
Experiments
We compare our approach against state-of-the-art methods in a large number of scenarios that differ in function complexity, signal-to-noise ratio and observability of the system.
Our empirical ﬁndings suggest that our model leads to improved reliability of long-term predictions while being able to successfully encapsulate autonomous dynamics from interactions effects.
In all our experiments, we use the RK4 ODE solver and ACA library [42] for integration and gradient computation.
Due to the space limit, we refer to the Supplementary Material for more detailed information about the experimental setup and comparison methods (see also Table 1 for an overview).
Our PyTorch [43] implementation can be found in https://github.com/boschresearch/iGPODE (GNU AGPL v.3.0 license).
5.1 Experiment Details
Datasets We illustrate the performance of our model on two benchmark datasets: bouncing balls [44] and charges [2].
These datasets involve freely moving N a = 3 balls that collide with each other and N a = 5 particles that carry randomly assigned positive or negative charges leading to attraction or repulsion, respectively.
All simulations are performed in frictionless, square boxes in 2 D .
We generate 100 bouncing balls training sequences with different levels of Gaussian perturbations to simulate measurement noise (see Supplementary Section A.3 for details).
Since the charges dataset requires inferring the charge information, we use 10 k train sequences without observation noise as in [2] and similarly use 500 training sequences when velocity information is missing.
Partial observability
Prior works typically evaluate their methods on datasets in which the observations consist of position and velocity [1, 2].
However, having access to the full state of an object is for many real-world problems unrealistic, e.g. many tracking devices can measure positions, but cannot measure velocity or acceleration.
To test the model performance in such scenarios, we consider an additional bouncing balls dataset in which the velocities are not observed.
On the charges dataset, we assume that position and velocity are observed, but treat the charge information as missing.
Reported metrics We quantify the model ﬁt by computing the expected log-likelihood ( ELL ) of the data points under the predictive distribution.
Further, we report the mean squared error ( MSE ) between ground truth and predictions over all predicted time steps and over all objects (see Supplementary
Section A.3 for the exact deﬁnitions).
Each experiment is repeated ﬁve times and we report the mean and standard deviation of both metrics on test sequences.
5.2 Empirical Findings
Due to our latent variable construction, ODE state dimensionality D could be arbitrary even though the observations are four-dimensional.
To choose an appropriate state dimensionality, we study if the model performance can be improved by augmenting the state space with auxiliary dimensions ( D > 4 ).
Table 8 shows that increasing the model ﬂexibility beyond need leads to overﬁtting as we observe lower training but signiﬁcantly higher test error.
Consequently, if not stated otherwise, we use a four-dimensional latent space for each object that corresponds to position and velocity in x and y coordinates and observations consist of their noisy versions.
Next, we discuss the main ﬁndings.
5.2.1 Interacting dynamics are superior over standard formulation
[Table]
We consider three bouncing ball datasets with varying noise levels to reﬂect different levels of problem difﬁculties.
To demonstrate the merits of our decomposed formulation in Eq.
(4), we compare it against a standard, GP-based noninteracting dynamical model (GPODE) in which the time differential takes the whole state vector H ( t ) as input [11].
As shown in Table 2, our interaction model consistently outperforms its standard counterpart irrespective of the noise level and function approximator.
We also note that the results are consistent when we replace the GP approximation with deterministic and
Bayesian neural networks (NNs), which indicates the robustness of the inductive bias.
(see Table 12).
GP approximation yields more calibrated uncertainties
[Table]
Next [Table] we compare our GP interaction model (I-GPODE) against neural network variants with/without uncertainty (I-NODE, I-BNODE), obtained by replacing GPs with NNs and Bayesian
NNs.
The ﬁndings in Table 3 strongly indicate that I-GPODE learns better calibrated uncertainties, measured by ELL , than I-BNODE
and I-NODE (see Figure 2 and 5 for visual illustrations).
In terms of MSE , we observe that I-NODE tends to attain the smallest values.
However, increasing noise levels deteriorates the performance of I-NODE and eventually the results become comparable to
I-GPODE.
The results stay consistent when we repeat the experiment with two balls (see Table 11). 5.2.3 GP approximation enables disentangled function learning In the following
[Table]
we study whether the estimated functions can disentangle independent kinematics from interaction effects.
We train I-GPODE, I-NODE and
I-BNODE on a dataset with three balls and evaluate the trained independent dynamics function f
s on a test dataset with one ball (since the test dataset incorporates a single object, the dynamics do not involve the interaction function f b ).
Three test sequences as well as the independent dynamics function predictions are illustrated in Figure 3 (see Tablefor a quantitative comparison and Figure 4 for additional illustrations).
We observe that I-NODE predictions tend to deviate from the test trajectory more quickly compared to I-GPODE predictions.
We conjecture that this behaviour is because neural networks are overﬂexible and thus the learned functions may not necessarily decompose independent kinematics from interaction effects whereas the function-level regularization of I-GPODE helps with disentanglement.
5.2.4 Structured dynamics improve latent dynamics learning
[Table]
S 21 . 8 ± 0 .− 115 ±
In our last bouncing ball experiment, we move to a setting in which the velocities are no longer observed.
First, we keep the velocities as latent states and contrast two variants of our model, i.e. with structured latent space (I-GPODE-L-S) and with unstructured latent space (IGPODE-L).
As shown in Table 4, injecting strong prior knowledge helps in this challenging setting in which the ﬁrst order model clearly fails (see also Tablefor training results).
Finally, we compare
I-GPODE-L-S with I-GPODE, which drops the velocity component from the latent space (hence learning the dynamics in the data space).
As demonstrated in Table 4, I-GPODE is clearly outper- formed by I-GPODE-L-S. It can thus be suggested that our latent variable construction is necessary in presence of missing states.
5.2.5 Global latent variables boost performance
[Table]
In the ﬁnal part of the experiments, we consider a more challenging dataset of charged particles.
Since the dynamics are modulated by unknown charges, we turn to our global latent variable formulation.
In other words, our learning task becomes simultaneously inferring the functions f s and f b , the initial latent states
H 1 , as well as a latent variable c a ∈ R associated with each observed trajectory y a 1: N .
To form the upper and lower performance bounds, we include two baselines in which the charges are either observed or completely dropped from the model description.
The results are shown in Table 5.
We notice that the structured state space formulation boosts the performance of I-GPODE.
However, the effect is less pronounced compared to the previous setting in which dynamic information is missing.
See Table 14 for more results with different global variable encoders.
Discussion
We have presented the ﬁrst uncertainty-aware model for continuous-time interacting dynamical systems.
By embedding the dynamics into a latent space, our formulation yields a ﬂexible model family that can be applied to a variety of different scenarios.
In our experiments, we found that our approach leads to improved disentanglement and superior calibration of long-term predictions.
Exploring useful applications of our disentangled representation is also an interesting direction for future research.
Accurate identiﬁcation of independent kinematics and interaction effects could enable useful downstream functionalities.
For instance, one can perform algorithmic recourse [45] by counterfactual interventions at the object or interaction level.
Modeling limitations The capacity of our GP is limited by the choice of the kernel, e.g., the RBF kernel assumes that the dynamics are stationary. While our model formulation can be combined with arbitrary kernel functions and it is possible to increase the kernel expressiveness, e.g by building composite kernels or coming up with hand-crafted features, these approaches are often time-consuming and lead to highly parameterized kernels that are difﬁcult to learn. Approximation errors Our posterior inference scheme is inaccurate due to our variational framework, approximation errors that accumulate in time during future prediction, and numerical errors caused by numerical integration of ideally continuous dynamics and its solution.
Broader impact In our work, we propose a methodological contribution which is blind to speciﬁc data distributions.
Its potential and unforeseeable side-effects in fairness-sensitive or safety-critical applications need to be investigated in a dedicated study. Acknowledgements
The Bosch group is carbon neutral. Administration, manufacturing and research actvities do not longer leave a carbon footprint. This also includes GPU clusters on which the experiments have been performed.
We would like to thank Jakob Lindinger and Michael Herman for discussions and proofreading. A Appendix A.1 Nonlinear Message Accumulations Our formulation accumulates the messages coming from neighbors by computing their sum (Eq. 4). Although the experimental ﬁndings indicate that our seemingly simple construction successfully learns the underlying dynamics, we introduce two straightforward extension of our framework.
First, one can learn an additional non-linear function f c that takes all the incoming messages as input and generates the time differential as output:
[Equation]
For notational convenience, we drop the neighboring graph from our write-up. In practice, f c would only receive the messages coming from the neighbors.
Since the messages interact in a non-linear way, this construction no longer disentangles the independent kinematics from the interactions. Next, we introduce another construction in which the neighboring messages are weighted via a nonlinear function f w :
[Equation]
When tested on bouncing ball datasets, this model achieved lower training and slightly higher test error (indicating overﬁtting). We leave further analysis of this new construction as an interesting future work. A.2 Induced Kernel on Interaction Term
Next, we study our interaction component under a kernel perspective. In our formulation, we achieved permutation invariance across neighboring objects by aggregating all in-coming messages via the sum function (that is order invariant), similarly to what is done in standard graph neural network architectures [1]. It is interesting to see that the permutation invariance of our model formulation can also be derived from a kernel perspective. Seminal work on invariant kernels has been done by [46] and [47] who showed that a kernel is invariant under a ﬁnite set of input transformations, e.g. permutations, if the kernel is invariant when transforming its arguments.
In our work, the Gaussian process prior on f b induces a Gaussian process prior on the interaction term with covariance,
[Equation]
which enforces the invariance by summing over all input combinations.
Using a double sum is a common strategy for creating invariant kernels (see [48, 49] for a more in-depth discussion).
A.3 Experiment Details
Interaction function parameterization In all our experiments, we assume a fully connected object graph and also parameterize the interaction function f b with the difference between object positions instead of absolute positions ( f b also takes the velocities and the global latent variables as input).
Since the interactions are typically expressed in terms of distance, injecting an inductive bias of this sort helps to increase the performance as validated by our experiments.
Datasets
To generate our datasets, we use the ofﬁcial implementations provided in [44, 2].
To ensure the accuracy of all numerical simulations in the respective code, we reduce the simulation step size.
The dataset speciﬁcs are given in Table 6.
Note that the noise values are proportional to the lower and upper limits of the position and velocity observations.
[Table]
Dynamics approximations and hyper-parameter selection Our proposed I-GPODE method, in which the unknown independent kinematics and interaction functions are approximated with GPs, is compared with I-NODE and I-BNODE baselines in the bouncing balls experiment.
To obtain the baselines, we simply replaced our GP approximation with multi-layer perceptrons (MLPs).
We consider the standard, weight-space mean-ﬁeld variational posterior for the BNN as done in [28].
In turn, the optimized dynamics parameters become the weights for I-NODE and the variational parameters for I-BNODE.
We perform an exhaustive comparison of hyper-parameters in different settings.
In particular, for the independent kinematics function f s , we consider MLPs with two hidden layers and
N = 64 / 128 / 256 / 512 hidden neurons, and sparse GPs with M = 100 / 250 / 500 inducing points.
For the interaction function f b , we test with N = 128 / 256 / 512 hidden neurons and M = 100 / 250 / 500 / 1000 inducing points. We furthermore search the best activation function among elu / relu / softplus / tanh / swish / lip-swish activations and consider a diagonal or full lower-diagonal approximation to the covariance matrix (of the variational posterior). We test all hyper-parameter conﬁgurations on three validation datasets with varying noise levels.
We found out that the softplus activation and diagonal covariance approximation consistently minimize the reported metrics on validation datasets. Other hyperparameters used in our experiments are reported in Table 7.
Note that the number of parameters of the simpler, non-interacting models are approximately matched with the corresponding interacting model.
[Table]
N
[Table]
N
[Table]
GPODE M =- NODE N =-
BNODE N =-
Initial value encoder Inspired by previous work [17, 11], we infer the initial position and velocity variables using a RNN-based encoder architecture. Our encoder with GRU cells processes the ﬁrst ﬁve observations in backward direction: Y 5 → Y 1 . The encoder output z 1 ∈ R 10 is mapped into position and velocity initial value distributions via two separate MLPs that take the non-overlapping 5-dimensional chunks of z 1 as input. Each MLP has the same architecture (one hidden layer,neurons, ReLU activations).
The model performance is somewhat robust against these encoder hyperparameters as validated by further comparisons. We ﬁnally note that the same encoder architecture is used for GP, NN and BNN-based models. Latent variable encoder To infer the latent variables in the charges experiment, we again utilize an RNN-based encoder.
Similar to the encoder used in [2], our architecture takes the ﬁrst 49 observations as input, i.e., the global latent variable c a associated with object a is extracted from all available observations y 1: A 1:49 . Since the overall performance crucially depends on the hyperparameter choices unlike the initial value extraction task, we consider two sets of encoders: a “large” encoder with z 1 ∈ R 100 and an MLP with 100 neurons as well as a “small” encoder with z 1 ∈ R 25 and an MLP with 50 neurons. We furthermore perform comparisons with relu and elu activation functions for the MLP. The results in the main paper are obtained with the “small” encoder with elu activation, which yields the best or runner-up performance across all settings.
Training details All model variants are trained with the Adam optimizer [50] with learning rates 5e-4, 5e-4 and 1e-4 for GP, NN and BNN-based models. We perform an incremental optimization scheme with three rounds, where randomly chosen 100 subsequences of length 5, 16, and 33 are used for training. We perform 25000 , 12500 and 12500 optimization iterations in each round.
Training each model respectively takes 9, 3 and 12 hours on NVIDIA Tesla V100 32GB. Finally, as proposed in [28], we stabilize the BNN learning by weighting the KL term KL [ q ( W ) || p ( W )] resulting from the BNN with a constant factor β = D/ |W| in order to counter-balance the penalties on latent variables h a ∈ R D and neural network weights W ∈ R |W| . Reported metrics For a single trajectory y 1: A 1: N , the MSE and ELL metrics are computed as follows:
[Equation]
with ˆ y a ( l ) n being the l -th Monte Carlo prediction of object a at time point t n and
[Equation]
Finally, we report averages of the test statistics over all trajectories.
A.4 Additional Results Latent neural ODE comparisons
The ODE systems in our framework as well as the baseline models may be composed of positions, velocities and global latent variables.
On the other hand, alternative black-box approaches [17, 28] typically consider a latent ODE system with arbitrary dimensionality and a VAE embedding between the observed and latent space.
We compare these two modeling paradigms on the bouncing ball dataset.
Since the reference methods are based on neural ODEs, we only consider neural network approximations for the differential functions. The results are presented in Table 8. In agreement with other comparisons, interaction models outperform their simpler, non-interacting counterparts.
Also, latent ODE models tend to reduce the training error and increase the test error, which is a strong indicator of overﬁtting.
[Table] [Table] [Table] [Table]
[Table]
This repeats paper
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]