N EURAL ODE P ROCESSES
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural Ordinary Differential Equations (NODEs) use a neural network to model the instantaneous rate of change in the state of a system.
However, despite their apparent suitability for dynamics-governed time-series, NODEs present a few disadvantages.
First, they are unable to adapt to incoming data-points, a fundamental requirement for real-time applications imposed by the natural direction of time.
Second, time-series are often composed of a sparse set of measurements, which could be explained by many possible underlying dynamics.
NODEs do not capture this uncertainty.
To this end, we introduce Neural ODE Processes (NDPs), a new class of stochastic processes determined by a distribution over Neural ODEs.
By maintaining an adaptive data-dependent distribution over the underlying ODE, we show that our model can successfully capture the dynamics of low-dimensional systems from just a few data-points.
At the same time, we demonstrate that NDPs scale up to challenging high-dimensional time-series with unknown latent dynamics such as rotating MNIST digits.
I NTRODUCTION
Many time-series that arise in the natural world, such as the state of a harmonic oscillator, the populations in an ecological network or the spread of a disease, are the product of some underlying dynamics.
Sometimes, as in the case of a video of a swinging pendulum, these dynamics are latent and do not manifest directly in the observation space (pixel space, in this case).
Neural Ordinary
Differential Equations (NODEs) (Chen et al., 2018), which use a neural network to parametrise the velocity of an ODE, have become a natural choice for capturing the dynamics of such time-series (C¸a˘gatay Yıldız et al., 2019; Rubanova et al., 2019;
Norcliffe et al., 2020; Kidger et al., 2020; Morrill et al., 2020).
However, despite their fundamental connection to dynamics-governed time-series, NODEs present certain limitations that hinder their adoption in these settings.
Firstly, NODEs cannot adjust predictions as more data is collected without retraining the model.
This ability is particularly important for real-time applications, where it is desirable that models adapt to incoming data points as time passes and more data is collected.
Secondly, without a larger number of regularly spaced measurements, there is usually a range of plausible underlying dynamics that can explain the data.
NODEs, however, only offer a single maximum likelihood prediction and are unable to learn stochastic dynamics.
As many real-world time-series are comprised of sparse sets of measurements, often irregularly sampled, the model can fail to represent the diversity of suitable solutions.
To address these limitations, we introduce Neural ODE Processes (NDPs), a new class of stochastic processes governed by stochastic data-adaptive dynamics.
Our probabilistic Neural ODE formulation relies on and extends the framework provided by Neural Processes (NPs), and runs parallel to other attempts to incorporate application-speciﬁc inductive biases in this class of models such as
Attentive NPs (Kim et al., 2019), ConvCNPs (Gordon et al., 2019), and MPNPs (Day et al., 2020).
We demonstrate that NDPs can adaptively capture many potential dynamics of low-dimensional systems when faced with limited amounts of data.
Additionally, we show that our approach scales to high-dimensional time series with latent dynamics such as rotating MNIST digits (Casale et al., 2018).
B ACKGROUND AND F ORMAL P ROBLEM S TATEMENT
Problem Statement We consider modelling random functions F : T → Y , where T =
[ t 0 , ∞ ) represents time and
Y ⊂ R d is a compact subset of R d .
We assume F has a distribution D , induced by another distribution D (cid:48) over some underlying dynamics that govern the time-series.
Given a speciﬁc instantation F of F , let C = { ( t C i , y C i ) } i ∈
I C be a set of samples from F with some indexing set I C .
We refer to C as the context points, as denoted by the superscript C. For a given context C, the task is to predict the values { y T j } j ∈ I T that F takes at a set of target times { t T j } j ∈ I T , where I T is another index set. We call T = { ( t T j , y T j ) } the target set.
Conventionally, as in Garnelo et al. (2018), the target set forms a superset of the context set and we have C ⊆ T.
During training, we assume access to a dataset of (potentially irregular) time-series sampled from F .
We are interested in learning the underlying distribution over the dynamics as well as the induced distribution over functions.
We note that when the dynamics are not latent and manifest directly in the observation space Y , the distribution over ODE trajectories and the distribution over functions coincide.
Neural ODEs NODEs are a class of models that parametrize the velocity ˙
z of a state z with the help of a neural network ˙
z = f θ ( z , t ) .
Given the initial time t 0 and target time t T i , NODEs predict the corresponding state ˆ y T i by performing the following integration and decoding operations:
[Equation]
where h 1 and h 2 can be neural networks.
When the dimensionality of z is greater than that of y and h 1 , h 2 are linear, the resulting model is an Augmented Neural ODE (Dupont et al., 2019) with input layer augmentation (Massaroli et al., 2020).
The extra dimensions offer the model additional ﬂexibility as well as the ability to learn higher-order dynamics (Norcliffe et al., 2020).
When applied to time-series prediction, NODEs are typically trained by minimising the mean squared error (MSE) between ˆ y C i and y C i over all time-steps { t C i } with i ∈ I .
They provide only a maximum-likelihood prediction and accounting for additional data requires further training or retraining entirely.
Neural Processes (NPs) NPs model a random function F : X → Y , where
X ⊆ R d 1 and
Y ⊆ R d 2 .
The NP represents a given instantiation F of F through the global latent variable z , which parametrises the variation in F .
Thus, we have F ( x i ) = g ( x i , z ) .
For a given context set
C = { ( x C i , y C i ) } and target set x 1: n , y 1: n , the generative process is given by:
[Equation]
where p ( z ) is chosen to be a multivariate standard normal distribution and y 1: n is a shorthand for the sequence ( y 1 , . .
y n ) .
The model can be trained using an amortised variational inference procedure that naturally gives rise to a permutation-invariant encoder q θ ( z | C ) , which stores the information about the context points.
Conditioned on this information, the decoder g ( x , z ) can make predictions at any input location x .
We note that while the domain X of the random function F is arbitrary, in this work we are interested only in stochastic functions with domain on the real line (time-series).
Therefore, from here our notation will reﬂect that, using t as the input instead of x .
The output y remains the same.
N EURAL ODE P ROCESSES
Model Overview We introduce Neural ODE Processes (NDPs), a class of dynamics-based models that learn to approximate random functions deﬁned over time.
To that end, we consider an NP whose context Z is used to determine a distribution over ODEs.
Concretely, Z gives a distribution over the initial position (and optionally – the initial velocity) and, at the same time, stochastically conditions its instantaneous velocity.
The positions given by the ODE trajectories at any time t T i are then decoded to give the predictions at t T i .
In what follows, we offer a detailed description of each component of the model.
A schematic of the model can be seen in Figure 1.
We ﬁrst describe the generative process behind NDPs. A graphical model perspective of this process is also included in Figure 2. Encoder and Aggregator Consider a given context set C
= { ( t C i , y C i ) } i ∈ I of points coming from a time-series.
As in NPs, we are ﬁrst interested in producing a distribution over functions induced by the conditional prior over Z ∼ q ( z | C ) .
To represent this distribution, the NDP encoder produces a representation r i = f e (( t C i , y C i )) for each context pair ( t C i , y C i ) .
The function f e is parametrised as a neural network, fully connected or convolutional, depending on the nature of y .
An aggregator combines all the representations r i to form a global representation, r , that parametrises the distribution of the global latent context, Z ∼ q ( z | C ) = N (cid:0) z | µ z ( r ) , diag( σ z ( r )) (cid:1) .
As the aggregator must preserve order invariance, we choose to take the element-wise mean.
Latent ODE Unlike NPs, which directly decode Z to obtain a distribution over functions, we are interested in capturing the dynamics that govern the time-series and exploiting the temporal nature of the data. To that end, we allow the latent context to evolve according to a Neural ODE (Chen et al., 2018) controlled by Z .
We begin by splitting the random context Z ∼ p ( z | C ) into an initial latent position L ( t 0 ) and a sub-context Z (cid:48) such that Z =
( L ( t 0 ) , Z (cid:48) ) .
Splitting Z factorises the uncertainty in the underlying dynamics into an uncertainty over the initial condition (given by
L ( t 0 ) ) and an uncertainty over the ODE derivative (given by Z (cid:48) ). Then, using the target times, t T 1: n = ( t T 1 , ..., t T N ) , the latent state at a given time is found by evolving a Neural ODE:
[Equation]
where f θ is a neural network that models the derivative of l .
As explained above, we allow z (cid:48) to modulate the derivative of this ODE. Ultimately, for ﬁxed initial conditions, this results in an uncertainty over the ODE trajectories. Decoder To obtain a prediction at a time t T i , we decode the random state of the ODE at time t T i , given by L ( t T i ) . Assuming that the outputs are noisy, for a given sample l ( t T i ) from this stochastic state, the decoder g produces a distribution over Y T t i ∼ p (cid:0) y T
i | g ( l ( t T i ) , t i ) (cid:1) parametrised by the decoder output. Concretely, for regression tasks, we take the target output to be normally distributed with constant (or optionally learned) variance Y T t i ∼ N (cid:0) y i | g ( l ( t i ) , t i ) , σ 2 (cid:1) . When Y T t i is a random vector formed of independent binary random variables (e.g. a black and white image), we use a
[Equation]
Putting everything together, for a set of observed context points C, the generative process of NDPs is given by:
[Equation]
where we highlight once again that l ( t i ) also implicitly depends on z . We remark that NDPs generalise NPs deﬁned over time . If the latent NODE learns the trivial velocity f θ ( l ( t ) , t, z (cid:48) ) = 0 , the random state L ( t ) =
L ( t 0 ) remains constant at all times t .
In this case, the distribution over functions is directly determined by Z ∼ p ( z | C ) , as in a regular NP.
For even greater ﬂexibility, one could also supply the sub-context z (cid:48) to the decoder g ( l ( t ) , t, z (cid:48) ) .
This shows that, in principle, NDPs are at least as expressive as NPs.
Therefore, NDPs could be a sensible choice even for applications where the time-series are not completely determined by some underlying dynamics, but are also inﬂuenced by other generative factors.
3. L EARNING AND I NFERENCE
Since the true posterior is intractable because of the highly non-linear generative process, the model is trained using an amortised variational inference procedure.
The variational lower-bound on the likelihood of the target values y m +1: n at t m +1:
n given the known context t 1: m , y 1: m is as follows:
[Equation]
where q gives variational posterior (the encoder described in Section 3.1).
The full derivation can be found in Appendix A.
We use the reparametrisation trick to backpropagate the gradients of this loss.
During training, we sample random contexts of different sizes to allow the model to become sensitive to the size of the context and the location of its points.
We train using mini-batches composed of multiple contexts.
For that, we use an extended ODE that concatenates the independent ODE states of each sample in the batch and integrates over the union of all the times in the batch (Rubanova et al., 2019).
Pseudo-code for this training procedure is also given in Appendix B.
3. M ODEL V ARIATIONS
Here we present the different ways to implement the model.
The majority of the variation is in the architecture of the decoder.
However, it is possible to vary the encoder such that f e (( t C i , y C i )) can be a multi-layer-perceptron, or additionally contain convolutions.
Neural ODE Process (NDP)
In this setup the decoder is a linear layer, which takes a concatenation of the latent state and a function of the latent state, time and sub-context as input: g ( l ( t T i ) , t T i , z (cid:48) )
= W ( l ( t T i ) (cid:107) h ( l ( t T i ) , t T i , z (cid:48) ))
+ b , where h can be a multi-layer-perceptron or additionally contain convolutional layers.
This type of model is particularly suitable for highdimensional time-series where the dynamics are fundamentally latent.
However, its additional representation power given by h make it a good default choice for most settings.
Second Order Neural ODE Process (ND2P)
This variation has the same decoder architecture as
NDP, however the latent ode evolves according to a second order ODE.
The latent state, l , is split in half into a “position”, l 1 and “velocity”, l 2 , with ˙ l 1 = l 2 and ˙ l 2 = f θ ( l 1 , l 2 , t, z (cid:48) ) .
This model is designed for time-series where the dynamics are second-order.
This is often the case given the use of Newton’s 2nd Law to describe physical laws (C¸a˘gatay Yıldız et al., 2019; Norcliffe et al., 2020).
NDP Latent-Only (NDP-L)
The decoder is a simple linear layer of the latent state g ( l ( t T i )) =
W ( l ( t T i )) + b .
This is based on the assumption that the latent state, which evolves according to an ODE, contains all the required information without any non-linearities.
This would be suitable for simple functions generated by ODEs, for example, sines and exponentials.
This decoder implicitly contains the time and context, because the latent state was evolved according to an ODE that depended on time and context given by Equation 3.
ND2P Latent-Only (ND2P-L)
This model combines the assumption of second-order underlying dynamics and the idea of the latent state containing all of the information.
The decoder is a linear layer of the latent state as in NDP-L, whose phase space dynamics are constrained as in ND2P.
3. N EURAL ODE P ROCESSES AS S TOCHASTIC P ROCESSES
The Kolmogorov Extension Theorem states that exchangeability and consistency are necessary and sufﬁcient conditions for a collection of joint marginal distributions to deﬁne a stochastic process Øksendal (2003); Garnelo et al.
We deﬁne these conditions and show that the NDP model satisﬁes them.
The proofs can be found in Appendix C.
Deﬁnition 3.1 (Exchangeability) .
Exchangeability refers to the invariance of the joint distribution ρ t 1: n ( y 1: n ) under permutations of y 1: n .
That is, for a permutation π of { 1 , 2 , ..., n } , π ( t 1: n ) =
( t π (1) , ..., t π ( n ) ) and π ( y 1: n ) =
( y π (1) , ..., y π ( n ) ) , the joint probability distribution ρ t 1: n ( y 1: n ) is invariant if ρ t 1: n ( y 1: n ) =
ρ π ( t 1: n ) ( π ( y 1: n )) .
Proposition 3.1.
NDPs satisfy the exchangeability condition.
Deﬁnition 3.2 (Consistency) .
Consistency says if a part of a sequence is marginalised out, then the joint probability distribution is the same as if it was only originally taken from the smaller sequence:
[Equation]
Proposition 3.2.
NDPs satisfy the consistency condition.
It is important to note that the stochasticity comes from sampling the full context z .
There is no stochasticity within the ODE, such as Brownian motion, though stochastic ODEs have previously been explored (Liu et al., 2019; Li et al., 2020; Tzen & Raginsky, 2019; Jia & Benson, 2020).
For any given context z , both the latent state trajectory and the observation space trajectory are fully determined.
3. R UNNING T IME C OMPLEXITY
For a model with n context points and m target points, an NP has running time complexity O ( n + m ) , since the model only has to encode each context point and decode each target point.
However, a
Neural ODE Process has added complexity due to the integration process.
Firstly, the integration itself has runtime complexity O ( NFE ) , where NFE is the number of function evaluations.
In turn, the worst-case NFE depends on the minimum step size δ the ODE solver has to use and the maximum time we are interested in, which we denote by t max .
Secondly, for settings where the target times are not already ordered, an additional O (cid:0) m log( m ) (cid:1) term is added for sorting them.
This ordering is required by the ODE solver.
Therefore, given that m ≥ n and assuming a constant t max exists, the worst-case complexity of NDPs is O (cid:0) m log( m ) (cid:1) .
For applications where the times are already sorted (e.g. real-time applications), the complexity falls back to the original O (cid:0) n + m (cid:1) . In either case, NDPs scale well with the size of the input. We note, however, that the integration steps t max / δ could result in a very large constant, hidden by the big- O notation.
Nonetheless, modern ODE solvers use adaptive step sizes that adjust to the data that has been supplied and this is unlikely to be a problem in practice.
In our experiments, when sorting is used, we notice the NDP models are between 1 and 1 .
5 orders of magnitude slower to train than NPs in terms of wall-clock time.
At the same time, this limitation of the method is traded-off by a signiﬁcantly faster loss decay per epoch and superior ﬁnal performance.
We provide a table of time ratios from our 1D experiments, from section 4.1, in Appendix D.
E XPERIMENTS
To test the proposed advantages of NDPs we carried out various experiments on time series data.
Additional details for every task considered can be found in Appendix G.
Model details For the low-dimensional experiments in Sections 4.1 and 4.2, we use an MLP architecture for the encoder and decoder.
For the high-dimensional experiments in Section 4.3, we use a convolutional architecture for both.
We train the models using RMSprop (Tieleman & Hinton, 2012) with learning rate 1 × 10 − 3 .
Additional model details can be found in Appendix F.
4. O NE D IMENSIONAL R EGRESSION
We begin with a set of 1D regression tasks of differing complexity—sine waves, exponentials, straight lines and damped oscillators—that can be described by ODEs.
For each task, the functions are determined by a set of parameters (amplitude, shift, etc) with pre-deﬁned ranges. To generate the distribution over functions, we sample these parameters from a uniform distribution over their respective ranges.
We use 490 time-series for training and evaluate on 10 separate test time-series. Each series contains 100 points. We repeat this procedure across 5 different random seeds to compute the standard error.
Additional details can be found in Appendix G.1.
First, we visualise in the top-left corner of Figure 3 how NPs and NDPs adapt on the sine task to incoming data points. When a single data-point has been supplied, NPs have wrongly collapsed the distribution over functions to a set of almost horizontal lines. On the other hand, NDPs correctly capture a wider range of possible trajectories. Even when a large number of points has been supplied, the NP posterior does not collapse on a perfect ﬁt, whereas NDPs correctly capture the true sine curve.
In the other plots of Figure 3, we visualise the test-set MSE as a function of the training epoch.
It can be seen that NDPs train in fewer iterations to a lower test loss due to the additional encoded structure of the latent ode state. At the same time, NDPs achieve this despite having approximately 10% fewer parameters than NPs. [Figure] y
NP on [Figure] − [Figure]
−−   3 t −  y
[Figure]
[Figure] [Figure] [Figure] E
[Figure] MSE On Exponential Data
[Figure]
NPNDP Further to this, we carry out an ablation study, training all model variants on all the 1D datasets.
Final test MSE losses are provided in Table 1 and the training plots are presented in Appendix G.1. [Table]
We ﬁnd that NDPs either strongly outperform NPs (sine, linear), or their standard errors overlap (exponential, oscillators).
For the exponential and harmonic oscillator tasks, where the models perform similarly, many points are close to zero in each example and as such it is possible to achieve a low MSE score by producing outputs that are also around zero. In contrast, the sine and linear datasets have a signiﬁcant variation in the y -values over the range, and we observe that NPs perform considerably worse than the NDP models on these tasks. The difference between the NDP and the best of the model variants is not signiﬁcant across the set of tasks.
As such, we continue to use only NDPs for the remainder of the paper as this is the least constrained model variant: they have unrestricted latent phase-space dynamics, unlike the secondorder counterparts, and a more expressive decoder architecture, unlike the latent-only variants.
In addition, NDPs train in a faster wall clock time than the other variants, as shown in Appendix D.
4. P REDATOR -P REY D YNAMICS
The Lotka-Volterra Equations are used to model the dynamics of a two species system, where one species predates on the other.
The populations of the prey, u , and the predator, v , are given by the differential equations:
[Equation]
for positive real parameters, α, β, δ, γ .
Intuitively, when prey is plentiful, the predator population increases ( + δuv ), and when there are many predators, the prey population falls ( − βuv ).
The populations exhibit periodic behaviour, with the phase-space orbit determined by the conserved quantity
V = δu − γ ln( u ) + βv − α ln( v ) .
Thus for any predator-prey system there exists a range of stable functions describing the dynamics, with any particular realisation being determined by the initial conditions, ( u 0 , v 0 ) .
We consider the system ( α, β, γ, δ ) =
[Figure]
N P
Posterior
PreyPredator
[Figure]
[-N D P 0 . 
 4 t 2 . V-]GroundTruthNPNDP
[Figure]
r
LV Phase [Figure]
GroundTruthNPNDP
[Figure]
t e x t
[Figure]
- N P v - N D P
Target
Context
NP
NDP
[Table]
We generate sample time-series from the Lotka Volterra system by considering different starting conﬁgurations; ( u 0 , v 0 ) =
(2 E, E ) , where E is sampled from a uniform distribution in the range (0.25, 1.0).
The training set consists of 40 such samples, with a further 10 samples forming the test set.
As before, each time series consists of 100 time samples and we evaluate across 5 different random seeds to obtain a standard error.
We ﬁnd that NDPs are able to train in fewer epochs to a lower loss (Appendix G.2).
We record ﬁnal test MSEs ( × 10 − 2 ) at 44 ± 4 for the NPs and 15 ± 2 for the NDPs. As in the 1D tasks, NDPs perform better despite having a representation r and context z with lower dimensionality, leading to 10% fewer parameters than NPs. Figure 4 presents these advantages for a single time series. 4. R OTATING MNIST
We consider the rotating MNIST digits task (Casale et al., 2018; C¸a ˘ gatay Yıldız et al., 2019).
Following C¸a ˘ gatay Yıldız et al.
(2019), we remove the digit corresponding to the 4th rotation from all time-series in the dataset and use it as a test frame, in order to compare directly with their method.
Additionally, we remove four random rotation angles from each sequence in the dataset to simulate irregularly sampled data.
At test time, we report the MSE for the 4th rotation angle.
We perform two types of evaluation.
We evaluate against NPs in predicting the 4th digit given a random context set C, as in previous tasks.
To compare with ODE 2 VAE (C¸a ˘ gatay Yıldız et al., 2019), we consider the restricted setting where the context set supplied at testing time C
= { ( y ( t 0 ) , t 0 ) } is formed of the single image from time t 0 .
The latter setup is required as the ODE 2 VAE is designed to evolve a latent Bayesian Neural ODE whose distribution over the initial position is given by y ( t 0 ) only.
In contrast, NDPs can handle arbitrary contexts.
We delay a detailed analysis of the differences between NDPs and ODE 2 VAE to Section 5.
The top part of Table 2 gives the results for the restricted setting.
As expected, NDPs perform better than NPs.
At the same time, NDPs have a competitive MSE to ODE 2 VAE, despite having the disadvantage of not being especially designed or trained for operating in this setting. The second part of Table 2 gives the MSE when a random context of size seven was used. Once again, NDPs show superior performance. We also illustrate this advantage qualitatively in Figure 5.
In Appendix
G.4 we show that the model is also able to capture diverse stylistic aberrations. [Figure] Additionally, we introduce a more challenging version of the Rotating MNIST dataset.
Samples in the original task start upright and rotate once over 16 frames (= 360 ◦ s − 1 ) (i.e. constant angular velocity, zero angular shift). In our adaptation, the angular velocity varies between samples in the range (360 ◦ ± 90 ◦ ) s − 1 and each sample starts at a random initial rotation.
As shown in Figure 6,
NDPs are able to extrapolate on the variable velocity MNIST dataset, although such extrapolation is not evident with the standard rotating MNIST dataset.
We hypothesise that this capability is due to the different angular velocities in the dataset inducing the model to unlearn the relatively trivial interpolation between frames in favour of learning the underlying dynamics.
In contrast, NPs struggle on this much more challenging domain and are unable to recover the stylistic aberrations of the digits.
D ISCUSSION AND R ELATED W ORK
We now consider two perspectives on how Neural ODE Processes relate to existing work and discuss the model in these contexts.
NDPs as Neural Processes From the perspective of stochastic processes, NDPs can be seen as a generalisation of NPs deﬁned over time and, as such, existing improvements in this family are likely orthogonal to our own.
For instance, following the work of Kim et al.
(2019), we would expect adding an attention mechanism to NDPs to reduce uncertainty around context points.
Additionally, the intrinsic sequential nature of time could be further exploited to model a dynamically changing sequence of NDPs as in Sequential NPs (Singh et al., 2019).
For application domains where the observations evolve on a graph structure, such as trafﬁc networks, relational information could be exploited with message passing operation as in MPNPs (Day et al., 2020).
NDPs as Neural ODEs From a dynamics perspective, NDPs can be thought of as an amortised
Bayesian Neural ODE.
In this sense, ODE 2 VAE (C¸a˘gatay Yıldız et al., 2019) is the model that is most closely related to our method.
While there are many common ideas between the two, signiﬁcant differences exist.
Firstly, NDPs do not use an explicit Bayesian Neural Network but are linked to them through the theoretical connections inherited from NPs (Garnelo et al., 2018).
NDPs handle the uncertainty in the velocity (and acceleration) through Z (cid:48) , whereas ODE 2 VAE uses a distribution over the NODE’s weights. Secondly, NDPs stochastically condition the latent NODE’s output (the particle’s velocity) and initial position on an arbitrary context set of variable size.
In contrast,
ODE 2 VAE conditions only the initial position and initial velocity on the ﬁrst element and the ﬁrst M elements in the sequence, respectively.
Therefore, our model can dynamically adapt the dynamics to any observed time points.
From that point of view, our model also runs parallel to other attempts of making Neural ODEs capable to dynamically adapt to irregularly sampled data points (Kidger et al., 2020).
C ONCLUSION
We introduce Neural ODE Processes (NDPs), a new class of stochastic processes suitable for modelling data-adaptive stochastic dynamics.
NDPs tackle the two main problems faced by Neural
ODEs applied to dynamics-governed time series: adaptability to incoming data points and uncertainty in the underlying dynamics when the data is sparse and, potentially, irregularly sampled.
To do so, NDPs include a probabilistic ODE as an additional encoded structure, thereby incorporating the inductive bias that the time-series is the direct or latent manifestation of an underlying ODE.
Furthermore, NDPs maintain the scalability of NPs to very large inputs.
We evaluate our model on synthetic 1D and 2D data, as well as higher-dimensional problems such as rotating MNIST digits.
Our method exhibits superior training performance when compared with NPs, yielding a lower loss in fewer iterations.
Whether or not the underlying ODE of the data is latent, we ﬁnd that where there is a fundamental ODE governing the dynamics, NDPs perform well.