Neural ODE Processes: A Short Summary
Abstract
Neural Ordinary Differential Equations (NODEs) use a neural network to model the instantaneous rate of change in the state of a system.
However, despite their apparent suitability for dynamics-governed time-series, NODEs present a few disadvantages.
First, they are unable to adapt to incoming data-points, a fundamental requirement for real-time applications imposed by the natural direction of time.
Second, time-series are often composed of a sparse set of measurements, which could be explained by many possible underlying dynamics.
NODEs do not capture this uncertainty.
To this end, we introduce Neural ODE Processes (NDPs), a new class of stochastic processes determined by a distribution over Neural ODEs.
By maintaining an adaptive data-dependent distribution over the underlying ODE, we show that our model can successfully capture the dynamics of low-dimensional systems from just a few data-points.
At the same time, we demonstrate that NDPs scale up to challenging high-dimensional time-series with unknown latent dynamics such as rotating MNIST digits.
Code is available online at https://github.com/crisbodnar/ndp .
Background and Formal Problem Statement
Problem Statement We consider modelling random functions F : T → Y , where T =
[ t 0 , ∞ ) represents time and Y ⊂ R d is a compact subset of R d .
We assume F has a distribution D , induced by another distribution D (cid:48) over some underlying dynamics that govern the time-series.
For example, ﬁlming a pendulum, F has a distribution in pixel space which is induced by the distribution of the pendulum dynamics.
Given a speciﬁc instantation F of F let C = ( t C i , y C i ) } i ∈
I C be a set of samples from F with some indexing set I C .
We refer to C as the context points, as denoted by the superscript C. For a given context C, the task is to predict the values { y T j } j ∈ I T that F takes at a set of target times { t T j } j ∈ I T , where I T is another index set.
We call T = { ( t T j , y T j ) } the target set.
Additionally let t C
= { t i | i ∈ I C } and similarly deﬁne y C , t T and y T .
Conventionally, as in Garnelo et al.
[5], the target set forms a superset of the context set
and we have C ⊆
T. Optionally, it might also be natural to consider that the initial time and observation ( t 0 , y 0 ) are always included in C. During training, we let the model learn from a dataset of (potentially irregular) time-series sampled from F .
We are interested in learning the underlying distribution over the dynamics as well as the induced distribution ∗ Equal contribution. over functions.
We note that when the dynamics are not latent and manifest directly in the observation space Y , the distribution over ODE trajectories and the distribution over functions coincide.
Neural ODEs NODEs are a class of models that parametrize the velocity ˙
z of a state z with the help of a neural network ˙
z = f θ ( z , t ) .
Given the initial time t 0 and target time t T i , NODEs predict the corresponding state ˆ y T i by performing the following integration and decoding operations:
[Equation]
where h 1 and h 2 can be neural networks.
When the dimensionality of z is greater than that of y and h 1 , h 2 are linear, the resulting model is an Augmented Neural ODE [4] with input layer augmentation [7].
The extra dimensions offer the model additional ﬂexibility as well as the ability to learn higher-order dynamics [8].
Neural Processes (NPs) NPs [5] model a random function F : T → Y , where in general T ⊆
R d 1 and
Y ⊆ R d 2 .
The NP represents a given instantiation F of F through the global latent variable z , which parametrises the variation in F .
Thus, we have F ( t i ) =
g ( t i , z ) .
For a given context set
C = { ( t C i , y C i ) } and target set t 1: n , y 1: n , the generative process is given by:
[Equation]
where p ( z ) is chosen to be a multivariate standard normal distribution and y 1: n is a shorthand for the sequence ( y 1 , . .
y n ) .
The model can be trained using an amortised variational inference procedure that naturally gives rise to a permutation-invariant encoder q θ ( z | C ) , which stores the information about the context points.
Conditioned on this information, the decoder g ( t , z ) can make predictions at any input location t .
We note that while the domain T of the random function F is arbitrary, in this work we are interested only in stochastic functions with domain on the real line (time-series).
Therefore, from here we use a scalar variable t , instead of t .
The output y remains the same.
Neural ODE Processes
Model Overview We introduce Neural ODE Processes (NDPs), a class of dynamics-based models that learn to approximate random functions deﬁned over time.
To that end, we consider an NP whose context is used to determine a distribution over ODEs.
Concretely, the context infers a distribution over the initial position (and optionally – the initial velocity) and, at the same time, stochastically controls its derivative function.
The positions given by the ODE trajectories at any time t T i are then decoded to give the predictions.
In what follows, we offer a detailed description of each component of the model.
A schematic of the model can be seen in Figure 1.
2.1 Generative Process
We ﬁrst describe the generative process behind NDPs.
A graphical model perspective of this process is also included in Figure 2.
Encoder and Aggregator Consider a given context set C
= { ( t C i , y C i ) } i ∈
I C of observed points.
We encode this context into two latent variables L 0 ∼ q L ( l ( t 0 ) | C ) and D ∼ q D ( d | C ) , rep- resenting the initial state and the global control of an ODE, respectively.
To parametrise the distribution of the latter variable, the NDP encoder produces a representation r i = f e (( t C i , y C i )) for each context pair ( t C i , y C i ) .
The function f e is as a neural network, fully connected or convolutional, depending on the nature of y .
An aggregator combines all the representations r i to form a global representation, r , that parametrises the distribution of the global latent context, D ∼ q D ( d | C ) =
N (cid:0) d | µ D ( r ) , diag( σ D ( r )) (cid:1) .
As the aggregator must preserve order invariance, we choose to take the element-wise mean.
The distribution of L 0 might be parametrised identically as a function of the whole context by q L ( l ( t 0 ) | C ) , and, in particular, if the initial observation y 0 is always known, then q L ( l ( t 0 ) [Equation] ) = [Equation]
[Equation]
( t 0 ) | y [Equation] ) =
N (cid:0) [Equation] ( t 0 ) |
[Equation]
L ( y 0 ) , diag(
[Equation] ( y 0 )) (cid:1) . Latent ODE To obtain a distribution over functions, we are interested in capturing the dynamics that govern the time-series and exploiting the temporal nature of the data.
To that end, we allow the latent context to evolve according to a Neural ODE [2] with initial position L 0 and controlled by D .
These two random variables factorise the uncertainty in the underlying dynamics into an uncertainty over the initial conditions (given by L 0 ) and an uncertainty over the ODE derivative, given by D .
By using the target times, t T 1: n = ( t T 1 , ..., t T N ) , the latent state at a given time is found by evolving a Neural ODE:
[Equation]
where f θ is a neural network that models the derivative of l .
As explained above, we allow d to modulate the derivative of this ODE by acting as a global control signal.
Ultimately, for ﬁxed initial conditions, this results in an uncertainty over the ODE trajectories.
Decoder To obtain a prediction at a time t T i , we decode the latent state of the ODE at time t T i , given by l ( t T i ) .
Assuming that the outputs are noisy, for a given sample l ( t T i ) from this stochastic state, the decoder g produces a distribution over Y T t i ∼ p (cid:0) y T
i | g ( l ( t T i ) , t T i ) (cid:1) parametrised by the decoder output.
Concretely, for regression tasks, we take the target output to be normally distributed with constant (or optionally learned) variance Y T t i ∼ N (cid:0) y T i | g ( l ( t T i ) , t T i ) , σ 2 (cid:1) .
When Y T t i is a random vector formed of independent binary random variables (e.g. a black and white image), we use a Bernoulli distribution
Y T t i ∼ (cid:81) dim( Y ) j =
Bernoulli
[Equation]
Putting everything together, for a set of observed context points C, the generative process of NDPs is given by the expression below, where we emphasise once again that l ( t i ) also implicitly depends on l ( t 0 ) and d .
[Equation]
We remark that NDPs generalise NPs deﬁned over time .
If the latent NODE learns the trivial velocity f θ ( l ( t ) , d , t ) = 0 , the random state L ( t ) =
L 0 remains constant at all times t .
In this case, the distribution over functions is directly determined by L 0 ∼ p ( l ( t 0 ) | C ) , which substitutes the random variable Z from a regular NP.
For greater ﬂexibility, the control signal d can also be supplied [Figure] to the decoder g ( l ( t ) , d , t ) .
This shows that, in principle, NDPs are at least as expressive as NPs.
Therefore, NDPs could be a sensible choice even in applications where the time-series are not solely determined by some underlying dynamics, but are also inﬂuenced by other generative factors.
2.2 Learning and Inference
Since the true posterior is intractable because of the highly non-linear generative process, the model is trained using an amortised variational inference procedure.
The variational lower-bound on the probability of the target values given the known context log p
( y T | t T , y C ) is as follows:
[Equation]
where q L , q D give the variational posteriors (the encoders described in Section 2.1).
The full derivation can be found in
Appendix E. We use the reparametrisation trick to backpropagate the gradients of this loss.
During training, we sample random contexts of different sizes to allow the model to become sensitive to the size of the context and the location of its points.
We train using mini-batches composed of multiple contexts.
For that, we use an extended ODE that concatenates the independent ODE states of each sample in the batch and integrates over the union of all the times in the batch [9].
Pseudo-code for this training procedure is also given in Appendix F.
Experiments
To test our model on high-dimensional time-series with latent dynamics, we consider the rotating
MNIST digits [1, 12].
In the original task, samples of digit “3” start upright and rotate once over 16 frames (= 360 ◦ s − 1 ) (i.e. constant angular velocity, zero angular shift).
However, since we are interested in time-series with variable latent dynamics and increased variability in the initial conditions as in our formal problem statement, we consider a more challenging version of the task.
In our adaptation, the angular velocity varies between samples in the range (360 ◦ ± 60 ◦ ) s − 1 and each sample starts at a random initial rotation.
To induce some irregularity in each time-series in the training dataset, we remove ﬁve randomly chosen time-steps (excluding the initial time t 0 ) from each time-series.
Overall, we generate a dataset with 1 , 000 training time-series, 100 validation time-series and 200 test time-series, each using disjoint combinations of different calligraphic styles and dynamics.
We compare NPs and NDPs using identical convolutional networks for encoding the images in the context.
We assume that the initial image y 0 (i.e. the image at t 0 ) is always present in the context.
As such, for NDPs, we compute the distribution of L 0 purely by encoding y 0 and disregarding the other samples in the context, as described in Section 2.
We train the NP for 500 epochs and use the validation set error to checkpoint the best model for testing.
We follow a similar procedure for the NDP model but, due to the additional computational load introduced by the integration operation, only train for 50 epochs.
In Figure 3, we include the predictions offered by the two models on a time-series from the test dataset, which was not seen in training by either of the models.
Despite the lower number of epochs they are trained for, NDPs are able to interpolate and even extrapolate on the variable velocity MNIST dataset, while also accurately capturing the calligraphic style of the digit.
NPs struggle on this challenging task and are unable to produce anything resembling the digits.
In order to better understand this wide performance gap, we also train in Appendix J.3 the exact same models on the easier Rotating MNIST task from Ça˘gatay Yıldız et al.
[12] where the angular velocity and initial rotation are constant.
In this setting, the two models perform similarly since the NP model can rely on simple interpolations without learning any dynamics.
Conclusion
We introduce Neural ODE Processes (NDPs), a new class of stochastic processes suitable for modelling data-adaptive stochastic dynamics.
First, NDPs tackle the two main problems faced by Neural ODEs applied to dynamics-governed time series: adaptability to incoming data points and uncertainty in the underlying dynamics when the data is sparse and, potentially, irregularly sampled.
Second, they add an explicit treatment of time as an additional inductive bias inside Neural
Processes.
To do so, NDPs include a probabilistic ODE as an additional encoded structure, thereby incorporating the assumption that the time-series is the direct or latent manifestation of an underlying
ODE.
Furthermore, NDPs maintain the scalability of NPs to large inputs.
We evaluate our model on a high-dimensional problem – the rotating MNIST digits.
Our method exhibits superior training performance when compared with NPs, yielding a lower loss in fewer iterations.
We ﬁnd that when there is a fundamental ODE governing the dynamics, NDPs perform well.
A Running Time Complexity
For a model with n context points and m target points, an NP has running time complexity O ( n + m ) , since the model only has to encode each context point and decode each target point.
However, a
Neural ODE Process has added complexity due to the integration process.
Firstly, the integration itself has runtime complexity O ( NFE ) , where NFE is the number of function evaluations.
In turn, the worst-case NFE depends on the minimum step size δ the ODE solver has to use and the maximum integration time we are interested in, which we denote by ∆ t max .
Secondly, for settings where the target times are not already ordered, an additional O (cid:0) m log( m ) (cid:1) term is added for sorting them.
This ordering is required by the ODE solver.
Therefore, given that m ≥ n and assuming a constant ∆ t max exists, the worst-case complexity of NDPs is O (cid:0) m log( m ) (cid:1) .
For applications where the times are already sorted (e.g. real-time applications), the complexity falls back to the original O (cid:0) n + m (cid:1) .
In either case, NDPs scale well with the size of the input.
We note, however, that the integration steps ∆ t max / δ could result in a very large constant, hidden by the big- O notation.
Nonetheless, modern ODE solvers use adaptive step sizes that adjust to the data that has been supplied and this should alleviate this problem.
In our experiments, when sorting is used, we notice the NDP models are between 1 and 1 .
5 orders of magnitude slower to train than NPs in terms of wall-clock time.
At the same time, this limitation of the method is traded-off by a signiﬁcantly faster loss decay per epoch and superior ﬁnal performance.
We provide a table of time ratios from our 1D experiments, from Appendix C.1, in Appendix G.
B Model Variations
Here we present the different ways to implement the model.
The majority of the variation is in the architecture of the decoder.
However, it is possible to vary the encoder such that f e (( t C i , y C i )) can be a multi-layer-perceptron, or additionally contain convolutions.
Neural ODE Process (NDP)
In this setup the decoder is an arbitrary function g ( l ( t T i ) , d , t T i ) of the latent position at the time of interest, the control signal, and time.
This type of model is particularly suitable for high-dimensional time-series where the dynamics are fundamentally latent.
The inclusion of d in the decoder offers the model additional ﬂexibility and makes it a good default choice for most tasks.
Second Order Neural ODE Process (ND2P)
This variation has the same decoder architecture as
NDP, however the latent ODE evolves according to a second order ODE.
The latent state, l , is split into a “position”, l 1 and “velocity”, l 2 , with ˙ l 1 = l 2 and ˙ l 2 = f θ ( l 1 , l 2 , d , t ) .
This model is designed for time-series where the dynamics are second-order, which is often the case for physical systems [8, 12].
NDP Latent-Only (NDP-L)
The decoder is a linear transformation of the latent state g ( l ( t T i )) =
W ( l ( t T i ))
+ b .
This model is suitable for the setting when the dynamics are fully observed (i.e. they are not latent) and, therefore, do not require any decoding.
This would be suitable for simple functions generated by ODEs, for example, sines and exponentials.
This decoder implicitly contains information about time and d because the ODE evolution depends on these variables as described in
Equation 3.
ND2P Latent-Only (ND2P-L)
This model combines the assumption of second-order dynamics with the idea that the dynamics are fully observed.
The decoder is a linear layer of the latent state as in NDP-L and the phase space dynamics are constrained as in ND2P.
C Low-dimensional experiments
To test the proposed advantages of NDPs we carried out various experiments on time series data.
For the low-dimensional experiments in Sections C.1 and C.2, we use an MLP architecture for the encoder and decoder.
For the high-dimensional experiments in Section 3, we use a convolutional architecture for both.
We train the models using RMSprop [10] with learning rate 1 × 10 − 3 .
Additional model and task details can be found in Appendices I and J, respectively.
−  y
NP on Sine Data −−−   3 t −  y −
NDP on Sine Data −−−   3 t −        30 Epoch 0 .
T e s t M
S E
Test MSE On Sine Data
NPNDP
C.1 One Dimensional Regression
We begin with a set of 1D regression tasks of differing complexity—sine waves, exponentials, straight lines and damped oscillators—that can be described by ODEs.
For each task, the functions are determined by a set of parameters (amplitude, shift, etc.) with pre-deﬁned ranges.
To generate the distribution over functions, we sample these parameters from a uniform distribution over their respective ranges.
We use 490 time-series for training and evaluate on 10 separate test time-series.
Each series contains 100 points.
We repeat this procedure across 5 different random seeds to compute the standard error.
Additional details can be found in Appendix J.1.
The left and middle panels of Figure 4 show how NPs and NDPs adapt on the sine task to incoming data points.
When a single data-point has been supplied, NPs have incorrectly collapsed the distribution over functions to a set of almost horizontal lines.
NDPs, on the other hand, are able to produce a wide range of possible trajectories.
Even when a large number of points have been supplied, the
NP posterior does not converge on a good ﬁt, whereas NDPs correctly capture the true sine curve.
In the right panel of Figure 4, we show the test-set MSE as a function of the training epoch.
It can be seen that NDPs train in fewer iterations to a lower test loss despite having approximately 10% fewer parameters than NPs.
We conducted an ablation study, training all model variants on all the 1D datasets, with ﬁnal test MSE losses provided in Table 1 and training plots in Appendix J.1.
We ﬁnd that NDPs either strongly outperform NPs (sine, linear), or their standard errors overlap (exponential, oscillators).
For the exponential and harmonic oscillator tasks, where the models perform similarly, many points are close to zero in each example and as such it is possible to achieve a low MSE score by producing outputs that are also around zero.
In contrast, the sine and linear datasets have a signiﬁcant variation in the y -values over the range, and we observe that NPs perform considerably worse than the NDP models on these tasks.
The difference between NDP and the best of the other model variants is not signiﬁcant across the set of tasks.
As such, we consider only NDPs for the remainder of the paper as this is the least constrained model version: they have unrestricted latent phase-space dynamics, unlike the secondorder counterparts, and a more expressive decoder architecture, unlike the latent-only variants.
In addition, NDPs train in a faster wall clock time than the other variants, as shown in Appendix G.
Active Learning We perform an active learning experiment on the sines dataset to evaluate both the uncertainty estimates produced by the models and how well they adapt to new information.
Provided with an initial context point, additional points are greedily queried according to the model uncertainty.
Higher quality uncertainty estimation and better adaptation will result in more information being acquired at each step, and therefore a faster and greater reduction in error.
As shown in Figure 5,
NDPs also perform better in this setting.
C.2 Predator-Prey Dynamics
The Lotka-Volterra Equations are used to model the dynamics of a two species system, where one species predates on the other.
The populations of the prey, u , and the predator, v , are given by
−−−   3 t − 1 .− 0 .
75 − 0 .− 0 .
N P A c t i v e
N = 1 N =
N =−−−   3 t − 1 .− 0 .
75 − 0 .− 0 .
N D P
A c t i v e    10 Number of Points 0 .0 .0 .0 .0 .0 .0 .0 .
M S E
NP Random NP Active
NDP Random NDP Active
[Table]
the differential equations ˙ u
= αu − βuv, ˙ v = δuv
− γv , for positive real parameters, α, β, δ, γ .
Intuitively, when prey is plentiful, the predator population increases ( + δuv ), and when there are many predators, the prey population falls ( − βuv ).
The populations exhibit periodic behaviour, with the phase-space orbit determined by the conserved quantity V = δu − γ ln( u ) + βv − α ln( v ) .
Thus for any predator-prey system there exists a range of stable functions describing the dynamics, with any particular realisation being determined by the initial conditions, ( u 0 , v 0 ) .
We consider the system ( α, β, γ, δ ) =
We generate sample time-series from the Lotka Volterra system by considering different starting conﬁgurations; ( u 0 , v 0 ) =
(2 E, E ) , where E is sampled from a uniform distribution in the range (0.25, 1.0).
The training set consists of 40 such samples, with a further 10 samples forming the test set.
As before, each time series consists of 100 time samples and we evaluate across 5 different random seeds to obtain a standard error.
We ﬁnd that NDPs are able to train in fewer epochs to a lower loss (Appendix J.2).
We record ﬁnal test MSEs ( × 10 − 2 ) at 44 ± 4 for the NPs and 15 ± 2 for the NDPs.
As in the 1D tasks, NDPs perform better despite having a representation r and context z with lower dimensionality, leading to 10% fewer parameters than NPs.
Figure 6 presents these advantages for a single time series.
D Stochastic Process Proofs
Before giving the proofs, we state the following important Lemma.
Lemma D.1.
As in NPs, the decoder output g ( l ( t ) , t ) can be seen as a function F ( t ) for a given ﬁxed l ( t 0 ) and d .
Proof.
This follows directly from the fact that l ( t ) =
l ( t 0 ) + (cid:82) T t
f θ ( l ( t ) , t, d ) dt can be seen as a function of t and that the integration process is deterministic for a given pair l ( t 0 ) and
d (i.e. for ﬁxed initial conditions and control).
N D P
Prey u
Predator v
Context 0 .1 .
[Figure]
N P 0 .1 .
5 Time 2 .2 .
V = δ u − γ l n u + β
v
−
α l n v
Groundtruth
NP
NDP 0 .1 .
5 Prey population u 0 .0 .
P r e d a t o r p
o
pu l a t
i o n v
Targets
NP
NDP
Proposition D.1 NDPs satisfy the exchangeability condition.
Proof.
This follows directly from Lemma D.1, since any permutation on t 1: n would automatically act on F 1: n and consequently on p ( y 1: n , l ( t 0 ) , d | t 1: n ) , for any given l ( t 0 ) , d .
Proposition D.2 NDPs satisfy the consistency condition.
Proof.
Based on Lemma D.1 we can write the joint distribution (similarly to a regular NP) as follows:
[Equation]
Because the density of any y i depends only on the corresponding t i , integrating out any subset of y 1: n gives the joint distribution of the remaining random variables in the sequence.
Thus, consistency is guaranteed.
E ELBO Derivation
As noted in Lemma D.1, the joint probability p ( y , l ( t 0 ) , d | t ) =
p ( l ( t 0 ))
p ( d ) p ( y | g ( l ( t ) , d , t )) can still be seen as a function that depends only on t , since the ODE integration process is deterministic for a given l ( t 0 ) and d .
Therefore, the ELBO derivation proceeds as usual [5].
For convenience, let z =
( l ( t 0 ) , d ) denote the concatenation of the two latent vectors and q ( z ) =
q L ( l ( t 0 ))
q D ( d ) .
First, we derive the ELBO for log p ( y T | t T ) .
[Equation]
Noting that at training time, we want to maximise log p ( y T | t T , y C ) .
Using the derivation above, we obtain a similar lower-bound, but with a new prior p ( z | t C , y C ) , updated to reﬂect the additional information supplied by the context.
[Equation]
If we approximate the true p ( z | t C , y C ) with the variational posterior, this takes the ﬁnal form
[Equation]
Splitting z =
( l ( t 0 ) , d ) back into its constituent parts, we obtain the loss function
[Equation]
F Learning and Inference Procedure
We include below the pseudocode for training NDPs.
For clarity of exposition, we give code for a single time-series.
However, in practice, we batch all the operations in lines 6 − 15 .
Algorithm
Input : A dataset of time-series { X k } , k ≤ K , where K is the total number of time-series 1 Initialise NDP model with parameters θ 2 Let m be the number of context points and n the number of extra target points 3 for i ← 0 to training_steps do
Sample m from U[ min _ context _ points , max _ context _ points ]
Sample n from U[ min _ extra _ target _ points , max _ extra _ target _ points ]
Uniformly sample a time-series X k
Uniformly sample from X k the target points T
=
( t T , y T ) , where t T is the time batch with shape ( m + n , 1 ) and y T is the corresponding outputs batch with shape ( m + n , dim( y ) )
Extract the unordered) context set m
Compute q l
0 ) , d | C ) using the variational encoder
Compute q ( l t 0 ) , d | T ) using the variational encoder //
During training, we sample from q ( l t 0 ) d | T )
Sample l ( t 0 ) , d from q ( l ( t
, d
T )
Integrate to compute l ( t ) as in Equation 3 for all times t ∈ t T foreach time t ∈ t T do
Use decoder to compute p y t t )
Compute loss L ELBO based on Equation
It is worth highlighting that during training we sample l ( t 0 ) , d from the target-conditioned posterior, rather than the context-conditioned posterior.
In contrast, at inference time we sample from the context-conditioned posterior.
G Wall Clock Training Times
To explore the additional term in the runtime given in Section A, we record the wall clock time for each model to train for 30 epochs on the 1D synthetic datasets, over 5 seeds.
Then we take the ratio of a given model and the NP.
The experiments were run on an Nvidia Titan XP .
The results can be seen in Table 2.
H Size of Latent ODE
To investigate how many dimensions the ODE l should have, we carry out an ablation study, looking at the performance on the 1D sine dataset.
We train models with l -dimension { 1 , 2 , 5 , 10 , 15 , 20 } for 30 epochs.
Figure 7 shows training plots for dim ( l ) = { 1 , 2 , 10 , 20 } , and ﬁnal MSE values are given in Table 3.
We see that when dim ( l ) = 1 , NDPs are slow to train and require more epochs.
This is because sine curves are second-order ODEs, and at least two dimensions are required to learn second-order
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
dynamics (one for the position and one for the velocity).
When dim ( l ) = 1 , NDPs perform similarly to NPs, which is expected when the latent ODE is unable to capture the underlying dynamics.
We then see that for all other dimensions, NDPs train at approximately the same rate (over epochs) and have similar ﬁnal MSE scores.
As the dimension increases beyond 10, the test MSE increases, indicating overﬁtting.
I Architectural Details
For the experiments with low dimensionality (1D, 2D), the architectural details are as follows:
• Encoder :
[ t i , y i ] −→ r i : Multilayer Perceptron, 2 hidden layers, ReLU activations.
• Aggregator : r 1: n −→ r : Taking the mean.
• Representation to Hidden : r −→ h : One linear layer followed by ReLU.
• Hidden to L 0 Mean : h −→ µ L : One linear layer.
• Hidden to L
Variance : h −→ σ L : One linear layer, followed by sigmoid, multiplied by 0.9 add 0.1, i.e. σ L = 0 .
1 + 0 .
9 × sigmoid ( W h + b ) .
• Hidden to D Mean : h −→ µ D : One linear layer.
• Hidden to D Variance : h −→ σ D : One linear layer, followed by sigmoid, multiplied by 0.
add 0.1, i.e. σ D = 0 .
1 + 0 .
9 × sigmoid ( W h + b ) .
• ODE Layers : [ l , d , t ] −→ ˙ l : Multilayer Perceptron, two hidden layers, tanh activations.
• Decoder : g ( l ( t T i ) , d , t T i ) −→ y T
i , for the NDP model and ND2P described in
section B,
this function is a linear layer, acting on a concatenation of the latent state and a function of
l (
t T i
d , and t T i .
g
l
t T i
) , d , t T i ) =
W ( l ( t T i ) ||
h ( l ( t T i ) , d , t T i ))
+ b .
Where h is a Multilayer
Perceptron with two hidden layers and
ReLU
activations.
[Table]
• Encoder :
[ t i , y i ] −→ r i : Convolutional Neural Network, 4 layers with 16, 32, 64,channels respectively and kernel size of 5, stride 2.
ReLU activations.
Batch normalisation.
• Aggregator : r 1: n −→ r : Taking the mean.
• Representation to D Hidden : r −→ h D : One linear layer followed by ReLU.
• D Hidden to D Mean : h D −→ µ D : One linear layer.
• D Hidden to D Variance : h D −→ σ D : One linear layer, followed by sigmoid, multiplied by 0.9 add 0.1, i.e. σ D = 0 .
1 + 0 .
9 × sigmoid ( W h D + b ) .
• y 0 to L 0 Hidden :
y
−→ h L : Convolutional Neural Network, 4 layers with 16, 32, 64,channels respectively and kernel size of 5, stride 2.
ReLU activations.
Batch normalisation.
• L 0 Hidden to L 0 Mean : h L −→ µ L : One linear layer.
• L 0 Hidden to L 0 Variance : h L −→ σ L : One linear layer, followed by sigmoid, multiplied by 0.9 add 0.1, i.e. σ L = 0 .
1 + 0 .
9 × sigmoid ( W h L + b ) .
• ODE Layers : [ l , d , t ] −→ ˙ l : Multilayer Perceptron, two hidden layers, tanh activations.
• Decoder : g ( l ( t T i ))
−→ y T i : 1 linear layer followed by a 4 layer transposed Convolutional Neural Network with 32, 128, 64, 32 channels respectively.
ReLU activations.
Batch normalisation.
J Task Details and Additional Results J.1 One Dimensional Regression
We carried out an ablation study over model variations on various 1D synthetic tasks—sines, exponentials, straight lines and harmonic oscillators.
Each task is based on some function described by a set of parameters that are sampled over to produce a distribution over functions.
In every case, the parameters are sampled from uniform distributions.
A trajectory example is formed by sampling from the parameter distributions and then sampling from that function at evenly spaced timestamps, t , over a ﬁxed range to produce 100 data points ( t, y ) .
We give the equations for these tasks in terms of their deﬁning parameters and the ranges for these parameters in Table 4.
To test after each epoch, 10 random context points are taken, and then the mean-squared error and negative log probability are calculated over all the points (not just a subset of the target points).
Each model was trained 5 times on each dataset (with different weight initialisation).
We used a batch size of 5, with context size ranging from 1 to 10, and the extra target size ranging from 0 to 5.
The results are presented in Figure 8.
All models perform better than NPs, with fewer parameters (approximately 10% less).
Because there are no signiﬁcant differences between the different models, we use NDP in the remainder of
[Table]
the experiments, because it has the fewest model restrictions.
The phase space dynamics are not restricted like its second-order variant, and the decoder has a more expressive architecture than the latent-only variants.
It also trains the fastest in wall clock time seen in Appendix G.
J.2 Lotka-Volterra System
To generate samples from the Lotka Volterra system, we sample different starting conﬁgurations, ( u 0 , v 0 ) =
(2 E, E ) , where E is sampled from a uniform distribution in the range (0.25, 1.0).
We then evolve the Lotka Volterra system du
[Equation]
(14) using ( α, β, γ, δ ) =
This is evolved from t = 0 to t = 15 and then the times are rescaled by dividing by 10.
The training for the Lotka-Volterra system can be seen in Figure 9.
This was taken across 5 seeds, with a training set of 40 trajectories, 10 test trajectories and batch size 5.
We use a context size ranging from 1 to 100, and extra target size ranging from 0 to 45.
The test context size was ﬁxed at 90 query times.
NDPs train slightly faster with lower loss, as expected.
J.3 Rotating MNIST & Additional Results
To better understand what makes vanilla NPs fail on our Variable Rotating MNIST from Section 3, we train the exact same models on the simpler Rotating MNIST dataset [12].
In this dataset, all digits start in the same position and rotate with constant velocity.
Additionally, the fourth rotation is removed from all the time-series in the training dataset.
We follow the same training procedure as in
Section 3.
We report in Figure 10 the predictions for the two models on a random time-series from the validation dataset.
First, NPs and NDPs perform similarly well at interpolation and extrapolation within the time-interval used in training.
As an exception but in agreement with the results from ODE 2 VAE,
NDPs produces a slightly better reconstruction for the fourth time step in the time-series.
Second, neither model is able to extrapolate the dynamics beyond the time-range seen in training (i.e. the last ﬁve time-steps).
Overall, these observations suggest that for the simpler RotMNIST dataset, explicit modelling of the dynamics is not necessary and the tasks can be learnt easily by interpolating between the context points.
And indeed, it seems that even NDPs, which should be able to learn solutions that extrapolate, collapse on these simpler solutions present in the parameter space, instead of properly learning the desired latent dynamics.
A possible explanation is that the Variable Rotating MNIST dataset can be seen as an image augmentation process which makes the convolutional features to be approximately rotation equivariant.
In this way, the NDP can also learn rotation dynamics in the spatial dimensions of the convolutional features.
Finally, in Figure 11, we plot the reconstructions of different digit styles on the test dataset of Variable
Rotating MNIST.
This conﬁrms that NDPs are able to capture different calligraphic styles.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
J.4 Handwritten Characters
The CharacterTrajectories dataset consists of single-stroke handwritten digits recorded using an electronic tablet [3, 11].
The trajectories of the pen tip in two dimensions, ( x, y ) , are of varying length, with a force cut-off used to determine the start and end of a stroke.
We consider a reduced dataset, containing only letters that were written in a single stroke, this disregards letters such as “f”, “i” and “t”.
Whilst it is not obvious that character trajectories should follow an ODE, the related
Neural Controlled Differential Equation (NCDEs) model has been applied successfully to this task [6].
We train with a training set with 49600 examples, a test set with 400 examples and a batch size of 200.
We use a context size ranging between 1 and 100, an extra target size ranging between 0 and 100 and a ﬁxed test context size of 20.
We visualise the training of the models in Figure 12 and the models plotting posteriors in Figure 13.
[Figure]
LV - Training Loss
[Figure]
(p)
NPNDP
[Figure]
We observe that NPs and NDPs are unable to successfully learn the time series as well as NCDEs.
We record ﬁnal test MSEs ( × 10 − 1 ) at 4 .
6 ± 0 . 1 for NPs and a slightly lower 3 .
4 ± 0 . 1 for NDPs.
We believe the reason is because handwritten digits do not follow an inherent ODE solution, especially given the diversity of handwriting styles for the same letter.
We conjecture that Neural Controlled
Differential Equations were able to perform well on this dataset due to the control process.
Controlled
ODEs follow the equation:
[Equation]
Where X ( t ) is the natural cubic spline through the observed points x ( t ) .
If the learnt f θ is an identity operation, then the result returned will be the cubic spline through the observed points.
Therefore, a controlled ODE can learn an identity with a small perturbation, which is easier to learn with the aid of a control process, rather than learning the entire ODE trajectory.
[Figure] [Figure]