A NAMNESIC N EURAL D IFFERENTIAL E QUATIONS WITH O RTHOGONAL P OLYNOMIALS P ROJECTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural ordinary differential equations (Neural ODEs) are an effective framework for learning dynamical systems from irregularly sampled time series data.
These models provide a continuous-time latent representation of the underlying dynamical system where new observations at arbitrary time points can be used to update the latent representation of the dynamical system.
Existing parameterizations for the dynamics functions of Neural ODEs limit the ability of the model to retain global information about the time series; speciﬁcally, a piece-wise integration of the latent process between observations can result in a loss of memory on the dynamic patterns of previously observed data points.
We propose PolyODE, a
Neural ODE that models the latent continuous-time process as a projection onto a basis of orthogonal polynomials.
This formulation enforces long-range memory and preserves a global representation of the underlying dynamical system.
Our construction is backed by favourable theoretical guarantees and in a series of experiments, we demonstrate that it outperforms previous works in the reconstruction of past and future data , and in downstream prediction tasks.
Our code is available at https://anonymous.4open.science/r/polyode/ .
I NTRODUCTION
Time series are ubiquitous in many ﬁelds of science such as healthcare, ﬁnance and biology, and as such, represent an important but challenging data modality for machine learning.
Indeed, their temporal nature, along with the potentially high dimensionality makes them arduous to manipulate as mathematical objects.
A long-standing line of research has thus focused on efforts in learning informative time series representations, such as simple vectors, that are capable of capturing local and global structure in such data (Franceschi et al., 2019; Gu et al., 2020).
Such architectures include recurrent neural networks (Malhotra et al., 2017), temporal transformers (Zhou et al., 2021) and neural ordinary differential equations (Rubanova et al., 2019; Chen et al., 2018).
In particular, neural ordinary differential equations (Neural ODEs) have emerged as a popular choice for time series modelling due to their sequential nature and their ability to handle irregularly sampled time-series data.
Sequential processing, as opposed to parallel transformer architectures, allows to us update the time series representation as new observations are reached.
By positing an underlying continuous time dynamic process, Neural ODEs sequentially process irregularly sampled time series via numerical integration of the dynamics between observations.
In doing so, they update the hidden representation for each new choice of observation.
The ﬂexibility of this model family arises from the use of neural networks to parameterize the dynamics function, and different choices of parameterizations of the dynamics function can have different properties.
For instance, bounding the output of the neural networks can enforce Lipschitz constants over the temporal process (De Brouwer et al., 2019; Onken et al., 2021).
The problem this work tackles is that the piece-wise integration of the latent process between observations can fail to retain a global representation of the time series.
Speciﬁcally, each change to the hidden state of the dynamical system from a new observation can result in a loss of memory about prior dynamical states the model was originally in.
This pathology limits the utility of Neural ODEs when there is a necessity to retain information about the recent and distant past; i.e. current Neural
ODE formulations are amnesic .
We illustrate this effect in Figure 1, where we see that backward integration of a learned Neural ODE (that is competent at forecasting) quickly diverges, indicating the state only retains sufﬁcient local information about the future dynamics.
[Figure]
One strategy that has been explored in the past to address this pathology is to regularize the model to encourage it to capture long-range patterns by reconstructing the time series from the last observation, using an auto-encoder architecture(Rubanova et al., 2019; Tschannen et al., 2018).
This class of approaches results in higher complexity and does not provide any guarantees on the retention of the history of a time series.
In contrast, our work proposes an alternative parameterization of the dynamics function that, by design , captures long-range memory within a Neural ODE.
Building upon the recent successes of the HiPPO framework Gu et al.
, we achieve this by enforcing that the dynamics of the hidden process follow the dynamics of the projection of the observed temporal process onto a basis of orthogonal polynomials.
The resulting model, PolyODE, thereby ensures encoding of long-range past information in the latent process and is thus anamnesic .
As depicted in Figure 1, the resulting time series embeddings are able to reconstruct the past time series with signiﬁcantly better accuracy.
Contributions
• We propose a novel dynamics function for a neural ODE resulting in PolyODE, a model that learns a global representation of high-dimensional time series and is capable of longterm forecasting and reconstruction by design.
• Methodologically, we highlight the practical challenges in learning PolyODE and show how adaptive solvers for ODEs can overcome them.
Theoretically, we provide bounds characterizing the quality of reconstruction of time series when using PolyODE.
Empirically, we study the performance of our approach by assessing the ability of the learnt embeddings to reconstruct the past of the time series and by studying their utility as inputs for downstream predictive tasks.
We show that our model provides better time series representations, relative to several existing Neural ODEs architectures, based on the ability of the representations to accurately make predictions on several downstream tasks based on chaotic time series and irregularly sampled data from patients in intensive care units.
R ELATED WORK
Time series modelling in machine learning: There is vast literature on the use of machine learning for time series modelling and we highlight some of the ideas that have been explored to adapt diverse kinds of models for irregular time series data.
Although not naturally well suited to learning representations of such data, there have been modiﬁcations proposed to discrete-time models such as recurrent neural networks (Hochreiter and Schmidhuber, 1997; Cho et al., 2014) to handle such data.
Models such as mTANs (Shukla and Marlin, 2021) leverage an attention-based approach to interpolate sequences to create discrete-time data from irregularly sampled data).
Another strategy has been architectural modiﬁcations to the recurrence equations e.g. CT-GRU (Mozer et al., 2017),
GRU-D (Che et al., 2018) and Unitary RNNs (Arjovsky et al., 2016).
Much more closely aligned to our work, and a natural ﬁt for irregularly sampled data is research that uses differential equations to model continuous-time processes(Chen et al., 2018).
By parameterizing the derivative of a time series using neural networks and integrating the dynamics over unobserved time points, this class of models is well suited to handle irregularly sampled data.
This includes models such as ODERNN (Rubanova et al., 2019), ODE-LSTM (Lechner and Hasani, 2020) and Neural CDE (Kidger et al., 2020).
ODE-based approaches require the use of differential equation solvers during training and inference, which can come at the cost of runtime (Shukla and Marlin, 2021).
PolyODEs lie in this family of models; speciﬁcally, this work proposes a new parameterization of the dynamics function and a practical method for learning that enables this model family to accurately forecast the future and reconstruct the past greatly enhancing the scope and utility of the learned embeddings.
Orthogonal polynomials: PolyODEs are inspired by a rich line of work in orthogonal decomposition of time series data.
Orthogonal polynomials have been a mainstay in the toolkit for engineering (Heuberger et al., 2003) and uncertainty quantiﬁcation (Li et al., 2011).
In the context of machine learning, the limitations of RNNs to retain long-term memory have been studied empirically and theoretically (Zhao et al., 2020).
Indeed, the GRU (Chung et al., 2014) and LSTM (Graves et al., 2007) architectures were created in part to improve the long-term memory of such models.
Recent approaches for discrete-time models have used orthogonal polynomials and their ability to represent temporal processes in a memory-efﬁcient manner.
The Legendre Memory Unit (Voelker et al., 2019) can be seen as a projection of data onto Legendre polynomials within the recurrence equation of RNNs.
Our method builds upon the HiPPO framework and its S4 extension (Gu et al., 2020; 2021), which present an overarching generalization of this and related (Zhang et al., 2018) ideas to Legendre, Laguerre or Chebyshev polynomials.
LMUs, HiPPO and S4 all rely on a linear interpolation in between observations and perform the projection on the hidden space of the model.
This prevents this approach from producing accurate time series reconstruction in general and informative embedding for sparsely observed time series.
Despite attempts to improve the computational performance of learning from long-term sequences (Morrill et al., 2021), we are not aware of approaches that leverage orthogonal polynomials to improve long-term memory for neural ODE models.
B ACKGROUND
Orthogonal Polynomial Projections: Orthogonal polynomials are deﬁned with respect to a measure µ as a sequence of polynomials { P 0 ( s ) , P 1 ( s ) , ... } such that deg ( P i ) =
i and
[Equation]
where ↵ n are normalizing scalars and  is the Kronecker delta.
For simplicity, we consider only absolutely continuous measures with respect to the Lebesgue measure, such that there exists a weight function !
( · ) such that dµ ( s ) = !
( s ) ds .
The measure µ determines the class of polynomials obtained from the conditions above (Eq. 1).
Examples include Legendre, Hermite or Laguerre classes of orthogonal polynomials.
The measure µ also deﬁnes an inner product h · , · i µ such that the orthogonal projection of a 1-dimensional continuous process f ( · ) : R !
R on the space of polynomials of degree N , P N is given as
[Equation]
This projection minimizes the distance k f  p k µ for all p 2 P N and is thus optimal with respect to the measure µ .
One can thus encode a process f by storing its projection coefﬁcients { c 0 , ..., c N } .
We write the vector of coefﬁcients up to degree N as c (the degree N is omitted) and c i =
c i .
Intuitively, the measure assigns different weights at times of the process and thus allows for modulating the importance of different parts of the input signal for the reconstruction.
Continuous update of approximation coefﬁcients: The projection of a process f onto a basis of orthogonal polynomials provides an optimal representation for reconstruction.
However, there is often a need to update this representation continuously as new observations of the process f become available.
Let f <t be the temporal process observed up until time t .
We wish to compute the coefﬁcients of this process at different times t .
We can deﬁne for this purpose a time-varying measure µ t and corresponding weight function !
t that can incorporate our requirements in terms of reconstruction abilities over time.
For instance, if one cares about reconstruction of a process  temporal units in the past, one could use a time-varying weight function !
t ( s ) =
I [ s 2 ( t   , t )] and corresponding measure.
This time-varying measure induces a time-varying basis of orthogonal polynomials P tn for n = 0 , ..., N .
We can deﬁne the time-varying orthogonal projection and its coefﬁcients c n ( t ) as
[Equation]
Dynamics of the projection coefﬁcients:
Computing the coefﬁcients of the projection at each time step would be both computationally wasteful and would require storing the whole time series in memory, going against the principle of sequential updates to the model.
Instead, we can leverage the fact that the coefﬁcients evolve according to known linear dynamics over time.
Remarkably, for a wide range of time-varying measures µ t , Gu et al.
(2020) show that the coefﬁcients c N ( t ) follow:
[Equation]
where A µ and B µ are ﬁxed matrices (for completeness, we provide a derivation of the relation for the translated Legendre measure in Appendix A).
We use the translated Legendre measure in all our experiments.
Using the dynamics of Eq.
4, it is possible to update the coefﬁcients of the projection sequentially by only using the new incoming sample f ( t ) , while retaining the desired reconstruction abilities.
Gu et al.
(2020) use a discretization of the above dynamics to model discrete timesequential data via a recurrent neural network architecture.
Speciﬁcally, their architecture projects the hidden representation of an RNN onto a single time series that is projected onto an polynomial basis.
Our approach differs in two ways.
First, we work with a continuous time model.
Second, we jointly model the evolution of d -dimensional time-varying process as a overparameterized hidden representation that uses orthogonal projections to serve as memory banks.
The resulting model is a new neural ODE architecture as we detail below.
M ETHODOLOGY
Problem Setup.
We consider a collection of sequences of temporal observations x
= { ( x i , m i , t i ) :
i 2 { 1 , ..., T }} that consist of a set of time-stamped observations and masks ( x i 2 R d , m i
R d , t i 2 R ) .
We write x i,j and m i,j for the value of the j th dimension of x i and m i respectively.
The mask m i encodes the presence of each dimension at a speciﬁc time point.
We set m i,j =if x i,j is observed and m i,j = 0 otherwise.
The number of observations for each sequence x , T , can vary across sequences.
We deﬁne the set of sequences as S and the distance between two time series observed at the same times as d ( x , x 0 )
= 1 T
P
Ti k x i  x 0 i k 2 .
Our goal is to be able to embed a sequence x into a vector h 2 R d h such that (1) h retains a maximal amount of information contained in x and (2) h is informative for downstream prediction tasks.
We formalize both objectives below.
Deﬁnition (Reverse reconstruction) .
Given an embedding h t of a time series x at time t , we deﬁne the reverse reconstruction ˆx <t as the predicted values of the time series at times prior to t .
We write the observed time series prior to t as x <t .
Objective 1 (Long memory representation) .
Let h t and h 0 t be two embeddings of the same time series x .
Let ˆ x <t and ˆ x 0 <t be their reverse reconstruction.
We say that h t enjoys more memory than h 0 t if d (ˆ x <t , x <t ) <
d (ˆ x 0 <t , x <t ) .
Objective 2 (Downstream task performance) .
Let y 2 R d
y be an auxiliary vector drawn from a unknown distribution depending on x .
Let ˆ y ( x ) and ˆ y ( x ) 0 be the predictions obtained from embeddings h t and h 0 t .
For a performance metric ↵ : S ⇥ R d y !
R , we say that h t is more informative than h 0 t if E x , y [ ↵ (ˆ y ( x ) , y )]
> E x , y [ ↵ (ˆ y ( x ) 0 , y )] .
4. P OLY ODE: A NAMNESIC N EURAL ODE S
We make the assumption that the observed time series x comes from an unknown but continuous temporal process x ( t ) .
Given h ( t ) 2 R d h and a read-out function g : R d h !
R d we posit the following generative process for the data:
[Equation]
where part of  ( · ) is parametrized via a neural network  ✓ ( · ) .
The augmentation of the state space is a known technique to improve the expressivity of Neural
ODEs (Dupont et al., 2019).
Here, to ensure that the hidden representation in our model has the capacity to retain long-term memory, we augment the state space of our model by including the dynamics of coefﬁcients of orthogonal polynomials as described in Equation 4.
We structure the hidden state as h ( t ) =
[ h 0 ( t ) , h 1 ( t ) , . . .
, h d ( t )] where h 0 ( t ) 2 R d has the same dimension as the input process x , h i ( t ) 2 R N , 8 i 2 1 , . . .
, d , has the same dimension as the vector of projection coefﬁcients
c i ( t ) and [ · , · ] is the concatenation operator.
We deﬁne the readout function g i ( · ) : R ( N +1) d !
R such that g i ( h ( t ))
=
h 0 ( t ) i .
That is, g i is ﬁxed and returns the i th value of the input vector.
This leads to the following system of ODEs that characterize the evolution of h ( t ) :
Integration Step.
[Equation]
This parametrization allows learning arbitrarily complex dynamics for the temporal process x .
We deﬁne a sub-system of equations of projection coefﬁcients update for each dimension of the input temporal process x ( t ) 2 R d .
This sub-system is equivalent to Equation 4, where we have substituted the input process by the prediction from the hidden process h ( t ) through a mapping g
i ( · ) .
The hidden process h 0 ( t ) acts similarly as in Neural ODEs and the processes c ( t ) captures long-range information about the observed time series.
At each time step, we can provide an estimate of the time series ˆ x ( t ) conditioned on the hidden process h ( t ) , with ˆ x ( t ) =
g ( h ( t )) .
[Figure]
The system of equations in Eq.
6 characterises the dynamics in between observations.
To easily update the state of the system when a new observation becomes available, This construction allows for the processing of new observations as follows.
Update Step.
At time t = t i , after observing x i and mask m i , we set
[Equation]
Remarks: Our model blends orthogonal polynomials with the ﬂexibility offered in modelling the observations with NeuralODEs.
The consequence of this is that while the coefﬁcients serve as memory banks for each dimension of the time series, the Neural ODE over h 0 ( t ) can be used to forecast from the model.
That said, we acknowledge that a signiﬁcant limitation of our current design is the need for the hidden dimension to track N coefﬁcients for each time-series dimension.
Given that many adjacent time series might be correlated, we anticipate that methods to reduce the space footprint of the coefﬁcients within our model is fertile ground for future work.
4. T RAINING
We train this architecture by minimizing the reconstruction error between the predictions and the observations: L =
P
Ti =1 k ˆ x ( t i )  x i k 22 .
We ﬁrst initialize the hidden processes c (0)
=
and h (0)
=
0 though they can be initialized with static information b , if available ( e.g. h (0)
= ✓ ( b ) ).
We subsequently alternate between integration steps between observations and update steps at observation times.
The loss is updated at each observation time t i .
A pseudo-code description of the overall procedure is given in Algorithm 1.
Numerical integration.
We integrate the system of differential equations of Equation 6 using differentiable numerical solvers as introduced in Chen et al.
However, one of the technical challenges that arise with learning PolyODE is that the dynamical system in Equation 6 is relatively stiff and integrating this process with acceptable precision would lead to prohibitive computation times with explicit solvers.
This arises because a subset of this dynamical system is linear inhomogeneous and can be expressed in the following form: dx ( t ) dt
=
A µ x ( t ) + f ( t ) .
Crucially, the real part of all eigenvalues of A are negative and the stiffness ratio, deﬁned as the ratio between the largest and lowest real parts of the eigenvalues of A grows large with the degree of the polynomial approximation N .
This is characteristic of stiff systems that will exhibit instabilities at large integration step sizes.
To deal with this instability we used an implicit solver such as Backward Euler or Adams-Moulton for the numerical integration Sauer (2011).
A comparison of numerical integration schemes and an analysis of the stability of the ODE are available in Appendix H.
Data: x , matrices A µ , B µ , number of dimensions d , number of observations T , number of polynomial coefﬁcients N
Result:
Training loss L over a whole sequence x t ⇤
Initialize h j (0) =
c j (0) = 0 N , 8 j 2 1 , ..., d ,
Loss L =for i 1 to T do
Integrate c 1 ,...,d ( t ) and h 0 ,..,d ( t ) from t =
t ⇤ until t = t
i
ˆ x i h 0 ( t ⇤ )
Update c 1 ,...,d ( t i ) and h 0 ,...,d ( t i ) with x i , m i .
[Equation]
t [Equation] end
Forecasting: From time t , we forecast the time series at an arbitrary time t ⇤ as:
[Equation]
where  ✓ ( · ) is the learned model that we use in the integration step and introduced in Eq.
Reverse Reconstruction: Using Equation 3, we can compute the reverse reconstruction of the time series at any time t using the projection coefﬁcients part of the hidden process:
[Equation]
The error between the prediction obtained during the integration step, ˆ x ( t ) , and the above reconstruction estimator is bounded above, as Result 4.1 shows.
The result addresses how well we should expect to perform reverse reconstructions after integrating forward in time over irregularly observed intervals.
Result 4.1.
For a shifted rectangular weighting function with width  , !
t ( x ) = 1  I [ t   ,t ] (which generate Legendre polynomials), the mean square error between the forward ( ˆ x ) and reverse prediction ( ˆ x <t ) at each time t is bounded by:
[Equation]
where K is the number of observations in the interval [ t   , t ] , L is the Lipschitz constant of the forward process, N is the degree of the polynomial approximation and ⇠ ( · , · ) is the Hurwitz zeta function.
S K =
[Equation]
is the sum of absolute errors between the forward process and observations incurred at the update steps.
C 0 , C 1 and C 2 are constants.
Expectedly, the bound goes to 0 as the degree of the approximation increases.
The lower cumulative absolute error S K also leads to a reduction of this bound.
As the cumulative absolute error S K and our loss function L share the same optimum, for ﬁxed  , L , K and N , our training objective therefore implicitly enforces a minimization of the reconstruction error.
This corresponds to optimizing
Objective 1, where we set d ( x , x 0 ) =
k x  x 0 k 2 µ t .
Our architecture thus jointly minimizes both global reconstruction and global reconstruction error.
Notably, when S K = 0 , this result boils down to the well-known projection error for orthogonal polynomials projection of continuous processes (Canuto and Quarteroni, 1982).
What is more, increasing the width of the weighting function (increasing  ) predictably results in higher reconstruction error.
However, this can be compensated by increasing the dimension of the polynomial basis accordingly.
We also note a quadratic dependency on the Lipschitz constant of the temporal process, which can limit the reverse reconstruction abilities for high-frequency components.
The full proof can be found in Appendix B.
E XPERIMENTS
We evaluate our approach on two objectives : (1) the ability of the learned embedding to encode global information about the time series, through the reverse reconstruction performance (or memorization) and (2) the ability of embedding to provide an informative input for a downstream task.
We study our methods on the following datasets:
Synthetic Univariate .
We validate our approach using a univariate synthetic time series.
We simulate 1000 realizations from this process and sample it at irregularly spaced time points using a Poisson point process with rate  [ 1 s ] .
For each generated irregularly sampled time series x , we create a binary label y
=
I
[ x (5) > 0 .
Further details about this dataset are to be found in Appendix F.
Chaotic Attractors .
Chaotic dynamical systems exhibit a large dependence of the dynamics on the initial conditions.
This means that a noisy or incomplete evaluation of the state space may not contain much information about the past of the time series.
We consider two widely used chaotic dynamical systems:
Lorenz63 and a 5-dimensional Lorenz96.
We generate 1000 irregularly sampled time series from different initial conditions.
We completely remove one dimension of the time series such that the state space is never fully observed.
This forces the model to remember the past trajectories to create an accurate estimate of the state space at each time t .
MIMIC-IIII dataset .
We use a pre-processed version of the MIMIC-III dataset (Johnson et al., 2016; Wang et al., 2020).
This consists of the ﬁrst 24 hours of follow-up for ICU patients.
We use the following longitudinal variables: heart rate, mean and diastolic blood pressure, oxygen saturation, respiratory rate, glucose, blood urea nitrogen, white blood cell count, temperature and creatinine.
For each time series, the label y is the in-hospital mortality.
Baselines: We compare our approach against two sets of baselines: Neural ODEs architecture and variants of recurrent neural networks architectures designed for long-term memory.
To ensure a fair comparison, we use the same dimensionality of the hidden state for all models.
Neural ODE baselines .
We use a ﬁltering implementation of Neural ODEs, GRU-ODE-Bayes (De Brouwer et al., 2019) and ODE-RNN Rubanova et al.
an auto-encoder relying on a
Neural ODE for both the encoder and the decoder part.
Because of the additional auto-encoder loss, we expect this approach to provide more informative time-series embeddings.
For theses baselines, we compute the reverse reconstruction by integrating the system of learnt ODEs backward in time.
In case of ODE-RNN, we use the ODE of the decoder.
Additionally, we compare against Neural
RDE Morrill et al.
(2021), a implementation of neural controlled differential equations for long time series.
Long-term memory RNN baselines .
We compare against HiPPO-RNN (Gu et al., 2020), a recurrent neural network architecture that uses orthogonal polynomial projections of the hidden process.
We also use a variant of this approach where we directly use the HiPPO operator on the observed time series, rather than on the hidden process.
We call this variant HiPPO-obs .
We also compare against S4, an efﬁcient state space model relying on the HiPPO matrix (Gu et al., 2021).
Long-range representation learning:
For each dataset, we evaluate our method and the various baselines on different tasks.
Implementation details are available in Appendix G.
Downstream Classiﬁcation.
We train the models on the available time series.
After training, we extract time series embedding from each model and use them as input to a multi-layer perceptron trained to predict the time series label y .
We report the area under the operator-characteristic curve evaluated on a left-out test set with 5 repetitions.
Time Series Reconstruction.
Similarly as for the downstream classiﬁcation, we extract the time series embeddings from models trained on the time series.
We then compute the reverse reconstruction ˆ x <t and evaluate the MSE with respect to the true time series.
Forecasting.
We compare the ability of all models to forecast the future of the time series.
We compute the embedding of the time series observed until some time t cond and predict over a horizon t horizon .
We then report the MSE between the prediction and true trajectories.
[Table]
Results for these tasks are presented in Table 1 for Synthetic and Lorenz datasets and in Table 2 for
MIMIC.
We report additional results in Appendix C, with a larger array of irregular sampling rates.
We observe that the reconstruction abilities of PolyODE clearly outperforms the other baselines, for all datasets under consideration.
A similar trend is to be noted for the downstream classiﬁcation for the synthetic and Lorenz datasets.
For these datasets, accurate prediction of the label y requires a global representation of the time series, which results in better performance for our approach
[Table] [Table]
For the MIMIC dataset, our approach compares favourably with the other methods for the downstream classiﬁcation objective and outperforms other methods for trajectory forecasting.
What is more, the reconstruction ability of PolyODE is signiﬁcantly better than compared approaches.
To showcase this ability further, we plot in Figure 3 the reverse reconstructions of PolyODEand the true values (dotted lines) for the heart rate, mean blood pressure and oxygen saturation of a random patient over the ﬁrst 24 hours in the ICU.
This reconstruction is obtained by ﬁrst sequentially processing the time series until t = 24 hours and subsequently using the hidden process to reconstruct the time series as in Equation 9.
We observe that PolyODE can indeed capture the overall trend of the time series over the whole history.
Ablation study: the importance of the auxiliary dynamical system: Is there utility to leveraging the neural network  ✓ ( · ) to learn the dynamics of the time series?
How well would various interpolation schemes for irregularly sampled observations perform in the context of reverse reconstruction and classiﬁcation?
In response to these questions, we ﬁrst note that they do not support extrapolation and are thus incapable of forecasting the future of the time series.
However, we compare the performance in terms of reverse reconstruction and classiﬁcation in Table 3.
We consider constant interpolation (last observation carried forward), linear interpolation and Hermite spline interpolation.
Our results indicate a signiﬁcant gap in performance between PolyODE and the linear and constant interpolation schemes.
The Hermite spline interpolation allows us to capture most of the signal needed for the downstream classiﬁcation task but results in signiﬁcantly lower performance in terms of the reverse reconstruction error.
These results therefore strongly support the importance of  ✓ ( · ) for producing informative time series embeddings.
Complementary results are available in Appendix C.
[Figure]
[Table]
Incorporating global time series uncertainty:
Previous experiments demonstrate the ability of PolyODE to retain memory of the past trajectory.
A similar capability can be obtained for capturing global model uncertainties over the time series history.
This allows having speciﬁc uncertainty estimates at different points of the reconstruction.
In Figure 4, we evaluate the association between the recovered uncertainties of PolyODE and the reverse reconstruction errors.
We plot the predicted uncertainties against the root mean square error (RMSE) on a logarithmic scale.
We compare our approach with using the uncertainty of the model at the last time step only.
We observe that the uncertainties recovered by PolyODE are signiﬁcantly more correlated with the errors (Pearson- ⇢ = 0 . 56 ) compared to using the uncertainties obtained from the last time step (Pearson- ⇢ = 0 . 11 ).
More details are available in Appendix E.
C ONCLUSION
Producing time series representations that are easy to manipulate, representative of global dynamics, practically useful for downstream tasks and robust to irregular sampling remains an ongoing challenge.
In this work, we took a step in that direction by proposing a simple but novel architecture that satisﬁes those requirements by design.
As a Neural ODE, PolyODE inherits the ability to handle irregular time series elegantly but at the same time, PolyODE also incurs computational cost associated with numerical integration.
Currently, our approach also requires a large hidden space dimension and ﬁnding methods to address this that exploit the correlation between dimensions of the time series is a fruitful direction for future work.
Reproducibility Statement Details for reproducing experiments shown are available in
Appendix G.
The code for reproducing all experiments will be made publicly available.