L EARNING L ATENT D YNAMICS FOR P ARTIALLY -O BSERVED C HAOTIC S YSTEMS
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper addresses the data-driven identiﬁcation of latent Ordinary Differential
Equation (ODE) representations of partially-observed dynamical systems, i.e. dynamical systems whose some components are never observed, with an emphasis on forecasting applications and long-term asymptotic patterns.
Whereas stateof-the-art data-driven approaches rely on an explicit mapping of the observation series onto a latent space, the proposed approach only relies on the deﬁnition of an augmented space, higher-dimensional than the manifold spanned by the observed variables, where the dynamics of the observations can be fully described by an ODE.
From a numerical point of view, the proposed approach exploits a neural-network ODE representation and the associated variational minimization scheme.
Numerical experiments support the relevance of the proposed framework w.r.t.
state-of-the-art approaches, including neural ODE schemes, both in terms of short-term forecasting errors and long-term behaviour.
We further discuss how the proposed framework relates to Koopman operator theory and Takens’ embedding theorem.
I NTRODUCTION
Learning the underlying dynamical representation of some observed variables x t ∈ R n (where t ∈ { t 0 , ..., T } is the temporal sampling time and n the dimension of the observations) is a key challenge in various scientiﬁc ﬁelds, including control theory, geoscience, ﬂuid dynamics, economics; for applications ranging from system identiﬁcation to forecasting and assimilation issues (Lai & Wei, 1982; Abarbanel & Lall, 1996; Jeong & Hussain, 1995; Koopmans, 1949).
For fully-observed systems, i e. when the observed variables x t relate to some underlying deterministic states s t according to a diffeomorphic mapping possibly corrupted by noise, recent advances (Brunton et al., 2016b; Fablet et al., 2018; Nguyen et al., 2019) have shown that one can identify the governing equations of the dynamics of s t from a representative dataset of observations { x t i } i .
Unfortunately, when the observed variables x t only relate to some but not all the components of underlying states s t , these approaches can not apply since no ODE or, more generally, no one-toone mapping deﬁned in the observation space can represent the time evolution of the observations.
In this context, Takens’s theorem states the conditions under which a delay embedding, formed by lagged versions of the observed variables, guarantees the existence of governing equations in the embedded space (Takens, 1981).
Takens’s theorem has motivated a rich literature of machine learning schemes to identify dynamical representations of partially-observed systems using a delay embedding.
This comprises both nonparametric schemes based on nearest-neighbors or analogs (Abarbanel, 1996a) as well as parametric schemes which include polynomial representations (Paduart et al., 2010), neural network models (Wan, 1993) and Support Vector Regression (SVR) models (Mattera & Haykin, 1999; M¨uller et al., 1999).
For all these approaches, the identiﬁcation of the appropriate delay embedding is a critical issue (Abarbanel, 1996b;c).
From a neural network and machine learning perspective, the inference of a latent space, within a State Space Model (SSM) framework, for dynamical systems has motivated a broad literature especially for time series forecasting (Ghahramani & Roweis, 1999;
Wang et al., 2006; Mirowski & LeCun, 2009; He et al., 2015; Krishnan et al., 2016).
Most of those techniques were introduced in the context of reduced order modeling (ROM) to infer low-dimensional manifolds, where the dynamics of the observations can be represented.
When considering partially-observed systems, these approaches state this issue as the inference of a (non-linear) projection of an input sequence in a latent space where the observations can be modeled.
This projection is usually computed in a probabilistic framework using Bayesian ﬁltering techniques.
However, recovering the attractor’s dynamics using iterative predictions is still an issue for such models since the explicit modeling of latent space as a delay embedding of the observations may limit the expressiveness of the latent states, especially when considering chaotic dynamics.
In this work, we show that we do not need to rely explicitly neither on a delay embedding nor on the learning of an inference model to map the observation series to a latent space.
We only assume that we may deﬁne an augmented space, higher-dimensional than the manifold spanned by the observed variables, where the dynamics of the observations can be fully described by an ODE.
Using neural-network representations for the parametrization of the dynamical model, it amounts to jointly learning the governing ODE and reconstructing the augmented latent states for a given observation dataset using a variational framework.
We report experiments on linear and chaotic dynamics, which illustrate the relevance of the proposed framework compared to state-of-the-art approaches.
We then further discuss the key features of this framework with respect to state-of-theart dynamical systems identiﬁcation tools such as Koopman operator theory (Koopman, 1931).
B ACKGROUND AND R ELATED W ORK
This section introduces the learning of dynamical representations for partially-observed systems and links this problem to recent advances in machine learning.
Let us consider an unobserved state variable s governed by an autonomous system of m differential equations ˙
s t = f
( s t ) .
Let us also assume that this system generates a ﬂow Φ t i ( s t 0 )
= (cid:82) t i t
f
( z w ) dw ∈ R m with trajectories that are asymptotic to a limit-cycle L of dimension d contained in R m .
We further assume that we are provided with a measurement function H that maps our state variable s to our observations
[Equation]
When considering the data-driven identiﬁcation of a dynamical mapping that governs some observation data, we ﬁrst need to evaluate whether the dynamics in the observation space can be described using a smooth 1 ODE.
Another way to tackle this question is to ﬁnd the conditions under which the deterministic properties of the unobserved limit-cycle L are preserved in the observation space in R n such that one can reliably perform forecasts in the observation space.
The general condition under which a mapping H preserves the topological properties of the initial limit-cycle involves a differential structure.
Assuming that L is a smooth compact differential manifold, the topological properties of L are preserved through a mapping H in R n if H is one-to-one and is an immersion of L in R n .
Under these conditions our observation mapping is called an embedding (Sauer et al., 1991).
The simplest example of an embedding involves an identity observation operator H .
With this embedding, we have direct access to the state variable s which is governed by a deterministic ODE.
This particular case has been widely studied in the literature.
Parametric representations have been for decades the most popular models thanks to their simplicity and interpretability (Paduart et al., 2010), (Brunton et al., 2016b).
Recently, these approaches have been enriched by neural network and deep learning schemes (Wiewel et al., 2018), (Raissi et al., 2018).
In particular, the link between residual networks (Chen et al., 2018; Ouala et al., 2019) and numerical integration schemes have opened new research avenues for learning extremely accurate dynamical models even from irregularly-sampled training data.
These schemes show greater interpretability and forecasting performance for the datadriven representation of systems governed by an ODE, compared with other state-of-the-art neural networks schemes, including Recurrent Neural Networks (RNN) such as LSTM (Long-Short-Term
Memory).
However, for a wide range of real-world systems, we are never provided with an observation operator that forms an embedding of the unobserved dynamical system.
In such situations, we do not have any guarantee on the existence of a smooth ODE that governs the temporal evolution of our observations.
From this point of view, the question of ﬁnding an appropriate dynamical representa- tion of some observed data may not be this straightforward.
The fact that our data may come from some unobserved governing equation may restrict the use of the above-mentioned state-of-the-art algorithms.
The main difﬁculty lies in the ability to map observation series to a latent space that provides at least a one-to-one mapping between two successive states.
From a geometrical point of view, the time delay theorem (Takens, 1981) provides a way to build a latent space that preserves the topological properties of the true (unobserved) dynamics limit-cycle.
A generalization of this theorem (Sauer et al., 1991) shows that one can reconstruct topologically similar limit-cycles using any appropriate smooth composition map of the observations.
The derivation of a dynamical system from such representations however encounters large disparities since no explicit relationships between the deﬁned phase space and an ODE formulation have been clearly made.
Classical state-of-the-art techniques such as polynomial representations (Brunton et al., 2016b) and K-Nearest
Neighbors (KNN) (Lguensat et al., 2017) algorithms were proposed but they often fail to achieve both accurate short-term forecasting performance and long-term topologically similar reconstructed limit-cycles (see experiments for an illustration).
We may also point out that the limitation of ODE-based representation in deep learning architecture has also been pointed out recently in (Dupont et al., 2019; Zhang et al., 2019) for classiﬁcation issues.
As ODE-derived trajectories do not intersect, it may limit the ability of neural ODE representations to reach relevant classiﬁcation performance in a given feature space.
To address this issue, (Dupont et al., 2019) and (Zhang et al., 2019) propose to consider an augmented state, simply by augmenting the observed state by a number of zeros to create a high-dimensional space in which an ODE representation can be identiﬁed.
Such a strategy cannot apply to time series modeling as successive augmented states cannot be forced to zero for some dimensions.
Advances in the inference of latent spaces in state space models was introduced essentially, from a dynamical systems perspective, to retrieve low-dimensional manifolds, where the dynamics of the system evolve.
When applied to partially-observed systems, the latent variables are typically inferred from a sequence of observations through a parametric modeling of the posterior distribution as in (He et al., 2015; Krishnan et al., 2016;
Chen et al., 2018) or through marginalization with model constraints as in (Ghahramani & Roweis, 1999; Wang et al., 2006)
However, such models often fail in accounting for long term patterns (as shown in the experiments).
This is due to the fact that the latent space is constrained to be a non-linear projection of a sequence of observations, which limits the expressiveness of the dynamical model.
Interestingly, (Mirowski & LeCun, 2009) does not involve the learning of an inference model as the reconstruction of the latent states is solved as gradient-based minimization of the dynamical prior w.r.t.
an observation series.
However, the dynamical prior relies on an explicit delay embedding as the dynamics of the latent state depend both on the previous latent state and on a delay embedding of the observations.
In this work, we address the identiﬁcation of a latent embedding, associated with an ODE representation, for partially-observed systems.
The core idea of this work is to infer an augmented latent space, governed by an ODE, which fully explain the observed time series and their dynamics.
In contrast to previous work (He et al., 2015; Krishnan et al., 2016; Chen et al., 2018; Ghahramani & Roweis, 1999; Wang et al., 2006), we do not exploit either a delay embedding or an explicit modeling of the inference model (i.e., the reconstruction of the latent states given the observed time series).
As such, our scheme only involves the selection of the class of ODEs of interest.
The expected beneﬁts are as follows: (i) our model ensures the existance of a latent embedding associated with an ODE, which may not be guaranteed when considering a parametric inference model and/or a delay embedding, (ii) our model reduces the complexity of the overall scheme to the complexity of the ODE representation, (iii) our model guarantees the consistency of the reconstructed latent states w.r.t.
the learnt ODE.
Our experiments suggest that this may be of key importance to jointly address short-term forecasting performance and long-term chaotic patterns.
L EARNING LATENT REPRESENTATIONS OF PARTIALLY - OBSERVED DYNAMICS
Augmented latent dynamics : Let us consider a continuous m -dimensional dynamical system s t governed by an autonomous ODE ˙ s t
= f
( s t ) with Φ t the corresponding ﬂow Φ t ( s t 0 ) =
(cid:82) t t 0 f
( s w ) dw .
We assume the state s t to be never fully-observed.
Formally, we can deﬁne an obser- vation function H : R m −→ R n such that the observation x t follows x t
= H ( s t ) .
An example of interest comprises the situation when one observes only the n ﬁrst variables of s t .
Our goal is to derive an ODE representation of x t ∈ R n .
However, the key question arising here is the extent to which the dynamics expressed in the observations space, reﬂect the true underlying dynamics in R m , and consequently, the conditions on H under which the predictable deterministic dynamical behavior of the hidden states is still predictable in the observations space.
To illustrate this issue, we may consider a linear dynamical system in the complex domain governed by the following linear ODE:
[Equation]
with s ∈ C a state variable and α ∈ C a complex imaginary number.
The solution of this problem is
[Equation]
with K a constant depending on s 0 .
Let us assume now that we are only provided with the real part as direct measurements of the unobserved state i.e. H ( . ) =
[Equation]
: x
[Equation]
s
[Equation]
Proposition 1 : The ﬂow of an ODE cannot represent the time evolution of x t .
The proof of the proposition is given in the appendix.
The intuition exempliﬁed for the 2dimensional system given by (1) is as follows.
If we assume that we are only provided the real part as direct measurements x t ∈ R of the true states s t , no smooth autonomous ODE model in the scalar observation space can describe the trajectories of the observations as the mapping between two observations is not one-to-one.
For example, assuming that s t 0 and s t 1 correspond to two states that have the same real part but distinct imaginary parts, the associated observed states are equal x t
= x t 1 .
However, the time evolution of the states s t 0 and s t 1 differ if they have different imaginary parts.
The observed states x t 0 + δ and x t 1 + δ are then no longer equal after any time increment δ .
As a consequence, a given observation may have more than one future state and this behavior cannot be represented by a smooth ODE in the observation space.
Hence, the application of an ODE mapping such as (Chen et al., 2018) and (Fablet et al., 2018) for such observations will lead to poor forecasting performance.
From a naive Neural networks point of view, ﬁtting such a model will most likely force the forecasting into an equilibrium point since we may iteratively match the same inputs to different output predictions.
This simple example emphasizes the need for considering augmented latent states.
In this context, Takens’s theorem guarantees the existence of an augmented space, in which a oneto-one mapping exists between successive time steps (Takens, 1981).
More precisely, for a given observation operator H of a deterministic underlying dynamical system that governs s t , one can deﬁne a delay embedding from an observation sequence, i.e. { x t 0 − aδ , x t 0 , . . .
, x t 0 + bδ } with δ the time delay and a, b two natural numbers.
An ODE representation in a latent space can than be formulated according to (Chen et al., 2018) by sampling the latent states from a parametrized posterior distribution given the delay embedding time series.
Here, we aim to identify an augmented latent space without resorting to an explicit mapping from the observed series to the latent space, such that the latent dynamics are governed by a smooth ODE and can be mapped to the observations.
Formally, let us introduce the following notations.
Let u t ∈ R d E deﬁne a d E -dimensional augmented latent state formed by an observed component 2 x t and an unobserved latent component referred to as z t .
[Equation]
Assuming that the augmented latent state u t is governed by an ODE leads to the the following state space model :
[Equation]
where the dynamical operator f θ belongs to a family of smooth operators (in order to guarantee uniqueness (Coddington & Levinson, 1955)) parametrized by θ .
G is a projection matrix that satisﬁes x t =
G ( u t ) .
It means that operator G only returns the observed component of the augmented latent state.
Compared with previous work, the key difference lies in the deﬁnition of the augmented latent space, where only the observed component is explicitly stated as a function of observation x t .
Similarly to neural ODE frameworks (Chen et al., 2018; Fablet et al., 2018), we consider a neural-network representation with Lipschitz nonlinearities and ﬁnite weights for ODE operator f θ .
As detailed in the next sections, we address the identiﬁcation of operator f θ from a dataset of observed state series { x 0 , . .
, x T } as well as the exploitation of the identiﬁed latent dynamics for the forecasting of the time evolution of the observed states, for instance unobserved future states { x T +1 , . . .
, x T + N } .
Both issues involve the joint inference of the unobserved component z t of the augmented latent states.
This formulation generalizes to reduced-order modeling (ROM) (Champion et al., 2019), where observation x t fully derives from a lower-dimensional variable.
We let the reader refer to the appendix for the associated generalization of the proposed framework.
Learning scheme : Given an observation time series { x 0 , . . .
, x T } we aim to identify the statespace model deﬁned by (4), which amounts to learning the parameters θ of the dynamical operator f θ .
However, as the component z t of the augmented state u t is never observed, this identiﬁcation requires the joint optimization of the model parameters θ as well as of the hidden component z t .
Formally, this problem is stated as the following minimization of the forecasting error on observed variables:
[Equation]
Subject to
[Equation]
(5) with Φ θ,t the one-step-ahead diffeomorphic ﬂow associated with operator f θ such that
[Equation]
In (5), the loss to be minimized involves the one-step-ahead forecasting error for the observed variable x t .
The constraints state that the augmented latent state sequence { u 0 , . .
, u T } deﬁned by (3) should be a solution of the ODE (4).
Here, we numerically minimize the equivalent formulation:
[Equation]
where u Tt =
[ x Tt , z Tt ] and λ a weighting parameter.
The term (cid:107) u t − Φ θ,t ( u t − 1 ) (cid:107) 2 may be regarded as a regularization term such that the inference of the unobserved component z t of the augmented state u t is not solved independently for each time step.
Using a neural-network parametrization for the ODE operator f θ , the corresponding forecasting ﬂow Φ θ,t is also stated as a neural network based on a numerical integration scheme formulation (typically a 4 th -order Runge-Kutta scheme).
This architecture, very much similar to a ResNet (He et al., 2015), allows very accurate identiﬁcation of ODE models (Fablet et al., 2018; Ouala et al., 2019).
Hence, for a given observation sequence { x 0 , . . .
, x T } , we minimize ( 6 ) jointly w.r.t.
θ and unobserved variables { z 0 , . . .
, z T } .
Application to forecasting : We also apply the proposed framework to the forecasting of the observed states x t .
Given a trained latent dynamical model (4), forecasting future states for x t relies on the forecasting of the entire augmented latent state u t .
The latter amounts to determining an initial condition of the unobserved component z t and performing a numerical integration of the trained
ODE (4).
Let us denote by x nt , t ∈ { t 0 , ..., T } a new series of observed states.
We aim to forecast future states x nt , t ∈ { T + 1 , ..., T + δT } .
Following (6), we infer the unobserved component (cid:98)
z T of the augmented state u nT at time T from the following minimization:
[Equation]
Here, we only minimize w.r.t.
latent variables { z nt } given the trained forecasting operator Φ θ,t .
This minimization relates to a variational assimilation issue with partially-observed states and known dynamical and observation operators (Lynch & Huang, 2010).
Similarly to the learning step, we beneﬁt from the neural-network parameterization of operator Φ θ,t and from the associated automatic differentiation tool to compute the solution of the above minimization using a gradient descent.
We may consider different initialization strategies for this minimization problem.
Besides a simple random initialization, we may beneﬁt from the information gained on the manifold spanned by the unobserved components during the training stage.
The basic idea comes to assume that the training dataset is likely to comprise state trajectories which are similar to the new ones.
As the training step embeds the inference of the whole latent state sequence, we may pick as initialization for minimization (7) the inferred augmented latent state in the training dataset which leads to the observed state trajectory that is the most similar (in the sense of the L2 norm) to the new observed sequence x nt .
The interest of this initialization scheme is two-fold: (i) speeding-up the convergence of minimization (7) as we expect to be closer to the minimum; (ii) considering an initial condition which is in the basin of attraction of the reconstructed limit-cycle.
The latter may be critical as we cannot guarantee that the learnt model does not involve other limit-cycles than the ones truly revealed by the training dataset, which may lead to a convergence to a local and poorly relevant minimum.
N UMERICAL EXPERIMENTS
In this section, we report numerical experiments to illustrate the key features of proposed framework.
We consider three case-studies: a linear ODE case-study; a chaotic system, namely Lorenz-63 dynamics, and real upper ocean data.
Application to a linear ODE :
In order to illustrate the key principles of the proposed framework, we consider the following linear ODE in the complex domain:
[Equation]
(8) with α = − 0 .
1 − 0 .
5 j , j 2 = − 1 and s 0 = 0 .
As α ∈ C with Real ( α ) < 0 and s 0 (cid:54) = 0 , the solution of this ODE is an ellipse in the complex plane (Fig. 1).
As observations, we consider the real part of the underlying state, i.e. the observation function
H : C −→ R is given by x t
= Real ( s t ) .
This is a typical example, where the mapping between two successive observations is not a one-to-one mapping since all the states that have the same real part lead to the same observation.
As explained in section 3, one cannot identify an autonomous ODE model that will reproduce the dynamical behavior of the observations in the observations space.
We apply the proposed framework to this toy example.
We consider a 2-dimensional augmented state u t =
[ x t , z 1 t ] .
As neural-network parametrization for operator f θ , we consider a neural network with a single linear fully-connected layer.
We use an observation series of 10000 time steps as training data.
As illustrated in Fig.1, given the same initial condition over the observable state, the inferred latent state dynamics, though different from the true ones, depicts a similar spiral pattern.
This result is in agreement with the geometrical reconstruction techniques (Takens, 1981) of the latent dynamics up to a diffeomorphic mapping.
Overall, our model learns a dynamical behavior similar to the true model represented by an elliptic transient and an equilibrium point limit-set.
Furthermore, the projection of the augmented latent space and the true solution of Eq.
(8) in the real axis illustrate the relevance of the proposed framework in forecasting the observations dynamics (mean square error <
1 E − 6 ).
Lorenz-63 dynamics : Lorenz-63 dynamical system is a 3-dimensional model that involves, under some speciﬁc parametrizations (Lorenz, 1963), chaotic dynamics with a strange attractor.
We simulate chaotic Lorenz-63 state sequences with the same model parameters as proposed in (Lorenz, 1963) using the LOSDA ODE solver (Hindmarsh, 1983) with an integration step of 0.01.
We assume that only the ﬁrst Lorenz-63 variable is observed x t = s t, 1 .
We apply the proposed framework to this experimental setting using a training sequence of 4000 time-steps.
For benchmarking purposes, we perform a quantitative comparison with state-of-the-art approaches using delay embedding representations (Takens, 1981).
The parameters of the delay embedding
Time 0 2575 100150
G ( Φ θ , t ( u t − 1 )) , R e a l ( s t ) − 0 .
4 − 0 .
I m a g ( s t ) , z t − 0 .
6 − 0 .
4 − 0 .0 .
Prediction
Extrapolation
True solution
Time 0 2575 100150
G ( Φ θ , t ( u t − 1 )) , R e a l ( s t ) − 1 .− 0 .0 .0 .1 .
I m a g ( s t ) , z t − 1 . 0 − 0 .0 .
Prediction
Extrapolation
True solution representation, namely the lag τ and the dimension d E of the augmented space were computed using state-of-the-art techniques.
Speciﬁcally, the lag parameter was computed using both the mutual information and correlation techniques (Abarbanel, 1996b), respectively denoted as τ MI and τ Corr .
Regarding the dimension of the embedding representation, we used the Takens embedding condition d E = 2 d + 1 with d the dimension of the hidden limit-cycle.
The delay embedding dimension was also computed using the False Nearest Neighbors (FNN) method (Abarbanel, 1996c).
We also tested arbitrary parameters for the delay embedding dimension.
Given the delay embedding representation, we tested two state-of-the-art data-driven representations of the dynamics.
The Analog Forecasting technique (AF) which is based on the nearest neighbours algorithm (Lguensat et al., 2017) and the
Sparse Regression (SR) method on a second order polynomial representation of the delay embedding states.
Regarding deep learning models, we compare our method to a stacked Bidirectional LSTM (RNN) and to the Latent-ODE model as proposed in (Chen et al., 2018).
Finally, the proposed framework, referred to as Neural embedding for Dynamical Systems (NbedDyn) was tested for different dimensions of the augmented state space, namely from 3 to 6 (please refer to the apendix for details on the neural networks based architectures) 3 .
Fig.
2 illustrates the learning process for the latent space from the initialization to the last training epoch.
We also report the analysis of short-term forecasting performance as well as the long-term asymptotic behavior characterized by the largest Lyapunov exponent of the benchmarked models in Tab 1.
The proposed model leads to signiﬁcant improvements in terms of short term forecasting performance with respect to the other approaches.
Surprisingly, The Latent-ODE and RNN models lead to the poorest performance both in terms of forecasting error and asymptotic behavior.
This is mainly due, in the Latent-ODE case, to the fact that the latent space is seen as a non linear projection of the observed variables through the optimization of the ELBO loss (Krishnan et al., 2016).
By contrast, our latent embedding formulation optimizes the latent states to forecast the observed variables which explicitly constrain the latent space to be an embedding of the true underlying dynamics.
The RNN model in the other hand converges to a periodic solution (please refer to the appendix for forecasting ﬁgures) with still a poor short term forcasting performances.
Overall, this results on deep learning models suggest that one should use such tools with care to guarantee satisfying the speciﬁcations of the underlying system.
The SR model seems to lead to better short term forecast (using ad hoc parameters ( τ = 6 , d E = 3 ), however, it does not capture well the chaotic neural networks based models were averaged over 5 runs except for NbedDyn d E =and d E = 5 where we only launched 3 runs.
We will include other runs for the ﬁnal version of the manuscript
[Table]
[Table] [Table]
[Table] [Table]
[Table]
.x 1 0 . 0 0 .
z 1 0 . 0 0 .
6 0 .z 2 0 .
Epoch =
Last d E = 4 0 . 0 0 .
6 0 .x 1 0 . 0 0 .
z 1 0 . 0 0 .
6 0 .z 2 0 .
6 0 .x 1 0 . 0 0 .
z 1 0 . 0 0 .
6 0 .z 2 0 .
80 .x 1 0 . 0 0 .
z 1 0 .
6 0 .z 2 0 .
80 .x 1 0 . 0 0 .
z 1 0 .
6 0 .z 2 0 .
80 .x 1 0 . 0 0 .
z 1 0 . 0 0 .
6 0 .z 2 0 .
6 0 .d E = 5 0 . 0 0 .
8 1 .x 1 0 . 0 0 .
z 1 0 . 0 0 .
8 1 .z 2 0 . 0 0 .
6 0 .1 .x 1 0 . 0 0 .
z 1 0 .
80 .z 2 0 .
80 .x 1 0 . 0 0 .
z 1 0 .
6 0 .z 2 0 .
6 0 .1 .x 1 0 . 0 0 .
z 1 0 .
8 1 .z 2 0 .
6 0 .x 1 0 . 0 0 .
z 1 0 . 0 0 .
8 1 .z 2 0 .
6 0 .d E = 6 0 . 0 0 .
6 0 .x 1 0 . 0 0 .
z 1 0 . 0 0 .
8 1 .z 2 0 . 0 0 .
6 0 .x 1 0 . 0 0 .
z 1 0 .
70 .z 2 0 .
80 .x 1 0 . 0 0 .
z 1 0 . 0 0 .
6 0 .z 2 0 .
80 .x 1 0 . 0 0 .
z 1 0 . 0 0 .
6 0 .z 2 0 .
8 1 .x 1 0 . 0 0 .
z 1 0 .
8 1 .z 2 0 .
6 0 .x 1 0 . 0 0 .
4 0 .0 .1 .z 1 0 . 0 0 .
8 1 .z 1 0 . 0 0 .
[Figure] [Figure]
patterns, which are associated to a positive largest Lyapunov exponent.
This may suggest the combination of the SR model and a delay embedding may require additional investigation as a good geometrical reconstruction of the phase space as stated in Takens’ theorem does not guarantee the existence of a parametric ODE model based on the corresponding delay embedding variables.
Better performance is reported using an analog forecasting approach.
The performance however greatly varies depending on the considered deﬁnition of the delay embedding.
Using ad hoc parameters ( τ = 6 , d E = 3 ), one may retrieve the expected long-term chaotic behavior ( λ 1 = 0 . 87 ) with a relatively low short-term forecasting error ( 8 .
e − 4 for a one-step-ahead forecast).
When considering the proposed model, we report for all parametrizations, augmented space dimensions fromto 6, performance at least in the same range as the best analog forecasting setting.
Besides, when increasing the dimension of the augmented space, we signiﬁcantly decrease short-term forecasting errors ( < 1 .e − 4 for a one-step-ahead forecast when considering the best ﬁt for d E = 6 , i.e. one order of magnitude compared to the best benchmark model) while keeping an appropriate chaotic long-term pattern ( λ 1 = 0 . 92) .
Modeling Sea Level Anomaly (SLA) :
The data driven identiﬁcation of dynamical representations of real data is an extremely difﬁcult task especially when the underlying processes involve non stable behaviors such as chaotic attractors.
This is mainly due to the fact that we do not have any exact knowledge of the closed form of the equations governing the temporal evolution of our variables.
Furthermore, the measured quantity may depend on other unobserved variables which makes the exploitation of data-driven techniques highly challenging.
In this context, we report an application to SLA (Sea Level Anomaly) dynamics, which relate to upper ocean dynamics and are monitored by satellite altimeters (Calmant et al., 2008).
Sea surface dynamics are chaotic and clearly involve latent processes, typically subsurface and atmospheric processes.
The dataset used in our experiments is a SLA time series obtained using the WMOP product (Juza et al., 2016).
The spatial resolution of our data is a 0 .
05 ◦ and the temporal resolution h
= 1 day.
We use the data from January 2009 to December 2014 as training data and we tested our approach on the last month of the year 2014.
The considered region is located on south Mallorca ( 2 . 5 ◦ E to 4 .
25 ◦ E , 37 .
25 ◦ N to 39 .
5 ◦ N ).
Finally, and in order to identify a ROM, our observations are mapped to a low dimensional space using a projection matrix deﬁned ofﬂine using a PCA as follow : r t =
M ( x t ) ∈
R k with k = 15 which amounts to capture 92% of the total variance.
The proposed framework is applied simply by replacing x t by the PCA components r t .
Please refere to the appendix for a generalization of the proposed approache in the case of ROM.
[Figure]
+ h t 0 +
h t
+
[Table]
We report forecasting performance for our model and include a comparison with analog methods (AF),
Sparse regression (SR),
LSTM (RNN) and a neural ODE setting (Latent-ODE) in Tab.
2 (The results of the neural networks based models were averaged over 5 runs).
Regarding the proposed NbedDyn model we consider an augmented latent space with d E = 60 .
Our model clearly outperforms the three benchmarked schemes with a very signiﬁcant gain for the forecasting performance at one day (relative gain greater than 90 %) and two days (relative gain greater than 90 %).
For a 4-day-ahead forecasting, our model still outperforms the other ones though the gain is lower (relative gain of 40%).
Finally, and in order to illustrate the inﬂuence of adding extra dimensions to deﬁne an augmented latent space on real data, we also tested the proposed NbedDyn model directly on the PCA space ( d E = k
= 15 ) this model is referred to as NbedDynZERO and the inﬂuence of the latent components is clear from the results in Tab. 2 witch allows a relative gain up to 90 % with respect to the same model directly on the PCA space.
We let the reader refer to the Supplementary
Material for a more detailed analysis of these experiments, including visual comparisons of the forecasts.
D ISCUSSION
In this work, we address the data-driven identiﬁcation of latent dynamics for systems which are only partially observed, i.e. when some components of the system of interest are never observed.
The reported forecasting performance for Lorenz-63 dynamics is in line with the forecasting performance of state-of-the-art learning-based approaches for a noise-free and fully-observed setting.
This is of key interest for real-world applications, where observing systems most often monitor only some components of the underlying dynamics.
As a typical example, the SLA forecasting experiment clearly motivates the proposed framework in the context of ocean dynamics for which neither in situ nor satellite observing systems can provide direct observations for all state variables (e.g., subsurface velocities, ﬁne-scale sea surface currents).
We may also further discuss how the proposed framework relates to state-of-the-art dynamical system theory approaches.
Most of these approaches rely on delay embedding, as Takens’s theorem states the existence of a delay embedding in which the topological properties of the hidden dynamical system are equivalent to those of the true systems up to a diffeomorphic mapping.
Hence, stateof-the-art approaches typically combine the selection of a delay embedding representation within classic regression models to represent the one-step-ahead mapping in the considered embedding.
Here, we consider latent dynamics governed by an unknown ODE (4) but we do not explicitly state the latent space.
This is however implicit in our forecasting framework.
By construction, the considered forecasting model relies on the integration of the learnt ODE (4) from an initial condition given as the solution of minimization (7).
Let us consider the following embedding ψ such that:
[Equation] [Equation]
Given this embedding, the resulting one-step-ahead forecasting for the observed variable may written as:
[Equation]
Hence, ψ deﬁnes a delay embedding representation implicitly stated through minimization (7).
In this embedding, the dynamics of the observed system x is governed by the composition of observation operator G and forecasting operator Φ θ,t .
Regarding the literature on Koopman operator theory, most approaches rely on the explicit identiﬁcation of eigenfunctions and eigenvalues of the Koopman operator (Koopman, 1931; Brunton et al., 2016a;
Tu et al., 2014).
Our framework relates to the identiﬁcation of the inﬁnitesimal generator f θ of the one-parameter subgroup deﬁned by Koopman operator through the ODE representation (4).
By construction, the Koopman operator associated with the identiﬁed operator f ˆ θ is also diagonalizable, such that the identiﬁcation of inﬁnitesimal generator f ˆ θ provides an implicit decomposition of the Koopman operator of the underlying and unknown dynamical system onto the eigenbasis of the learnt latent dynamics governed by ODE (4).
Future work will further explore methodological aspects, especially the application to highdimensional and stochastic systems.
In the considered framework, when considering ROM of the
SLA experiment, the projection from the observations space to the low dimensional space was carried using simple PCA projection.
Although for the geociences community, using PCA to reduce the dimensionality is motivated by the Galerkin derivation of reduced order models from complex high dimensional governing partial differential equations (Holmes et al., 2012), using auto-encoders have shown promising results in discovering optimal coordinates when trained jointly with a dynamical system.
The combination of the proposed framework with the variational setting considered in the
Latent-ODE model (Chen et al., 2018) also appears as an interesting direction for future work.
The extension to stochastic systems through the identiﬁcation of a Stochastic ODE is also of key interest, for instance for future applications of the proposed framework to geophysical random ﬂows, especially to the simulation and forecasting of ocean-atmosphere dynamics in which stochastic components naturally arise (Chapron et al., 2018).
A PPENDIX A P ROOF OF PROPOSITION
This proposition can be easily extended to any observation function that doesn’t form an embedding of the initial unobserved ODE.
However, for the sake of simplicity, we will consider the example given in Eq.
Lets suppose a a smooth ODE in the observation space that governs the time evolution of x from
Eq.
[Equation]
This ODE generates a ﬂow x t =
Ψ t ( x 0 ) .
Since our observation operator is not one-to-one, we can assume the existence of some ˆ t, t 1 , twhere Real (Φ ˆ t ( s t 1 ))
=
Real (Φ ˆ t ( s t 2 )) with Real (s t 1 ) (cid:54) =
Real (s t 2 ) ( Φ is the ﬂow generated by the unobserved ODE illustrated in Eq.
Projecting this equality to the observation space leads to :
[Equation]
Since the above ODE is smooth (or continuously differentiable), we can show that f is locally
Lipschitz on any interval containing t 0 (Sohrab, 2003) which garentees by Picard’s Existence Theorem the existance of a unique solution (Coddington & Levinson, 1955).
Formally, for the times ˆ t, t 1 , t 2 , Ψ ˆ t ( x t 1 ) =
Ψ ˆ t ( x t 2 ) if and only if x t 1 = x t 2 .
This contradicts the assumption that x t 1 (cid:54) =
x t 2 and thus, there is no existence of a ˆ t such that Real (Φ ˆ t ( s t 1 )) =
Real (Φ ˆ t ( s t 2 )) with
Real (s t 1 ) (cid:54) =
Real (s t 2 ) .
B A UGMENTED R EDUCED O RDER M ODELING
With a view to providing a general formulation which also accounts for reduced-order modeling (ROM) (Champion et al., 2019), we assume that the observations x t are fully characterized by a lower-dimensional state M ( x t ) ∈
R k , with M a global coordinates chart, mapping each observation to its coordinates M ( x t ) ∈ R k .
Typical examples for operator M includes the identity as in the equations of section 3.
In ROM, M may include PCA projections (Benner et al., 2005) for
Galerkin derivation of ROM for complex high dimensional partial differential equations (Rowley et al., 2004).
Formally, the augmented state space is formulated as :
[Equation] [Equation]
with the corresponding SSM :
[Equation] [Equation]
Based on the above formulation, we can state the optimization problem stated in Eq.
(5) as:
[Equation]
Subject to
[Equation]
with the corresponding non constrained formulation :
[Equation] [Equation]
Finally, when considering forecasting applications of new observations, we can restate the equation 7 in the context of ROM as follow :
[Equation]
C L IST OF S YMBOLS
s
Unobserved state governing the underlying dynamics in R m x
Observed component of s in R n , usually related to some components of s
M
In the case of ROM ( x high dimensional), M is a projection operator from R n to R k with n >> k u
Augmented latent space of dimension R d E which consists on a concatenation of an observed components ( M ( x ) ) and a latent components z
z
Latent component concatenated to M ( x ) to form the augmented state u f ( . )
Unknown dynamical system governing the unobserved state s Φ t ( . )
Flow map of the Unknown dynamical system f ( . )
f θ ( . )
Dynamical system governing the augmented state u
Φ t,θ ( . )
Flow map of the dynamical system f θ ( . )
D D IMENSIONALITY ANALYSIS OF THE N BED D YN MODEL
One of the Key parameters of the proposed model is the dimension of the latent space.
Despite the fact that it is extremely challenging to get a prior idea of the dimension of the model in the case of real data experiments, one can analyze the spawned manifold of the learnt latent states to get an idea of the true dimension of the underlying model (true here stands for a sufﬁcient dimension of the latent space).
The idea here is to compute the modulus of the eigenvalues of the Jacobian matrix for each input of the training data.
An eigenvalue does not inﬂuence the temporal evolution of the latent state if it has a modulus that tend to zero.
The number of non-zero eigenvalues can then be seen as a sufﬁcient dimension of the latent space.
Regarding the identiﬁcation of an ODE model governing the ﬁrst state variable of the Lorenzmodel, Fig. 3 illustrates the eigenvalues of the Jacobian matrix and their modulus for a dimension of the latent space d E = 6 .
Interestingly, only 3 eigenvalues have non-zero modulus and are effectively inﬂuencing the underlying dynamics.
This result shows that one can use a 3 dimensional latent-space as a sufﬁcient dimension to identify an ODE model governing the ﬁrst state of the Lorenz 63 system which is the same dimension as the true Lorenz 63 model. [Figure] [Figure] d E =
The analysis of the eigenvalues of the Sea Level Anomaly model in the other hand are not as straightforward as in the case of the Lorenz model since we do not have any idea on the analytical form of the underlying dynamical model.
Fig.
4 illustrates that using a 60 dimensional latent space for the
NbedDyn model, only 50 eigenvalues have non-zero modulus and thus, are effectively inﬂuencing the underlying dynamics.
The conclusion in this case is that the observed SLA data evolve in adimensional latent space parametrised by the dynamical model f θ .
E A DDITIONAL FIGURES OF THE L ORENZ 63 EXPERIMENT
We illustrate the forecasting performance of the tested models for the Lorenz-63 experiment through an example of forecasted trajectories in Fig.
Our model with d E = 6 leads to a trajectory similar to the true one up to 7 Lyapunov times [Figure] when the best alternative approach diverge from the true trajectory beyond 4 Lyapunov times [Figure]
An other interesting experiment is to ﬁnd the initial condition for new observation data.
This issue is addressed as presented in section 3 as follow.
Given a new noisy and partial observation sequence (Fig. 6), we ﬁrst look for a potential initial condition in the inferred training latent state sequence.
[-10 Lorenztime − 20 − 15 − 10 −
-]
This initial condition is then optimized using the cost function described by equation (7) to minimize the forecasting error of the new observation sequence.
[Figure]
F A DDITIONAL FIGURES OF THE S EA LEVEL A NOMALY EXPERIMENT
Forecasted states of the Sea Level Anomaly are illustrated in Fig.
8 and 9.
The visual analysis of the forecasted SLA states emphasize the relevance of the proposed NbedDyn model.
While state of the art approaches generally overestimate the time evolution of some structures such as eddies, our model is the only one to give near perfect forecasting up to 4 days [Figure] i r s t [Figure]
C A c
[Figure]
m p o n e n t [Figure] F i r s t P C A c o m p o n e n t
[Figure]
p
[Figure]
n
[Figure]
n
[Figure]
[Figure]
[Table]
−0.
V a l u
[Table]
(c
[Table]
[Table]
−
[Table]
[Table]
[Table]
V a l u e
[Table]
) RNN
[Table]
[Table]
V a l u e
[Table]
(e) NbedDynZERO
[Table]
[Table]
V a l
[Table]
[Table]
−
[Table]
[Figure] [Figure] G N EURAL NETWORKS HYPERPARAMETERS The neural-network parametrization for the dynamical model operator f θ is a bilinear architecture as presented in (Fablet et al., 2018).
The idea of this model is to embed second order polynomial repre- sentations of the input states by combining fully-connected layers and an element-wise product.
This bilinear layer are then concatenated to classical fully connected layers to perform regression.
This architecture incorporates physical knowledge on the dynamics since physical dynamical systems often involve bilinear non-linearities.
Among classic physical models, one may cite for instance advection-diffusion dynamics or shallow water equation.
This model is than integrated numerically to compute the ﬂow-map Φ θ,t using a fourth order Runge
Kutta scheme.
In practice, the integration scheme is implemented as a ResNet with four layers sharing the dynamical model.
G. L ORENZ 63 EXPERIMENTS HYPERPARAMETERS
[Table]
[Table]
[Table]
G. SLA EXPERIMENTS HYPERPARAMETERS
[Table] [Table]
(Chen et al., 2018) for more details.
[Table]
H S COOP AND LIMITATIONS
Constraining limit cycles The proposed augmented ODE formulation does not suppose any prior knowledge on the underlying dynamics responsible for the temporal evolution of the observations.
This can lead in some cases (especially when working on chaotic dynamics) to output a dynamical representation that has several attracting regions in addition to the one leading to the observations limit cycle.
This can lead to inappropriate results when trying to ﬁnd an initial condition that forecasts a given observation sequence.
The Idea of using the manifold spanned by the augmented training data allows to bypass this issue but we believe that adding additional constraints (energy preserving constraints, known symmetries in the models ...etc.) can signiﬁcantly improve the quality of the learnt dynamical models.
I C ODE SAMPLE
# import l i b s from g e n e r a t
e
d a t a import g e n e r a t e d a t a import numpy as np import t
o r c h from t o r c h .
autograd import V a ri a b le seed =np .
random .
seed ( seed ) t
o r c h .
manual seed ( seed ) # Generate data c
l a s s GD: model = ’ Lorenz 63 ’ c
l a s s parameters : sigma = 10.
rho = 28.
beta = 8 .
/d t
i n t e g r a t i o n = 0.01 # i n t e g r a t i o n time n b l o o p d a t a = 50.01 # s i z e of the catalog t
e s t s a m p l e s =
# run the data g e n e r a t i o n catalog , xt , yo =
g e n e r a t e d a t a (GD)
X t e s t =
c a t a l o g .
analogs [ c a t a l o g .
analogs .
shape [0] − GD.
t e s t s a m p l e s : , : 1 ]
X t r a i n = c a t a l o g .
analogs [ : c a t a l o g .
analogs .
shape [0] − GD.
t e s t s a m p l e s , : 1 ]
Grad t
= np .
g r a d i e n t ( X t r a i n [ : , 0 ] ) .
reshape ( X t r a i n .
shape [ 0 ] , 1 ) \ /GD.
d t i n t e g r a t i o n x = V ar i a b le ( t o r c h .
from numpy ( X t r a i n ) .
f l o a t ( ) )
z = V ar i a b le ( t o r c h .
from numpy ( Grad t ) .
f l o a t ( ) )
d a t a
s i z e =
X t r a i n .
shape [ 0 ] # neural net params params = {} params [ ’ t r a n s i
t
i o n l a y e r s ’ ]= params [ ’ b i l i n e a r l a y e r s ’ ]= params [ ’ d i m h
i d d e n l i n e a r ’ ]
=params [ ’ dim input ’ ]= params [ ’ d i m l a t e n t ’ ]= params [ ’ dim output ’ ]= params [ ’ dim input ’ ]
+ params [ ’ d i m l a t e n t ’ ] params [ ’ d t i n t e g r a t
i o n ’ ]
= GD.
d t i n t e g r a t i o n # Dynamical model c l a s s FC net ( t o r c h .
nn . Module ) :
def i n i t ( s e l f , params ) : super ( FC net , s e l f ) .
i n i t ( ) y aug =
) ) s l f .
l i n e a r C e l l = t
o r c h .
nn .
Linear ( output ’ ] \ , params [ ’ d
i m h
i d d e n l i n e a r ’ ] ) s e l f .
B l i n e a r C e l l 1 = \ t
o r c h .
nn .
ModuleList ( \ [ t o r c h .
nn .
Linear ( params [ ’ dim output ’ ] , 1) \ for i in range ( params [ ’ b i l i n e a r l a y e r s ’ ] ) ] ) s e l f .
B l i n e a r C e l l 2 = \ t
o r c h .
nn .
ModuleList ( \ [ t o r c h .
nn .
Linear ( params [ ’ dim output ’ ] , 1) \ for i in range ( params [ ’ b i l i n e a r l a y e r s ’ ] ) ] ) augmented size = params [ ’ b i l i n e a r l a y e r s ’ ]
\ + params [ ’ d i m h i
d
d e n l i n e a r ’ ]
s e l f .
t r a n s L a y e r s = \ t
o r c h .
nn .
ModuleList ( \ [ t o r c h .
nn .
Linear ( augmented size , params [ ’ dim output ’ ] ) ] ) s e l f .
t r a n s L a y e r s .
extend ( \ [ t o r c h .
nn .
Linear ( params [ ’ dim output ’ ] , params [ ’ dim output ’ ] ) \ for i in range (1 , params [ ’ t r a n s i
t
i o n l a y e r s ’ ] ) ] ) s e l f .
outputLayer =
t o r c h .
nn .
Linear ( params [ ’ dim output ’ ] , \ params [ ’ dim output ’ ] ) def forward ( s e l f , inp , dt ) :
i f inp .
shape [ − 1] < params [ ’ d i m l a t e n t ’ ]
+ params [ ’ dim input ’ ] : aug inp =
t o r c h .
c a t ( ( inp , s e l f .
y aug ) , dim =1) e l s e : aug inp = inp
BP outp =
V a r ia b l e ( t o r c h .
zeros ( ( aug inp .
s i
z e ( ) [ 0 ] , \ params [ ’ b i l i n e a r l a y e r s ’ ] ) ) )
L outp =
s e l f .
l i n e a r C e l l ( aug inp ) for i in range ( ( params [ ’ b i l i n e a r l a y e r s ’ ] ) ) :
BP outp [ : , i ]= s e l f .
B l i n e a r C e l l 1 [ i ] ( aug inp ) [ : , 0 ] ∗ \ s e l f .
B l i n e a r C e l l 2 [ i ] ( aug inp ) [ : , 0 ] aug vect =
t o r c h .
c a t ( ( L outp , BP outp ) , dim =1) for i in range ( ( params [ ’ t r a n s i t i o n l a y e r s ’ ] ) ) : aug vect =
( s e l f .
t r a n s L a y e r s [ i ] ( aug vect ) )
grad =
s e l f .
outputLayer ( aug vect ) return grad , aug inp model =
FC net ( params ) # compute flow : RK c l a s s INT net ( t o r c h .
nn . Module ) :
def i n i t ( s e l f , params ) : super ( INT net , s e l f ) .
i n i t ( ) # s e l f . add module ( ’ Dyn net ’ , FC net ( params ) )
s e l f .
Dyn net = model def forward ( s e l f , inp , dt ) : k1 , aug inp =
s e l f .
Dyn net ( inp , dt ) inp k2 = aug inp + 0.5 ∗ params [ ’ d t i n t e g r a t i o n ’ ] ∗ k k2 , tmp = s
e l f .
Dyn net ( inp k2 , dt ) inp k3 = aug inp + 0.5 ∗ params [ ’ d t i n t e g r a t
i o n ’ ] ∗ k k3 , tmp = s e l f .
Dyn net ( inp k3 , dt ) inp k4 =
aug inp + params [ ’ d t i n t e g r a t i o n ’ ] ∗ k k4 , tmp = s e l f .
Dyn net ( inp k4 , dt ) pred = aug inp + dt ∗ ( k1+2 ∗ k2+2 ∗ k3+k4 ) / 6 return pred , k1 , inp , aug inp # I n s t a n c i a t e the model modelRINN =
INT net ( params ) c
r i t e r i o n = t
o r c h .
nn .
MSELoss ( r e d u c t
i
o n = ’ elementwise mean ’ )
o p t i m i
z
e r = t
o r c h .
optim .
Adam( model .
parameters ( ) )
# P r e t r a i n i n g : f i t the g r a d
i e n t params [ ’ n t r a i n ’ ]=
[300000 ,10000] for t in range ( params [ ’ n t r a i n ’ ] [ 0 ] ) :
pred , grad , inp , aug inp =
modelRINN ( x , params [ ’ d t i n t e g r a t i o n ’ ] )
l o s s
= c
r
i t e r i o n ( grad [ : , : 1 ] , z ) l o s s
=
c r i t e r i o n ( pred [: − 1 ,:] , aug inp [ 1 : , : ] ) l o s s =
0.9 ∗ l o s s 1 +0.1 ∗ l o s sprint ( t , l o s s )
o p t i m i
z
e r .
z e r o g r a d ( ) l
o s s .
backward ( r e t a i n g r a p
h
=
True )
o p t i m i
z
e r .
s t e p ( ) # t r a i n i n g for t in range ( params [ ’ n t r a i n ’ ] [ 1 ] ) : pred , grad , inp , aug inp =
modelRINN ( x , params [ ’ d t i n t e g r a t i o n ’ ] )
l o s s
= c
r
i t e r i o n ( pred [: − 1 ,:] , aug inp [ 1 : , : ] ) l o s s
= c r i t e r i o n ( pred [: − 1 ,1:] , aug inp [ 1 : , 1 : ] ) l o s s =10.0 ∗ l o s s 1 + 1.0 ∗ l o s sprint ( t , l o s s )
o p t i m i
z
e r .
z e r o g r a d ( ) l
o s s .
backward ( r e t a i n g r a p
h
=
True )
o p t i m i
z
e r .
s t e p ( )