Original: L EARNING SPACE TIME DYNAMICS WITHPDE GUIDED NEURAL NETWORKS
Revised: L EARNING PARTIALLY OBSERVABLE PDE DYNAMICS WITH NEURAL NETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Spatio-Temporal processes bear a central importance in many applied scientiﬁc ﬁelds.
Revised: Spatio-Temporal processes bear a central importance in many applied scientiﬁc ﬁelds.

Original: Generally, differential equations are used to describe these processes.
Revised: Generally, differential equations are used to describe these processes.

Original: In this work, we address the problem of learning spatio-temporal dynamics with neural networks when only partial information on the system’s state is available.
Revised: In this work, we address the problem of learning spatio-temporal dynamics with neural networks when only partial information on the system’s state is available.

Original: Taking inspiration from the dynamical system approach, we outline a general framework in which complex dynamics generated by families of differential equations can be learned in a principled way.
Revised: Taking inspiration from the dynamical system approach, we outline a general framework in which complex dynamics generated by families of differential equations can be learned in a principled way.

Original: Two models are derived from this framework.
Revised: Two models are derived from this framework.

Original: We demonstrate how they can be applied in practice by considering the problem of forecasting ﬂuid ﬂows.
Revised: We demonstrate how they can be applied in practice by considering the problem of forecasting ﬂuid ﬂows.

Original: We show how the underlying equations ﬁt into our formalism and evaluate our method by comparing with standard baselines.
Revised: We show how the underlying equations ﬁt into our formalism and evaluate our method by comparing with standard baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We consider the problem of learning complex dynamics typical of physical systems using deep learning models.
Revised: We consider the problem of learning complex dynamics typical of physical systems using deep learning models.

Original: Modeling these dynamics is central to almost any problem in applied physics.
Revised: Modeling these dynamics is central to almost any problem in applied physics.

Original: The classical approach to this problem is model based : the dynamics of the physical system is usually formulated using Partial Differential Equations (PDEs) describing the evolution of state variables.
Revised: The classical approach to this problem is model based : the dynamics of the physical system is usually formulated using Partial Differential Equations (PDEs) describing the evolution of state variables.

Original: PDEs are then solved through some clever numerical scheme (as in Meerschaert & Tadjeran (2006) for example).
Revised: PDEs are then solved through some clever numerical scheme (as in Meerschaert & Tadjeran (2006) for example).

Original: This methodology is used in many application domains like climate science, ﬁnancial engineering, etc.. Although it has been extensively developed, several challenges remain open.
Revised: This methodology is used in many application domains like climate science, ﬁnancial engineering, etc.. Although it has been extensively developed, several challenges remain open.

Original: Modeling physical phenomena accurately can be tedious, if not unfeasible : the relations between state variables are often complex and, even when those are known, specifying a functional form for the dynamics can be difﬁcult (Lions et al.
Revised: Modeling physical phenomena accurately can be tedious, if not unfeasible : the relations between state variables are often complex and, even when those are known, specifying a functional form for the dynamics can be difﬁcult (Lions et al.

Original: (1992), Peebles (1980)).
Revised: (1992), Peebles (1980)).

Original: Moreover, designing accurate, robust and computationally feasible numerical schemes for PDEs is usually a difﬁcult problem for which current solutions rely on extensive domain speciﬁc background.
Revised: Moreover, designing accurate, robust and computationally feasible numerical schemes for PDEs is usually a difﬁcult problem for which current solutions rely on extensive domain speciﬁc background.

Original: The availability of huge data quantities gathered from different types of sensors or simulations is a strong incentive to develop data driven approaches, as an alternative or complement to standard model based methods.
Revised: The availability of huge data quantities gathered from different types of sensors or simulations is a strong incentive to develop data driven approaches, as an alternative or complement to standard model based methods.

Original: For many real world problems, the underlying physics of the phenomenon is not always known in its entirety and data-driven methods, inspired by our current physical knowledge, appear as a particularly promising alternative to model based approaches, see Bezenac et al.
Revised: For many real world problems, the underlying physics of the phenomenon is not always known in its entirety and data-driven methods, inspired by our current physical knowledge, appear as a particularly promising alternative to model based approaches, see Bezenac et al.

Original: (2018), Ling et al.
Revised: (2018), Ling et al.

Original: (2016), Stewart & Ermon (2016) or Rudy et al.
Revised: (2016), Stewart & Ermon (2016) or Rudy et al.

Original: (2017) for some examples.
Revised: (2017b) for some examples.

Original: On the other hand, mature ﬁelds such as differential equations could provide machine learning with useful insights and intuitions.
Revised: On the other hand, mature ﬁelds such as differential equations could provide machine learning with useful insights and intuitions.

Original: We advocate here a ML approach incorporating structural priors inspired from such a well developed domain.
Revised: We advocate here a ML approach incorporating structural priors inspired from such a well developed domain.

Original: We consider the problem of learning dynamical systems where available observations only provide partial information on the system state and dynamics which is a typical scenario in real world applications, be it in weather forecasting, ﬁnancial engineering, or even video prediction.
Revised: We consider the problem of learning dynamical systems where available observations only provide partial information on the system state and dynamics which is a typical scenario in real world applications, be it in weather forecasting, ﬁnancial engineering, or even video prediction.

Original: The objective is to learn from such observations the underlying system dynamics so as to1. forecast future observations, 2.
Revised: The objective is to learn from such observations the underlying system dynamics so as to forecast future observations.

Original: learn states and dynamics that follow as much as possible the laws of physics.

Original: We focus on data generated by complex highly non-linear differential equations and analyze whether these dynamics can be learned by Deep Learning models.
Revised: We focus on data generated by complex highly non-linear differential equations and analyze whether these dynamics can be learned by Deep Learning models.

Original: This context is particularly interesting since simulations relying on PDEs are essential in many applied disciplines and since this allows us to perform controlled experiments providing access to all the variables of the problem.
Revised: This context is particularly interesting since simulations relying on PDEs are essential in many applied disciplines and since this allows us to perform controlled experiments providing access to all the variables of the problem.

Original: Our contributions are the following :
Revised: Our contributions are the following :

Original: • We propose a general framework for modeling  dynamical systems governed by PDEs with neural networks and we derive two instances.
Revised: • We propose a general framework for modeling partially observed dynamical systems governed by PDEs with neural networks.

Original: This framework is inspired from the standard physical approach which has proven to be powerful to model complex real-world phenomena. Many forecasting problems can be cast into this framework, although this goes beyond the scope of this paper.
Revised: This framework is inspired by the way physicists incorporate observations into their forecasting pipeline. It should be ﬂexible enough to accommodate a large class of modeling problems other than forecasting observations.

Revised: • From this framework we derive two instances of training (PT and JT), with two different levels of prior injection, and two methods for multi-step forecasting (SSE and MSRE).

Original: • As a case study, we consider two PDEs describing the evolution of ﬂuid ﬂow, namely the Euler and Navier-Stokes equations , and show how they can be formulated within this formalism.
Revised: • As a case study, we consider two PDEs describing the evolution of ﬂuid ﬂow, namely the Euler and Navier-Stokes equations , and show how they can be formulated within this formalism.

Original: We study the performance and behaviour of our models on the problem of forecasting ﬂuid ﬂow.
Revised: We study the performance and behaviour of our models on the problem of forecasting ﬂuid ﬂow using two common baseline models and focusing on the interplay between injection of prior information and performance.

Original: We introduce our framework in section 2, detail the framework and its instances in section 3 and show how PDEs ﬁt within this framework in section 4.
Revised: We introduce our framework in section 2, detail the framework and its instances in section 3 and show how PDEs ﬁt within this framework in section 4.

Original: Experimental results and analysis are described in section 5 and recent related work is reviewed in section 6.
Revised: Experimental results and analysis are described in section 5 and recent related work is reviewed in section 6.

Original:  B ACKGROUND AND PROBLEM SETTING 2. L EARNING FROM PARTIALLY OBSERVED DATA
Revised:  B ACKGROUND AND PROBLEM SETTING 2. L EARNING FROM PARTIALLY OBSERVED DATA

Original: A dynamical system in our context can be broadly deﬁned as a function X with values in Ω which obeys a set of differential equations.
Revised: A dynamical system in our context can be broadly deﬁned as a function X with values in Ω which obeys a set of differential equations.

Original: We consider spatio-temporal dynamics for which X can be written as a function of ( t, x ) ∈ R × R d where t and x are respectively the time and space variables.
Revised: We consider spatio-temporal dynamics for which X can be written as a function of ( t, x ) ∈ R × R d where t and x are respectively the time and space variables.

Original: The spatial vector-valued function X t contains the quantities of interest describing a studied physical system at time t and gives the evolution in time and space of the system’s state.
Revised: The spatial vector-valued function X t contains the quantities of interest describing a studied physical system at time t and gives the evolution in time and space of the system’s state.

Original: More precisely, we will be interested in differential equations which can be written as :
Revised: More precisely, we will be interested in differential equations which can be written as :

Original: [Equation]
Revised: [Equation]

Original: ) where X t denotes the state vector at time t , D x X t the spatial derivatives of X t and F is a function with values in Ω (common instances for F involve the gradient operator ∇ X , the laplacian ∇ 2 X ,...).
Revised: where X t denotes the state vector at time t , D x X t the spatial derivatives of X t and F is a function with values in Ω (common instances for F involve the gradient operator ∇ X , the Laplacian ∇ 2 X ,...).

Original: By specifying a particular structure on F , one can describe the dynamics of many different classes of systems : an important example is that of the Navier-Stokes equations from ﬂuid dynamics which bear a central importance in many applied sciences and which will be our application example.
Revised: By specifying a particular structure on F , one can describe the dynamics of many different classes of systems : an important example is that of the Navier-Stokes equations from ﬂuid dynamics which bear a central importance in many applied sciences and which will be our application example.

Original: 2.1. N EURAL IMPLEMENTATION OF NUMERICAL SCHEMES
Revised: 2.1. N EURAL IMPLEMENTATION OF NUMERICAL SCHEMES

Original: Solving differential equations usually requires using a numerical scheme in order to approximate the solution.
Revised: Solving differential equations usually requires using a numerical scheme in order to approximate the solution.

Original: Finite difference methods for example make use of discrete approximations of the derivatives.
Revised: Finite difference methods for example make use of discrete approximations of the derivatives.

Original: Let us suppose available a sequence of states { ( X t ) t } , each X t being a function of the space variable x , X t and X t + δt being separated by a short time interval δt for all t .
Revised: Let us suppose available a sequence of states { ( X t ) t } , each X t being a function of the space variable x , X t and X t + δt being separated by a short time interval δt for all t .

Original: A straightforward discretization of equation 1 is the forward Euler scheme :
Revised: A straightforward discretization of equation 1 is the forward Euler scheme :

Original: [Equation]
Revised: [Equation]

Original: wher δt is the discretization timestep.
Revised: where δt is the discretization timestep.

Original: Using a parameterized neural network r θ , we can train it in order to ﬁt the relation above :
Revised: Using a parameterized neural network r θ , we can train it in order to ﬁt the relation above :

Original: [Equation]
Revised: [Equation]

Original: This is reminiscent of the skip connection structures used in the residual block of ResNet or in some
Revised: This is reminiscent of the skip connection structures used in the residual block of ResNet or in some

Original: RNN models.
Revised: RNN models.

Original: Linear multi-step methods extend this idea by performing several integration steps.
Revised: Linear multi-step methods extend this idea by performing several integration steps.

Original: A simple multi-step extension of the Euler scheme could then be :
Revised: A simple multi-step extension of the Euler scheme could then be :

Original: [Equation]
Revised: [Equation]

Original: This sequential process could be implemented by a ResNet architecture so that the latter could be used as a parametric model for approximating solutions of differential equations.
Revised: This sequential process could be implemented by a ResNet architecture so that the latter could be used as a parametric model for approximating solutions of differential equations.

Original: Moreover, spatial derivatives of X t at a given point x can classically be expressed with ﬁnite difference schemes over the values of the function at neighbouring points V x :
Revised: Moreover, spatial derivatives of X t at a given point x can classically be expressed with ﬁnite difference schemes over the values of the function at neighbouring points V x :

Original: [Equation]
Revised: [Equation]

Original: where K is a linear function.
Revised: where K is a linear function.

Original: Thus, we can express each component of D x X t as a convolution over
Revised: Thus, we can express each component of D x X t as a convolution over

Original: X t which motivates the use of Convolutional networks.
Revised: X t , motivating the use of convolutional networks.

Original: Thus, given a sufﬁciently large training set of state sequences and supposing that we are able to learn its dynamics, we could, just by knowing the initial state of our system, use the learned operator and forecast its behaviour for any time horizon.
Revised: Thus, given a sufﬁciently large training set of state sequences and supposing that we are able to learn their dynamics, we could, just by knowing the initial state of our system, use the learned operator and forecast its behaviour for any time horizon.

Revised: One could constrain the structure of K to express a particular set of derivatives.

Revised: This would be a way to add additional structure and knowledge into the forecasting operator.

Revised: One could also adopt a more hybrid approach such as in Long et al.

Revised: For the examples we studied in this paper, we use standard convolutional ResNets as we wanted to disentangle as much as possible the contributions of the elements we introduce, particularly when injecting priors into the estimator.

Revised: However, it would be interesting in future research to analyze the effects of more structural constraints added into the forecasting operator.

Original: 2.1. L INKING STATES TO PARTIAL OBSERVATIONS
Revised: 2.1. L INKING STATES TO PARTIAL OBSERVATIONS

Original: In realistic settings, the state is generally only partially observed e.g. , when studying the ocean’s circulation, variables contained in the system’s state such as temperature or salinity are observable while others such as velocity or pressure are not.
Revised: In realistic settings, the state is generally only partially observed e.g. , when studying the ocean’s circulation, variables contained in the system’s state such as temperature or salinity are observable while others such as velocity or pressure are not.

Original: In other words, the measured data is only a projection of themodel complete state X .
Revised: In other words, the measured data is only a projection of the complete state X .

Original: We model this measurement process with a ﬁxed operator H linking the system’s state X t to the corresponding observation Y t :
Revised: We model this measurement process with a ﬁxed operator H linking the system’s state X t to the corresponding observation Y t :

Original: [Equation]
Revised: [Equation]

Original: To use the former example, if only the surface temperatures are observed, the operator H will select among all the possible oceanic variables the ones related to sea surface temperature.

Original: In the following, H is supposed to be known and ﬁxed.
Revised: In the following, H is supposed to be known, ﬁxed and differentiable.

Revised: This setting is the one which we focus on in this paper.

Original:  F RAMEWORK
Revised:  F RAMEWORK

Original: To summarize, observations Y t are supposed to be generated by a state space model with variable
Revised: To summarize, observations Y t are supposed to be generated by a state space model with variable

Original: X t and the main objective is the prediction of future observations, e.g. Y t + k for  horizon k.
Revised: X t and the main objective is the prediction of future observations Y t + k for a certain horizon k.

Original: The general form of the model for a prediction at horizon 1 is then :
Revised: The general form of the model for a prediction at horizon 1 is then :

Original: [Equation]
Revised: [Equation]

Original: In this expression, prior information comes from the projection function H and from the state dynamics embedded in g .

Original: The classical approach in environmental science is data assimilation : starting from an initial estimate of the state at time t, a PDE numerical scheme is used to infer state at time t

Original: [-+ 1 , ˆ X t +
-]
Revised: [-+ 1 , ˆ X t +
-]

Original: and then ˆ Y t +1 .

Original: The initial state estimate is a function of past information.

Original: This approach relies on an explicit physical model expressed as a PDE for solving an inverse problem : infer X t from observations Y tt − k .

Original: The model in 2 could be alternatively implemented via a recurrent autoregressive NN architecture where past inputs are used together with the current state in order to predict the next state.

Original: We will use a variant of this formulation where the auto-regressive component and the dynamic model are separated, which is not the case with a generic RNN structure.

Original: We make the simplifying hypothesis that the information loss through the projection H is so that a state X t can be deterministically reconstructed using a long enough sequence.
Revised: We make the simplifying hypothesis that the information loss through the projection H is so that a state X t can be deterministically reconstructed using a long enough sequence.

Original: In other words, there exists an integer k and an operator E such that :
Revised: In other words, there exists an integer k and an operator E such that :

Original: [Equation]
Revised: [Equation]

Original: Let us consider, to simplify, one step predictions : given a sequence of initial observations ( Y 1 , ..., Y k ) , our main objective is to ﬁnd the best estimation (cid:98) Y k +1 for the observation
Revised: For more details on this assumption, refer to appendix C. For now, let us consider only one step predictions : given a sequence of initial observations ( Y 1 , ..., Y k ) , we want to ﬁnd the best estimation (cid:98) Y k +1 for the observation

Original: Y k +1  where observations Y are supposed to be generated an underlying process X .
Revised: Y k +1 , where observations Y are supposed to be generated an underlying process X .

Original: The proposed model proceeds in three steps :
Revised: We propose a framework where predicting (cid:98) Y k +1 is decomposed into three steps :

Original: • Prediction of the current state : Given past observations, we use a learnable operator e ω parameterized by ω to produce a (complete) state vector (cid:101) X t .
Revised: • Prediction of the current state : Given past observations, we use a learnable operator e ω parameterized by ω to produce a (complete) state vector (cid:101) X t .

Original: • Prediction of the future state : (cid:101)
Revised: • Prediction of the future state : (cid:101)

Original: X t is given as input to a second learnable operator f θ parameterized by θ which produces the next state (cid:98) X t
Revised: X t is given as input to a second learnable operator f θ parameterized by θ which produces the next state (cid:98) X t

Original: +1 .
Revised: +1 .

Original: • Projection of the future observation : (cid:98)
Revised: • Projection of the future observation : (cid:98)

Original: X t +1 is mapped onto the space of observations using operator H to predict future observation (cid:98) Y t +1 .
Revised: X t +1 is mapped onto the space of observations using operator H to predict future observation (cid:98) Y t +1 .

Original: This is summarized in the following system :
Revised: This is summarized in the following system :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The learning problems amounts to optimizing the following loss :
Revised: The learning problems amounts to optimizing the following loss :

Original: [Equation]
Revised: [Equation]

Original: where d is the loss metric and e and f are functions belonging to appropriate parametric families.
Revised: where d is the loss metric and e and f are functions belonging to appropriate parametric families.

Original: Time lag parameter k can be set up by cross validation.
Revised: Time lag parameter k can be set up by cross-validation.

Original: As discussed in section 2, a natural choice for f is a convolutional residual network modeling the system’s dynamics whereas for e  designing the right family of functions will depend on the structure of the studied data.
Revised: As discussed in section 2, a natural choice for f is a convolutional residual network modeling the system’s dynamics whereas, for e , designing the right family of functions will depend on the structure of the studied data.

Original: 3. F ORECASTING MODELS
Revised: 3. F ORECASTING MODELS

Original: Let us now introduce two instances of the above model.
Revised: Let us now introduce two instances of the framework described above.

Original: For simpliﬁcation weconsider ﬁrst  one step prediction and then introduce extensions to multistep predictions in section 3.2.
Revised: For clarity, we ﬁrst consider one step predictions and then introduce the multi-step extension in section 3.2.

Original: 3.1. J OINT TRAINING OF ESTIMATION AND FORECASTING
Revised: 3.1. J OINT TRAINING OF ESTIMATION AND FORECASTING

Original: When the information about the underlying process is limited to observations, the straightforward way to solve the minimization problem equation 4 is to train e and f jointly.
Revised: When the information about the underlying process is limited to observations, the straightforward way to solve the minimization problem equation 5 is to train e and f jointly.

Original: Prior information here is limited to the architectures of e and f as well as that of the pipeline equation 3 and the ﬁxed operator H .
Revised: Prior information here is limited to the architectures of e and f as well as that of the pipeline equation 4 and the ﬁxed operator H .

Original: Let (cid:8)(cid:0)
Revised: Let (cid:8)(cid:0)

Original: ( Y ( i ) t − k +1 , ..., Y ( i ) t ) , Y ( i ) t +1 (cid:1)(cid:9) i be a training set of observation sequences, each sequence being identiﬁed by index i .
Revised: ( Y ( i ) t − k +1 , ..., Y ( i ) t ) , Y ( i ) t +1 (cid:1)(cid:9) i be a training set of observation sequences, each sequence being identiﬁed by index i .

Original: The loss corresponding to sequence i is then :
Revised: The loss corresponding to sequence i is then :

Original: [Equation]
Revised: [Equation]

Original: With this approach only observations are needed for training without any additional supervision.
Revised: With this approach only observations are needed for training without any additional supervision.

Original: On the other hand, there is no reason for it to have any interpretable structure or correspond to the physical modelling of the studied system.
Revised: On the other hand, there is no reason for it to have any interpretable structure or correspond to the physical modelling of the studied system.

Original: 3.1. P RETRAINING THE STATE ESTIMATOR
Revised: 3.1. P RE - TRAINING THE STATE ESTIMATOR

Original: Often some form of additional knowledge on the system state is available and can be exploited.
Revised: It is not uncommon that some form of additional knowledge on the system state is available and can be exploited.

Original: • It is sometimes possible to access to a limited number of examples of state sequences through costly measurements.
Revised: • It is sometimes possible to access to a limited number of examples of state sequences through costly measurements.

Original: Then pretraining of e and f could be beneﬁcial, its success shall depend on the complexity of the studied dynamics and on theloss of information through H .
Revised: Then pre-training of e and f could be beneﬁcial, its success depending on the complexity of the studied dynamics and on the information loss associated with H .

Original: • More often it is possible to have access to a noisy or approximate version of the states X t through simulations.
Revised: • More often it is possible to have access to a noisy or approximate version of the states X t through simulations.

Original: Complex simulators and generated data are available in several domains : it is common for models to be developed using simulated data and then to be adapted to real observations.
Revised: Complex simulators and generated data are available in several domains : it is common for models to be developed using simulated data and then to be adapted to real observations.

Original: A lighter alternative is to use a simpliﬁed model which allows fast and cheap simulations.
Revised: A lighter alternative is to use a simpliﬁed model which allows fast and cheap simulations.

Original: Generated sequences can then be used e.g. for pretraining e and f before ﬁne-tuning with real observations or more complex simulations.
Revised: Generated sequences can then be used e.g. for pre-training e and f before ﬁne-tuning with real observations or more complex simulations.

Original: In both cases, the simpliﬁed dynamics plays the role of a prior over the underlying process. 
Revised: In both cases, the simpliﬁed dynamic plays the role of a prior over the underlying process. In order

Original: In the experiments in section 5.2, we will use a simpliﬁed dynamical model (Euler) as a prior for a more complex one (Navier Stokes) in order to illustrate and test this idea.
Revised: to illustrate and evaluate this idea, we will consider the Euler equations as a simpliﬁed version of the more complex Navier Stokes equations in the experiments (section

Original: 3. M ULTI - STEP FORECASTING
Revised: 3. M ULTI - STEP FORECASTING

Original: We present below two strategies that could be used for prediction at horizon l > 1 .
Revised: We present below two strategies that could be used for prediction at horizon l > 1 .

Original: In order to forecast l (cid:62) 2 steps ahead, oncewe have theestimated state at time t , (cid:101) X t , given by e ω , we may apply one of the following strategies : 
Revised: In order to forecast l (cid:62) 2 steps ahead, once the state at time t has been the estimated with (cid:101) X t using e ω , we may apply one of the following strategies : [Table]

Original:  • Single State Estimation ( SSE ) : the estimation operator e is used only once  to give the forecasting operator an initial state (cid:101)
Revised: [Figure] • Single State Estimation ( SSE ) : the estimation operator e is used only once in order to obtain an initial state (cid:101)

Original: X t : [Equation] [Table]
Revised: X t for the forecasting operator:

Original: [Figure]
Revised: [Equation]

Original: • Multiple State REstimation ( MSRE ) :
Revised: • Multiple State REstimation ( MSRE ) :

Original: We use the forecast operator to produce the next state
Revised: We use the forecast operator to produce the next state

Original: [Equation]
Revised: [Equation]

Original: Then, for the forecast at t + 2 , we estimate the state at time t + 1 by plugging the generated observation H ( (cid:98) X t +1 ) in the estimation operator e ω : 
Revised: Then, for the forecast at t + 2 , we estimate the state at time t + 1 by plugging the generated observation H ( (cid:98) X t +1 ) in the estimation operator e ω : (cid:101) X t +1 = e ω

Original: [Equation]
Revised: ( Y t − k +2 , ..., Y t , H ( (cid:98) X t +1 ))

Original: so that :
Revised: so that: (cid:98) X t +2 = f θ ( (cid:101)

Original: [Equation]
Revised: X t +1 )

Original: and then iterate for the following steps.
Revised: and then iterate for the following steps.

Original: In both cases,  observations up to Y t + l will be used as targets in the loss term when learning to predict at horizon l .
Revised: In both cases, ground truth observations up to Y t + l will be used as targets in the loss term when learning to predict at horizon l .

Revised: The SSE method is the classical one in dynamical systems.

Revised: However, especially when forecasting for long time horizons, it supposes that the estimator e is able to compress all relevant information about the long term evolution of the system into the initial state, which is a difﬁcult task.

Revised: On the other hand, the MSRE strategy, while less natural from the physical point of view, allows to gather information from the observations space at each step which should be an easier task.

Original: Note that using a multi-step forecasting objective during training could be useful since it imposes additional constraints on the learned functions.
Revised: Note that using a multi-step forecasting objective during training could be useful since it imposes additional constraints on the learned functions.

Original: This will be explored in the experimental section.
Revised: This will be explored in the experimental section.

Original: When using multi-step forecasting at training time, the error is back-propagated through the l prediction steps and one has to choose a principled way of supervising the training.
Revised: When using multi-step forecasting at training time, the error is back-propagated through the l prediction steps and one has to choose a principled way of supervising the training.

Original: Moreover, we have also used and compared different sampling strategies for long sequences training : systematic teacher forcing, no teacher forcing and scheduled sampling as presented in Bengio et al. (2015). Quantitative results are presented in the appendix. When training with long sequences, another problem arises : depending on the distribution of the eigenvalues of the matrices, we can observe weight values either diverging to inﬁnity or going to 0 .
Revised: Moreover, we compare in appendix E different sampling strategies for long sequences training.

Original: Following the analysis done in Saxe et al.
Revised: Finally, following the analysis done in Saxe et al.

Original: , we have observed that orthogonal initialization of the weights of f helps solving this issue.
Revised: , we have observed that orthogonal initialization of the weights of f is also important.

Original:  E XAMPLES FROM FLUID DYNAMICS : E ULER AND N AVIER -S TOKES
Revised:  E XAMPLES FROM FLUID DYNAMICS : E ULER AND N AVIER -S TOKES

Original: As an illustration, we now focus on the problem of  the evolution of a two-dimensional ﬂuid mechanics system.
Revised: As an illustration, we now focus on the problem of predicting the evolution of a two-dimensional ﬂuid mechanics system.

Original: The equations governing ﬂuids are among the most studied.
Revised: The equations governing ﬂuids are among the most studied.

Original: One of the challenging aspects of those highly non-linear equations is the chaotic behaviour of turbulent ﬂows.
Revised: One of the challenging aspects of those highly non-linear equations is the chaotic behaviour of turbulent ﬂows.

Original: We use, as examples, instances of Euler and Navier-Stokes equations which are brieﬂy introduced below.
Revised: We use, as examples, instances of Euler and Navier-Stokes equations which are brieﬂy introduced below.

Original: More information is available inthe appendix  and Acheson (1989) is a comprehensive reference textbook.
Revised: More information is available in appendix A, and Acheson (1989) is a comprehensive reference textbook.

Original: Euler being a simpler version of Navier Stokes, only the latter is presented below.
Revised: Euler being a simpliﬁed version of Navier Stokes, only the latter is presented below.

Original: Navier Stokes equations for incompressible ﬂuids are :
Revised: Navier Stokes equations for incompressible ﬂuids are :

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst equation results from Newton’s second law, the second comes from the conservation of density while the third is the incompressibility condition.
Revised: The ﬁrst equation results from Newton’s second law, the second comes from the conservation of density while the third is the incompressibility condition.

Original: In those equations, u denotes a twodimensional velocity ﬁeld, p is a pressure scalar ﬁeld, ρ is the ﬂuid density and g is the gravitational force.
Revised: In those equations, u denotes a twodimensional velocity ﬁeld, p is a pressure scalar ﬁeld, ρ is the ﬂuid density and g is the gravitational force.

Original: ∇ · u is the divergence of velocity u , u · ∇ is the advection operator u · ∇ = u x ∂. ∂x + u y ∂. ∂y
Revised: ∇ · u is the divergence of velocity u , u · ∇ is the advection operator u · ∇ = u x ∂. ∂x + u y ∂. ∂y

Original: Euler equations are similar to Navier Stokes but do not consider the particular nature of the ﬂuid and ignore the viscosity term ν ∇ 2 u in the ﬁrst equation of (5) by taking ν = 0 .
Revised: Euler equations are similar to Navier Stokes but do not consider the particular nature of the ﬂuid and ignore the viscosity term ν ∇ 2 u in the ﬁrst equation of (6) by taking ν = 0 .

Original: For Navier-Stokes, ν is the kinematic viscosity and ∇ 2 is the Lagrangian operator.
Revised: For Navier-Stokes, ν is the kinematic viscosity and ∇ 2 is the Lagrangian operator.

Original: The importance of viscosity in a ﬂuid, and thus a way to assess its qualitative behaviour, cannot be measured through ν alone.
Revised: The importance of viscosity in a ﬂuid, and thus a way to assess its qualitative behaviour, cannot be measured through ν alone.

Original: The most commonly used quantity to describe it is the Reynolds number R .
Revised: The most commonly used quantity to describe it is the Reynolds number R .

Original: This is the ratio of the inertia and the viscosity terms.
Revised: This is the ratio of the inertia and the viscosity terms.

Original: If R is small, it means that the ﬂuid is very viscous, behaving more like honey ;
Revised: If R is small, it means that the ﬂuid is very viscous, behaving more like honey ;

Original: If R is large, the viscosity term is small and the ﬂuid behaves more like water.
Revised: If R is large, the viscosity term is small and the ﬂuid behaves more like water.

Original: A deﬁnition of the Reynolds number is provided in the appendix.
Revised: A deﬁnition of the Reynolds number is provided in appendix A.

Original: Equation (5) is not of the same form as equation 1 as we still have the pressure variable p as well as the null divergence constraint.
Revised: Equation equation 6 is not of the same form as equation 1 as we still have the pressure variable p as well as the null divergence constraint.

Original: It is however possible to show that it follows the general form of equation 1 as there exists an operator P which outputs divergence-free vector ﬁelds so that, for Navier Stokes, the third equation is dropped and the ﬁrst equation takes the following form :
Revised: It is, however, possible to show that it follows the general form of equation 1 as there exists an operator P which outputs divergence-free vector ﬁelds so that, for Navier Stokes, the third equation is dropped and the ﬁrst equation takes the following form :

Original: [Equation]
Revised: [Equation]

Original: This is similar to equation 1 so that we can put the whole system in that form with X
Revised: This is similar to equation 1 so that we can put the whole system in that form with X

Original: = ( ρ, u ) .
Revised: = ( ρ, u ) .

Original: This means that all the results and developments in previous sections apply for this family of ﬂuid equations.
Revised: This means that all the results and developments in previous sections apply for this family of ﬂuid equations.

Original: We give more details about this in the appendix 
Revised: We give more details about this in the appendix B

Original: Note that in what follows, we have H ( X ) = ρ .
Revised: Note that in what follows, we have H ( X ) = ρ .

Original: In other words, we observe the scalar ﬁeld of densities and the complete state is obtained by concatenating the 2D velocity ﬂow u to ρ .
Revised: In other words, we observe the scalar ﬁeld of densities and the complete state is obtained by concatenating the 2D velocity ﬂow u to ρ .

Original:  E XPERIMENTS 
Revised:  E XPERIMENTS 5. D ETAILS

Original: The architecture of e ω is based on the Unet presented in Ronneberger et al.
Revised: The architecture of e ω is based on the UNet presented in Ronneberger et al.

Original: (2015) which is a convolutional-deconvolutional network with skip connections which takes as input k feature maps corresponding to the k different observations.
Revised: (2015) which is a convolutional-deconvolutional network with skip connections which takes as input k feature maps corresponding to the k different observations.

Original: In all the following experiments, we have chosen k = 3 .
Revised: In all the following experiments, we have chosen k = 3 .

Original: For the forecasting operator f θ , we use the ResNet architecture He et al.
Revised: For the forecasting operator f θ , we use the ResNet architecture He et al.

Original: (2016) with Leaky
Revised: (2016) with Leaky

Original: ReLU non-linearities of parameter 0 .
Revised: ReLU non-linearities of parameter 0 .

Original: 1 and 6 residual blocks.
Revised: 1 and 6 residual blocks.

Original: Our data is generated from a numerical simulation, using the Mantaﬂow ﬂuid simulation library Thuerey & Pfaff (2018).
Revised: Our data is generated from a numerical simulation, using the Mantaﬂow ﬂuid simulation library Thuerey & Pfaff (2018).

Original: We study three different dynamics : the ﬁrst one is governed by the Euler equations, the second and the third by the Navier-Stokes equations with R = 5000 and
Revised: We study three different dynamics : the ﬁrst one is governed by the Euler equations, the second and the third by the Navier-Stokes equations with R = 5000 and

Original: R = 100000 .
Revised: R = 100000 .

Original: The former corresponds to a viscous ﬂuid with complex dynamics while the latter corresponds to a low viscosity and is supposed to have a simpler dynamics.
Revised: The former corresponds to a viscous ﬂuid with complex dynamics while the latter corresponds to a low viscosity, closer to Euler.

Original: For each setting, we produce 300 training sequences of 500 timesteps each (one timestep being of duration 0 .
Revised: For each setting, we produce 300 training sequences of 500 timesteps each (one timestep being of duration 0 .

Original: 5 s ), randomly selecting initial conditions for each sequence.
Revised: 5 s ), randomly selecting initial conditions for each sequence.

Original: We then subsample 5 times in time (two successive states are then separated by 2 . 5 s ).
Revised: We then subsample 5 times in time (two successive states are then separated by 2 . 5 s ).

Original: In order to tune the hyperparameters of our experiments, wealso generate a validation set of 200samples as well as a test set of 100 samples.
Revised: In order to tune the hyperparameters of our experiments, we generate a validation set of 200 as well as a test set of 100 sequences generated randomly and independently.

Original: The choice of a normalization strategy for the data has not been immediate.

Original: We ﬁnally settled for the normalization by dividing each component of the state by its empirical standard deviation over the dataset.
Revised: As a normalization strategy, we chose dividing each component of the state by its empirical standard deviation over the dataset.

Original: We do not set the mean to zero as the positivity of the density is an important constraint of the problem.
Revised: We do not set the mean to zero as the positivity of the density is an important constraint of the problem.

Revised: All training horizons are set to l = 8 .

Revised: A more thorough exploration of this parameter is done in appendix D.

Original: As for the baselines, we have been using 1) asimple ResNet which takes  3 past observations as input to produce a future observation directly (for multi-step forecasting, we just give the produced observations back to the ResNet in an auto-regressive manner),  and 2) a Convolutional LSTM  with 2 layers (we have tried LSTMs with different numbers of layers and hidden dimensions and kept the one with the best validation score).
Revised: As for the baselines, we have used (1) a ResNet which takes k = 3 past observations as input to produce a future observation directly (for multi-step forecasting, we just give the produced obser- , it still provides us with an important information : forecasting the dynamics of those equations while following equation 1 automatically makes us ﬁnd a null-divergence velocity vector ﬁeld. vations back to the ResNet in an auto-regressive manner), which can be seen as an instance of our framework with e being the identity operator, and (2) a Convolutional LSTM (Shi et al. with 2 layers (we have tried LSTMs with different numbers of layers and hidden dimensions and selected the one with the lowest validation error).

Original: 5.  E ULER  EQUATIONSDYNAMICS
Revised: 5. F ORECASTING PERFORMANCE FOR E ULER AND N AVIER -S TOKES EQUATIONS

Revised: [Table] [Table] [Table]

Original: In this section, we study the  test results for the different algorithms for dynamics driven by the Euler equations.
Revised: In this section [Table] we study the performances on test samples for the different methods described with Euler and Navier-Stokes dynamics.

Original:  Table 1 compares the models to the two baselines ResNet and ConvLSTM. All make use of a training horizon l = 8 .
Revised: Pre-training (PT) is always done with Euler dynamics, even for the Navier-Stokes experiments. Table 1 shows results for Euler dynamics. We can observe that :

Original:  JT SSE clearly diverges.
Revised: • JT SSE clearly diverges.

Original: This is an overﬁtting problem that we have not been able to solve.
Revised: This is an overﬁtting problem that we have not been able to solve.

Original: SSE acts as an encoder-decoder architecture: all relevant information about t 0 +t 0 +t 0 +t 0 +t 0 +t 0 +t

Original: [Table]

Original: [Table]

Original: [Figure]

Original: JT

Original: [Figure] densities JT : state representation [Figure] the past observations are encoded into a single initial state,  the forecasting operator then generates a whole sequence of future state estimations.
Revised: SSE acts as an encoder-decoder architecture: all relevant information about the past observations is encoded into a single initial state, and the forecasting operator then generates a whole sequence of future state estimations.

Original: This seems to be a hard task without the additional prior given by the structure of the real physical state (which we have in the PT pretrained setting).
Revised: This seems to be a hard task without the additional prior given by the structure of the real physical state (which we have in the PT pre-trained setting).

Original:  All [Table]
Revised: • All other three introduced methods along with the ResNet perform relatively well, even for long training horizons.

Original: [Table] other three algorithms manage to produce good forecasts of the dynamics, even at long prediction horizons ( t 0 + 30 corresponds to a duration of 75 s ). For long horizons, it is  to be expected
Revised: Our algorithms beat the baselines, even if it is only by a small margin for the ResNet. • As expected, the PT SSE performs well for short to medium forecasting horizons but loses acccuracy in the long run.

Original: that the PT SSE algorithm should loose its efﬁciency as all information is supposed to be stored and rolled forward in a markovian fashion while the MSRE algorithms, as well as the ResNet and ConvLSTM baselines, work in an auto-regressive fashion over the observations which makes it easier to use information from the past.
Revised: This is due to the fact that all information is supposed to be stored and rolled forward in a Markovian fashion while the MSRE algorithms, as well as the ResNet and ConvLSTM baselines, work in an auto-regressive fashion over the observations which makes it easier to use information from the past.

Original: However, we can see that PT SSE is still on par with JT MSRE for all time horizons, which shows the efﬁciency of the true physical representation that the PT alorithm is conditioned on.
Revised: [Figure]

Original: Those results justify the fact that, from now on, we will only consider the PT SSE and the JT MSRE algorithms. Figure 2 shows prediction examples from the different models.
Revised: • We have justiﬁed in section 2 why the ResNet should perform well for this forecasting problem. We argue further in appendix C about the concatenation of observations used as input for the ResNet being a sufﬁcient state representation.

Original: First of all, let us notice how different the state representations are between the PT and JT instances : while the ﬁrst presents a structure which is very close to the true physical state, the second has a representation which cannot be interpreted as a velocity ﬁeld but is rather close to the predicted observation itself.
Revised: This explains the good results obtained by this very standard network. • Seen from this perspective, compared to the ResNet, JT MSRE only adds a learned transformation of the state optimized for forecasting. This can explain why this network performs so well.

Original: Despite this difference, both JT and ST manage to capture quite well the main features of the observed dynamics.
Revised: • Figure 5 shows that all three models manage to learn boundary conditions, which are kept constant throughout all datasets.

Original: ConvLSTM completely fails and ResNet does quite well even though it misses some subtle changes in density. This highlights the need for skip connections, which are present in the Unet as well as in the ResNet architectures, in order to capture high frequencies. However, let us stress the fact that the studied equations are chaotic in nature which means that even seemingly small errors may induce drastic changes in the evolution of density in the following time-steps and this is indeed what happens in the case of the process shown here 2 :we can actually see it already happening with the evolution of the small features of the density. Tables 2 and 3 show results for different training horizons l and forecasting horizons, for the two algorithms PT SSE and JT MSRE.
Revised: • Let us also note that the 30 steps we are considering here is somewhat unreasonable :

Original: At least up to a certain point, the longer the training horizon, the better the long term performance. This is especially true for the PT algorithm as it has to forecast the next states with only one state as input.
Revised: The studied equations are chaotic in nature and in the long run even the simulation which is taken as ground truth here diverges. Thus, the most meaningful results are those for earlier horizons.

Original: The improvements observed for the PT algorithm in table 3 are higher than the improvements for the JT algorithm in table 2  and  the PT algorithm supersedes
Revised: Figure 2 shows prediction examples from the different models and largely conﬁrms the observations above.

Original: JT at longer horizons. This seems to indicate that the  state representation with a physical meaning is more stable for long term forecasting. However, we still keep l = 8 for the other experiments for computational reasons. Concerning the sampling method, Teacher Forcing provides the lower performance : it is equivalent to a training horizon of 1 .
Revised: Let us also notice the difference between the SSE and the JT state representations : While the ﬁrst presents a structure which is very close to the true physical state, the second has a representation which cannot be interpreted as a velocity ﬁeld but is rather close to the predicted observation itself.

Original: What is more surprising is that No Teacher Forcing and Scheduled Sampling, with a tuned parameter α , have roughly the same performance. A quantitative comparison of the three methods is provided in tables 6 and7 of the appendix. [Table]
Revised: For the remaining part of the paper, we will focus on the PT SSE and the JT MSRE networks which seem to be the most interesting according to the observations above.

Original: [Table] 5. N AVIER -S TOKES EQUATIONS DYNAMICS Let us now consider themore complex dynamics of the Navier-Stokes equations.
Revised: Tables 2 and 3 consider the Navier-Stokes equations.

Original: As before, we use two algorithms: a jointly trained JT MSRE and a pretrained estimator PT SSE, pretrained using 
Revised: As before, we use two algorithms: a jointly trained JT MSRE and a pre-trained estimator PT SSE, in this case pre-trained using Euler equations :

Original: Eulerequations: Euler dynamics is thus used asa simple structural prior information for the more complex Navier Stokes dynamics.
Revised: Euler dynamics is thus used as structural prior information for the more complex Navier Stokes dynamics.

Original: Note that this is clearly an imperfect prior, especially when the viscosity is high ( i.e. when R is low).
Revised: Note that this is clearly an imperfect prior, especially when the viscosity is high ( i.e. when R is low).

Original: We then have a choice between keeping this imperfect estimator, hoping that the forecasting operator will make up for its shortcomings, and ﬁne tuning so that it adapts to the new dynamics.
Revised: We then have a choice between keeping this imperfect estimator, hoping that the forecasting operator will make up for its shortcomings, or ﬁne-tuning it on observations of Navier Stokes dynamics.

Original: We try the two options, the ﬁne tuning being done by jointly training the whole system, starting with a pretrained estimator, while dividing the learning rate of the estimator optimizer algorithm by a factor of 100 .
Revised: We try the two options, the ﬁne-tuning being done by jointly training the whole system, starting with a pre-trained estimator, while dividing the learning rate of the estimator’s optimizer algorithm by a factor of 100 .

Original: As table 4 shows, the second option yields clearly better results.
Revised: Results show that ﬁne-tuning yields clearly better results.

Original: We can also seefrom this table and from table 5 that the JT algorithm still gives  good results.
Revised: Moreover, we can also see that the JT algorithm still gives very good results and that the ResNet is consistently beaten by our methods.

Original: In order to further explore the difference in generalization power between the two algorithms, we trained both of them with only half of the training dataset. We can see from table 5 that the loss in performance affects the JT algorithm more than the PT one which becomes comparable to or better than JT.
Revised: [Table] [Table]

Original: This seems to show that the prior introduced by the pretraining of the estimator does play a role in facilitating the learning, as opposed to the JT where we try to ﬁnd a good representation from scratch :
Revised: [Table] pre-trained on Euler dynamics ﬁne-tuned on the Navier-Stokes data and a ResNet, for different forecasting horizons In order to further explore the difference in generalization performance between the different algorithms which we found to be efﬁcient, we trained them on smaller datasets.

Original: The PT approach is clearly valuable when training data is scarce.
Revised: Tables 4 and 5 show test results with halved datasets.

Original: It is also interesting to note that the PT is more competitive when compared to the JT for R = 100000 ,which is to be expected since this is a situation of low viscosity, more comparable to the setting the estimator has been pretrained in. Analyzing the visual outputs of the models (ﬁgures are provided in the appendix) for the two viscosity values show that the prediction task is more challenging than with Euler, especially at high viscosity : the models tend to diverge sooner from the ground-truth than for the simpler Euler.
Revised: We can see that all three algorithms are quite robust to this diminution of available training data which only slightly affects performance. However, when we make data even scarcer, in some cases as shown in table 6 where R = 5000 , the pre-trained network performs more consistently. This is understandable as those dynamics are quite complex (here the ﬂuid is very viscuous), the JT having more unconstrained parameters this explains its inability to generalize in this case while the ResNet does better but still sees its performance very affected by the lack of training of data.

Original: Note that, for the PT model, the pre-training has been done with a simpler dynamic which obviously makes the task harder than in the previous section.
Revised: We are convinced that this robustness of pre-trained networks is a very important feature and should be studied further as it could prove useful in many real-world settings, especially if combined with more unsupervised algorithms such as JT.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The dynamical system view on NN is not new.
Revised: The dynamical system view on NN is not new.

Original: In the 90s, several papers developed the analogy between recurrent neural networks and dynamical systems.
Revised: In the 90s, several papers developed the analogy between recurrent neural networks and dynamical systems.

Original: To quote only a few references Pearlmutter (1995) offers a nice review of state of the art on RNNs and their training algorithms including convergence and stability ; Tsoi & Back (1994) review RNN architectures with local recurrent connections together with their interpretation as non linear IIR/FIR ﬁlters ; Haykin (2009) introduces
Revised: To quote only a few references Pearlmutter (1995) offers a nice review of state of the art on RNNs and their training algorithms including convergence and stability ; Tsoi & Back (1994) review RNN architectures with local recurrent connections together with their interpretation as non linear IIR/FIR ﬁlters ; Haykin (2009) introduces

Original: RNNs with regard to state estimation in dynamical systems.
Revised: RNNs with regard to state estimation in dynamical systems.

Original: More recently, the success of ResNetlike architectures has motivated new developments and interpretations of NNs from a dynamical systems perspective.
Revised: More recently, the success of ResNetlike architectures has motivated new developments and interpretations of NNs from a dynamical systems perspective.

Original: The interpretation of residual blocks as implementing a simple forward Euler numerical scheme for ODE is highlighted by several authors such as Chang et al.
Revised: The interpretation of residual blocks as implementing a simple forward Euler numerical scheme for ODE is highlighted by several authors such as Chang et al.

Original: (2018b)
Revised: (2018b)

Original: ; Lu et al.
Revised: ; Lu et al.

Original: (2018); E (2017).
Revised: (2018); E (2017).

Original: For example, Lu et al.
Revised: For example, Lu et al.

Original: (2018) shows how different ResNet inspired modules can be considered as speciﬁc numerical schemes (forward or backward Euler, Runge Kutta, etc..).
Revised: (2018) shows how different ResNet inspired modules can be considered as speciﬁc numerical schemes (forward or backward Euler, Runge Kutta, etc..).

Original: Chang et al.
Revised: Chang et al.

Original: (2018b) links the depth of ResNet architectures with the size of the discretization step used in
Revised: (2018b) links the depth of ResNet architectures with the size of the discretization step used in

Original: ODE.
Revised: ODE.

Original: In Gomez et al.
Revised: In Gomez et al.

Original: ; Chang et al.
Revised: ; Chang et al.

Original: (2018a)
Revised: (2018a)

Original: , the connection with the numerical integration schemes is pushed further by exploiting the idea of reversibility in order to build architectures for which all activations can be reconstructed from the next layer, thus alleviating the need for storing the states of a networks and allowing to build larger architecture .
Revised: , the connection with the numerical integration schemes is pushed further by exploiting the idea of reversibility in order to build architectures for which all activations can be reconstructed from the next layer, thus alleviating the need for storing the states of a network and allowing to build larger architecture .

Original: Ruthotto & Haber (2018) develop
Revised: Ruthotto & Haber (2018) develop

Original: PDE inspired ResNet like models and establish stability results.
Revised: PDE inspired ResNet like models and establish stability results.

Original: This line of work concerns networks for static data and targets the development of alternative architectures or training algorithms.
Revised: This line of work concerns networks for static data and targets the development of alternative architectures or training algorithms.

Revised: Several works have already attempted to learn PDEs from data (Crutchﬁeld & Mcnamara (1987),

Revised: Alvarez et al.

Revised: More recently, Rudy et al.

Original:  For dynamic data, Kim et al.
Revised: (2017a) use sparse regression on a dictionary of differential terms to recover the underlying PDE. In Raissi & Karniadakis (2018), Raissi et al. (2017), they propose recovering the coefﬁcients of the differential terms by deriving a GP kernel from a linearized form of the PDE. For dynamic data, Kim et al.

Original: (2018) proposes NN models for ﬂuid simulations for computer graphics applications.
Revised: (2018) proposes NN models for ﬂuid simulations for computer graphics applications.

Original: The objective is to accelerate ﬂuid simulation w.r.t.
Revised: The objective is to accelerate ﬂuid simulation w.r.t.

Original: classical solvers.
Revised: classical solvers.

Original: They learn to generate 2D and 3D ﬂuid velocities (corresponding to states in our examples), using a fully supervised approach.
Revised: They learn to generate 2D and 3D ﬂuid velocities (corresponding to states in our examples), using a fully supervised approach.

Original: This shares some technical similarities with our own work, but the approach and objectives are clearly different.
Revised: This share some technical similarities with our own work but the approach and objectives are clearly different.

Original: A fully supervised NN implementing a multistep numerical scheme is also used in Long et al.
Revised: A fully supervised NN implementing a multistep numerical scheme is also used in Long et al.

Original: (2018) for learning PDE from simulations.
Revised: (2018) for learning PDEs from simulations.

Original: Raissi (2018) develops a NN framework for learning PDEs from data.
Revised: Raissi (2018) develops a NN framework for learning PDEs from data.

Original: In this work either the form of the PDE or the variable dependency is supposed to be known and the approach is again fully supervised.
Revised: In this work either the form of the PDE or the variable dependency is supposed to be known and the approach is again fully supervised over the state.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we address the problem of forecasting spatio-temporal dynamics governed by differential equations where information on the system state comes from incomplete observations.
Revised: In this work, we address the problem of forecasting spatio-temporal dynamics governed by differential equations, where information on the system state comes from incomplete observations.

Original: We have presented a general framework separating the task in two steps, state estimation and forecasting, and proposed a few instances of this framework in order to accommodatefor the indetermination in the  estimation step.
Revised: We have presented a general framework separating the task in two problems, state estimation and forecasting, and proposed several instances of this framework in order to accommodate the weakly supervised estimation step.

Original: This approach is tested on families of equations governing the dynamics of ﬂuids which are known to be challenging.
Revised: This approach is tested on families of equations governing the dynamics of ﬂuids which are known to be challenging.

Original: We have explored empirically some of the properties of the algorithms and evaluated its performance and behavior on high resolution simulated data.
Revised: We have explored empirically some of the properties of the algorithms and evaluated their performance and behavior on high resolution simulated data.

Original: We have been able to show that imposing physical priors over the estimator, by pre-training it on simple dynamics then ﬁne-tuning on the target equations, can be beneﬁcial, especially  when data is scarce, while jointly training the system without any supervision gives surprisingly good results, especially when lots of data are available.
Revised: We have been able to show that imposing physical priors over the estimator, by pre-training it on simple dynamics then ﬁne-tuning on the target equations, can be beneﬁcial. This is especially true when data is scarce. Jointly training the system without any supervision gives surprisingly good results, especially when lots of data are available.

Original: There are many ideas to explore in order to extend the results.
Revised: There are many ideas to explore in order to extend the results.

Original: Exploring whichadditional prior physical constraints could be beneﬁcial to the training and to the discovery of physically meaningful models is an important issue.
Revised: Exploring which physical constraints could be beneﬁcial to the training and to the discovery of physically meaningful models is an important issue.

Original: We also still face difﬁculties when trying to capture long-term velocity dynamics in the pre-trained setting and this has to be improved if we want to achieve a better performance.
Revised: Moreover, we still face difﬁculties when trying to capture long-term velocity dynamics in the pre-trained setting and this has to be improved if we want to achieve a better performance for the PT SSE model.

Original: Our approach  has to be tested on other families of differential equations.
Revised: Our approach also has to be tested on other families of differential equations.

Original: Furthermore, while we have been only considering simulated data until now, we must obviously aim for real data.
Revised: Furthermore, while we have been only considering simulated data, we must obviously aim for data from real-world settings.

Original: As a ﬁnal note, while we have only considered our physically inspired models for physics motivated datasets, we think that the insights provided by the different settings empirically studied in this work might prove useful for other tasks regarding the forecasting of dynamical systems
Revised: As a ﬁnal note, while we have only considered our physically inspired models for physics motivated datasets, we think that the insights provided by the different settings empirically studied in this work might prove useful for other tasks regarding the forecasting of dynamical systems.

Original: APPENDIX 8. N AVIER -S TOKES AND E ULER EQUATIONS
Revised: A N AVIER -S TOKES AND E ULER EQUATIONS

Original: Let us consider a ﬂuid blob of surface δS , velocity u and density ρ .
Revised: Let us consider a ﬂuid blob of surface δS , velocity u and density ρ .

Original: This blob has a pressure p which exerts a force −∇ pδS and is in a gravitational ﬁeld g so that Newton’s second law gives :
Revised: This blob has a pressure p which exerts a force −∇ pδS and is in a gravitational ﬁeld g so that Newton’s second law gives :

Original: [Equation]
Revised: [Equation]

Original: where Du
Revised: where Du

Original: Dt expresses the acceleration of the blob thus being the temporal derivative of the function t → u ( x ( t ) , y ( t )) which can also be expressed as :
Revised: Dt expresses the acceleration of the blob thus being the temporal derivative of the function t → u ( x ( t ) , y ( t )) which can also be expressed as :

Original: [Equation]
Revised: [Equation]

Original: Moreover, we will also make the incompressibility hypothesis which can be mathematically translated to :
Revised: Moreover, we will also make the incompressibility hypothesis which can be mathematically translated to :

Original: [Equation]
Revised: [Equation]

Original: meaning that u is of null divergence.
Revised: meaning that u is of null divergence.

Original: Finally, by deﬁnition, the mass of the blob we are studying is conserved so that :
Revised: Finally, by deﬁnition, the mass of the blob we are studying is conserved so that :

Original: [Equation]
Revised: [Equation]

Original: Those equations are the so called Euler equations and form the system :
Revised: Those equations are the so called Euler equations and form the system :

Original: [Equation]
Revised: [Equation]

Original: When deriving the Euler equations, we never considered the particular nature of the ﬂuid.
Revised: When deriving the Euler equations, we never considered the particular nature of the ﬂuid.

Original: This is taken into account by considering an additional force in the previous derivation : viscosity.
Revised: This is taken into account by considering an additional force in the previous derivation : viscosity.

Original: Even for small values, this additional force is of great importance :
Revised: Even for small values, this additional force is of great importance :

Original: For example, while air usually has quite small viscosity, without it one wouldn’t be able to explain why planes ﬂy.
Revised: For example, while air usually has quite small viscosity, without it one wouldn’t be able to explain why planes ﬂy.

Original: For newtonian ﬂuids 3 , viscosity can be taken into account with an additional term ν ∇ 2 u where ν
Revised: For Newtonian ﬂuids 2 , viscosity can be taken into account with an additional term ν ∇ 2 u where ν

Original: = µ ρ is the kinematic viscosity .
Revised: = µ ρ is the kinematic viscosity .

Original: This gives us the following equation system :
Revised: This gives us the following equation system :

Original: [Equation]
Revised: [Equation]

Original: Those are the Navier-Stokes equations which are central to ﬂuid dynamics.
Revised: Those are the Navier-Stokes equations which are central to ﬂuid dynamics.

Original: We can easily see how the Euler equations can be considered as a simpliﬁcation of this system, found by taking ν = 0 .
Revised: We can easily see how the Euler equations can be considered as a simpliﬁcation of this system, found by taking ν = 0 .

Original: The importance of viscosity in a ﬂuid, and thus a way to assess its qualitative behaviour, cannot be measured through ν alone as it has to be compared against the importance of advection.
Revised: The importance of viscosity in a ﬂuid, and thus a way to assess its qualitative behaviour, cannot be measured through ν alone as it has to be compared against the importance of advection.

Original: The most commonly used quantity to describe it is the Reynolds number .
Revised: The most commonly used quantity to describe it is the Reynolds number .

Original: If we take U to be a typical speed and L a typical distance of the studied ﬂow then it can be expressed as :
Revised: If we take U to be a typical speed and L a typical distance of the studied ﬂow then it can be expressed as :

Original: [Equation]
Revised: [Equation]

Original: To understand its meaning, let us recall that we have ( u · ∇ )
Revised: To understand its meaning, let us recall that we have ( u · ∇ )

Original: u ∼ U
Revised: u ∼ U

Revised: [Equation]

Original: L and ν ∇ 2 u ∼ νU L 2 so that :
Revised: and ν ∇ 2 u ∼ νU L 2 so that :

Original: [Equation]
Revised: [Equation]

Original: which means that R is the ratio of the inertia and the viscosity terms.
Revised: which means that R is the ratio of the inertia and the viscosity terms.

Original: Thus, if R is small, it means that the ﬂuid is very viscous, thus behaving more like honey ;
Revised: Thus, if R is small, it means that the ﬂuid is very viscous, thus behaving more like honey ;

Original: If R is large, the viscosity term is small and the ﬂuid behaves more like water (which doesn’t always mean that every such ﬂuid can be modelled with the Eulerian approximation as viscosity can play a role even when the corresponding term is small).
Revised: If R is large, the viscosity term is small and the ﬂuid behaves more like water (which doesn’t always mean that every such ﬂuid can be modelled with the Eulerian approximation as viscosity can play a role even when the corresponding term is small).

Original: 8. L ERAY PROJECTION : E ULER AND N AVIER STOKES EXPRESSED IN THE FORM OF EQUATION
Revised: B L ERAY PROJECTION : E ULER AND N AVIER STOKES EXPRESSED IN THE FORM OF EQUATION

Original: The two systems equation 8.1 and equation 8.1 are not of the form equation 1 as we still have the pressure variable p as well as the null divergence constraint.
Revised: The two systems equation A and equation A are not of the form equation 1 as we still have the pressure variable p as well as the null divergence constraint.

Original: Let us show how we can transform those equations in order to get rid of it.
Revised: Let us show how we can transform those equations in order to get rid of it.

Original: The Helmholz-Leray decomposition result states that for any vector ﬁeld a , there exists b and c such that : 
Revised: The Helmholz-Leray decomposition result states that for any vector ﬁeld a , there exists b and c such that : [Equation] and

Original: [Equation]
Revised: [Equation]

Original: where ∇· is the divergence operator.
Revised: where ∇· is the divergence operator.

Original: Moreover, this pair is unique up to an additive constant for b .
Revised: Moreover, this pair is unique up to an additive constant for b .

Original: Thus, we can deﬁne a linear operator P by :
Revised: Thus, we can deﬁne a linear operator P by :

Original: [Equation]
Revised: [Equation]

Original: This operator is a continuous linear projector which is the identity for divergence-free vector ﬁelds and vanishes for those deriving from a potential.
Revised: This operator is a continuous linear projector which is the identity for divergence-free vector ﬁelds and vanishes for those deriving from a potential.

Original: Let us take a solution of NS and apply P on the ﬁrst equation of equation 8.1, we have, as u is divergence free from the third equation and as g derives from a potential :
Revised: Let us take a solution of NS and apply P on the ﬁrst equation of equation A, we have, as u is divergence free from the third equation and as g derives from a potential :

Original: [Equation]
Revised: [Equation]

Original: where permuting derivation and P is justiﬁed by the continuity of the operator 4 .
Revised: where permuting derivation and P is justiﬁed by the continuity of the operator 3 .

Original: Thus, if u is solution to equation 8.1, it is also a solution to :
Revised: Thus, if u is solution to equation A, it is also a solution to :

Original: [Equation]
Revised: [Equation]

Original: which is of the form of equation
Revised: which is of the form of equation

Original: Conversely, the solution of the above system is such that :
Revised: Conversely, the solution of the above system is such that :

Original: [Equation]
Revised: [Equation]

Original: which gives, by exchanging P and the integral 5 :
Revised: which gives, by exchanging P and the integral 4 :

Original: [Equation]
Revised: [Equation]

Original: so that u is automatically of null divergence by deﬁnition of P .
Revised: so that u is automatically of null divergence by deﬁnition of P .

Original: The two systems are thus equivalent.
Revised: The two systems are thus equivalent.

Revised: C E XPRESSING THE STATE AS A FUNCTION OF OBSERVATIONS

Revised: All our work relies on the assumption stated in equation equation 3 which supposes that states can be expressed as a deterministic function of observations.

Revised: This is obviously not always true : If the observations are uncorrelated with the true state, one can certainly not retrieve it.

Original: [Table]
Revised: Thus, if one considers a physical system modelled by a state X t , then we are actually estimating the conditional expectation of this state given a sequence of observations ( Y t − k +1 , ..., Y t ) . Here, the stochasticity could come from some noise in the operator H as well as from the fact that the observations may not contain enough information to reconstruct the state. In other words, our assumption amounts to suppose that there exists k such that X t is { Y t − k , ..., Y t } -measurable. Another take on this matter would be to consider that what we want is simply to construct a state which is sufﬁcient for the forecasting.

Revised: In other words, in the language of dynamical systems, we want an embedding of the attractor our system is in which can be constructed with a sequence of observations.

Revised: The celebrated Takens embedding theorem tells us precisely that, for a prevalent class of observation functions H , there is a k such that, by putting X t

Revised: = ( Y t − k +1 , ..., Y t ) , there exists F for which F ( X t ) =

Revised: X t +1 .

Revised: In our case, this means that, as long as all we want is to have a state allowing the forecasting of observations, which is the underlying motivation of the JT models, our assumption is veriﬁed for more or less any observation operator H 5 .

Revised: All this being said, our overall intuition is that this assumption is not very restrictive in the deterministic case and it should work as long the observations are meaningful enough.

Revised: In the noisy case, which is not studied in this paper, one would have to take into account the sensibility of the studied dynamics to initial conditions which, in the case of chaotic systems, necessitates to control well enough the conditional variance at the estimation step.

Revised: D C OMPARISON OF TRAINING SEQUENCE LENGTHS

Revised: t 0 +t 0 +t 0 +t 0 +t 0 +t 0 +t 0 +

Original: [Table] 
Revised: [Table] [Table]

Revised: t 0 +t 0 +t 0 +t 0 +t 0 +t 0 +t 0 +

Revised: [Table]

Revised: [Table]

Revised: Tables 7 and 8 show results for different training horizons l for the two algorithms PT SSE and JT

Revised: MSRE.

Revised: We can clearly see that, at least up to a certain point, the longer the training horizon, the better the long term performance for both algorithms.

Revised: This is to be expected : A longer training horizon acts as an additional constraint for the forecasting operator and thus ensures a better generalization.

Original: 8. C OMPARISON OF SAMPLING STRATEGIES
Revised: This observation is especially true for the PT algorithm : The improvements observed for the PT algorithm in table 8 are higher than the improvements for the JT algorithm in table 7 and the PT algorithm supersedes JT at longer horizons. This seems to indicate that the state representation with [Table]

Revised: [Table]

Revised: a physical meaning is more stable for long term forecasting and beneﬁts more from this additional constraining.

Revised: However, we still keep l = 8 for the other experiments for computational reasons.

Original: Tables 6 and 7 compare the 3 sampling strategies introduced in section 3, namely systematic teacher forcing, no teacher forcing and scheduled sampling.
Revised: Tables 9 and 10 compare the 3 sampling strategies introduced in section 3, namely systematic teacher forcing, no teacher forcing and scheduled sampling.

Original: For the latter, we consider a decreasing sequence ( (cid:15) i ) i , starting from 1 and converging to 0 , so that, at time i , we pick Y t − 1 with probability (cid:15)
Revised: For the latter, we consider a decreasing sequence ( (cid:15) i ) i , starting from 1 and converging to 0 , so that, at time i , we pick Y t − 1 with probability (cid:15)

Original: i and (cid:98)
Revised: i and (cid:98)

Original: Y t − 1 with probability 1 − (cid:15) i .
Revised: Y t − 1 with probability 1 − (cid:15) i .

Original: This amounts to starting with one-step predictions then gradually increasing the time horizon as the model improves.
Revised: This amounts to starting with one-step predictions then gradually increasing the time horizon as the model improves.

Original: The ( (cid:15) i )
Revised: The ( (cid:15) i )

Original: i sequence has to decay at the same pace as the error of the model and thus has to be chosen carefully depending on the trained model.
Revised: i sequence has to decay at the same pace as the error of the model and thus has to be chosen carefully depending on the trained model.

Original: In this work, we have considered the exponential scheme :
Revised: In this work, we have considered the exponential scheme :

Original: [Equation]
Revised: [Equation]

Original:  teacher forcing performs poorly, surprisingly not using any teacher forcing works remarkably well,  as well as scheduled sampling while being much faster, which might be used to accelerate training while keeping a good performance.
Revised: Results show that using systematic teacher forcing performs poorly, which is to be expected as it is equivalent to a training horizon of 1 , and, surprisingly, not using any teacher forcing works remarkably well, almost as well as scheduled sampling while training much faster which might be used to accelerate training.

Original: In all the experiments, wehowever used scheduled sampling.
Revised: However, in all the experiments, we used scheduled sampling nonetheless.

Original: 8. A DDITIONAL FIGURES
Revised: F A DDITIONAL FIGURES

Original: We provide below some example of forecasting images obtained for Navier Stokes equations.
Revised: We provide below some example of forecasting images obtained for Navier Stokes equations.

Original: The setting of the experiments is introduced in the main text in section 5.
Revised: The setting of the experiments is introduced in the main text in section 5.

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]