Original: P HYSICS - AWARE , PROBABILISTIC MODEL ORDER REDUCTION WITH GUARANTEED STABILITY
Revised: P HYSICS - AWARE , PROBABILISTIC MODEL ORDER REDUCTION WITH GUARANTEED STABILITY

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Given (small amounts of) time-series’ data from a high-dimensional, ﬁne-grained, multiscale dynamical system, we propose a generative framework for learning an effective, lower-dimensional, coarse-grained dynamical model that is predictive of the ﬁne-grained system’s long-term evolution but also of its behavior under different initial conditions.
Revised: Given (small amounts of) time-series’ data from a high-dimensional, ﬁne-grained, multiscale dynamical system, we propose a generative framework for learning an effective, lower-dimensional, coarse-grained dynamical model that is predictive of the ﬁne-grained system’s long-term evolution but also of its behavior under different initial conditions.

Original: We target ﬁne-grained models as they arise in physical applications (e.g. molecular dynamics, agent-based models), the dynamics of which are strongly non-stationary but their transition to equilibrium is governed by unknown slow processes which are largely inaccessible by brute-force simulations.
Revised: We target ﬁne-grained models as they arise in physical applications (e.g. molecular dynamics, agent-based models), the dynamics of which are strongly non-stationary but their transition to equilibrium is governed by unknown slow processes which are largely inaccessible by brute-force simulations.

Original: Approaches based on domain knowledge heavily rely on physical insight in identifying temporally slow features and fail to enforce the long-term stability of the learned dynamics.
Revised: Approaches based on domain knowledge heavily rely on physical insight in identifying temporally slow features and fail to enforce the long-term stability of the learned dynamics.

Original: On the other hand, purely statistical frameworks lack interpretability and rely on large amounts of expensive simulation data (long and multiple trajectories) as they cannot infuse domain knowledge.
Revised: On the other hand, purely statistical frameworks lack interpretability and rely on large amounts of expensive simulation data (long and multiple trajectories) as they cannot infuse domain knowledge.

Original: The generative framework proposed achieves the aforementioned desiderata by employing a ﬂexible prior on the complex plane for the latent, slow processes, and an intermediate layer of physics-motivated latent variables that reduces reliance on data and imbues inductive bias.
Revised: The generative framework proposed achieves the aforementioned desiderata by employing a ﬂexible prior on the complex plane for the latent, slow processes, and an intermediate layer of physics-motivated latent variables that reduces reliance on data and imbues inductive bias.

Original: In contrast to existing schemes, it does not require the a priori deﬁnition of projection operators or encoders and addresses simultaneously the tasks of dimensionality reduction and model estimation.
Revised: In contrast to existing schemes, it does not require the a priori deﬁnition of projection operators or encoders and addresses simultaneously the tasks of dimensionality reduction and model estimation.

Original: We demonstrate its efﬁcacy and accuracy in multiscale physical systems of particle dynamics where probabilistic, long-term predictions of phenomena not contained in the training data are produced.
Revised: We demonstrate its efﬁcacy and accuracy in multiscale physical systems of particle dynamics where probabilistic, long-term predictions of phenomena not contained in the training data are produced.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: High-dimensional, nonlinear systems are ubiquitous in engineering and computational physics.
Revised: High-dimensional, nonlinear systems are ubiquitous in engineering and computational physics.

Original: Their nature is in general multi-scale 1 .
Revised: Their nature is in general multi-scale 1 .

Original: E.g. in materials, defects and cracks occur on scales of millimeters to centimeters whereas the atomic processes responsible for such defects take place at much ﬁner scales (Belytschko & Song, 2010).
Revised: E g. in materials, defects and cracks occur on scales of millimeters to centimeters whereas the atomic processes responsible for such defects take place at much ﬁner scales (Belytschko & Song, 2010).

Original: Local oscillations due to bonded interactions of atoms (Smit, 1996) take place at time scales of femtoseconds ( 10 − 15 s ), whereas protein folding processes which can be relevant for e.g. drug discovery happen at time scales larger than milliseconds ( 10 − 3 s ).
Revised: Local oscillations due to bonded interactions of atoms (Smit, 1996) take place at time scales of femtoseconds ( 10 − 15 s ), whereas protein folding processes which can be relevant for e.g. drug discovery happen at time scales larger than milliseconds ( 10 − 3 s ).

Original: In Fluid Mechanics, turbulence phenomena are characterized by ﬁne-scale spatiotemporal ﬂuctuations which affect the coarse-scale response (Laizet & Vassilicos, 2009).
Revised: In Fluid Mechanics, turbulence phenomena are characterized by ﬁne-scale spatiotemporal ﬂuctuations which affect the coarse-scale response (Laizet & Vassilicos, 2009).

Original: In all of these cases, macroscopic observables are the result of microscopic phenomena and a better understanding of the interactions between the different scales would be highly beneﬁcial for predicting the system’s evolution (Givon et al., 2004).
Revised: In all of these cases, macroscopic observables are the result of microscopic phenomena and a better understanding of the interactions between the different scales would be highly beneﬁcial for predicting the system’s evolution (Givon et al., 2004).

Original: The identiﬁcation of the different scales, their dynamics and connections however is a non-trivial task and is challenging from the perspective of statistical as well as physical modeling.
Revised: The identiﬁcation of the different scales, their dynamics and connections however is a non-trivial task and is challenging from the perspective of statistical as well as physical modeling.

Original: In this paper we propose a novel physics-aware, probabilistic model order reduction framework with guaranteed stability that combines recent advances in statistical learning with a hierarchical architecture that promotes the discovery of interpretable, low-dimensional representations.
Revised: In this paper we propose a novel physics-aware, probabilistic model order reduction framework with guaranteed stability that combines recent advances in statistical learning with a hierarchical architecture that promotes the discovery of interpretable, low-dimensional representations.

Original: We employ a generative state-space model with two layers of latent variables.
Revised: We employ a generative state-space model with two layers of latent variables.

Original: The ﬁrst describes the latent dynamics using a novel prior on the complex plane that guarantees stability and yields a clear distinction between fast and slow processes, the latter being responsible for the system’s long-term evolution.
Revised: The ﬁrst describes the latent dynamics using a novel prior on the complex plane that guarantees stability and yields a clear distinction between fast and slow processes, the latter being responsible for the system’s long-term evolution.

Original: The second layer involves physically-motivated latent variables which infuse inductive bias, enable connections with the very high-dimensional observables and reduce the data requirements for training.
Revised: The second layer involves physically-motivated latent variables which infuse inductive bias, enable connections with the very high-dimensional observables and reduce the data requirements for training.

Original: The probabilistic formulation adopted enables the quantiﬁcation of a crucial, and often neglected, component in any model compression process, i.e. the predictive uncertainty due to information loss.
Revised: The probabilistic formulation adopted enables the quantiﬁcation of a crucial, and often neglected, component in any model compression process, i.e. the predictive uncertainty due to information loss.

Original: We ﬁnally want to emphasize that the problems of interest are Small Data ones due to the computational expense of the physical simulators.
Revised: We ﬁnally want to emphasize that the problems of interest are Small Data ones due to the computational expense of the physical simulators.

Original: Hence the number of time-steps as well as the number of time-series used for training is small as compared to the dimension of the system and to the time-horizon over which predictions are sought.
Revised: Hence the number of time-steps as well as the number of time-series used for training is small as compared to the dimension of the system and to the time-horizon over which predictions are sought.

Original:  P HYSICS -A WARE , P ROBABILISTIC M ODEL O RDER R EDUCTION
Revised:  P HYSICS -A WARE , P ROBABILISTIC M ODEL O RDER R EDUCTION

Original: Our data consists of N times-series { x ( i ) 0: T } Ni =1 over T time-steps generated by a computational physics simulator.
Revised: Our data consists of N times-series { x ( i ) 0: T } Ni =1 over T time-steps generated by a computational physics simulator.

Original: This can represent positions and velocities of each particle in a ﬂuid or those of atoms in molecular dynamics.
Revised: This can represent positions and velocities of each particle in a ﬂuid or those of atoms in molecular dynamics.

Original: Their dimension is generally very high i.e. x t ∈ M ⊂ R f ( f >> 1 ).
Revised: Their dimension is generally very high i.e. x t ∈ M ⊂ R f ( f >> 1 ).

Original: In the context of state-space models, the goal is to ﬁnd a lower-dimensional set of collective variables or latent generators z t and their associated dynamics.
Revised: In the context of state-space models, the goal is to ﬁnd a lower-dimensional set of collective variables or latent generators z t and their associated dynamics.

Original: Given the difﬁculties associated with these tasks and the solutions that have been proposed in statistics and computational physics literature, we advocate the use of an intermediate layer of physically-motivated, lower-dimensional variables
Revised: Given the difﬁculties associated with these tasks and the solutions that have been proposed in statistics and computational physics literature, we advocate the use of an intermediate layer of physically-motivated, lower-dimensional variables

Original: X t (e.g. density or velocity ﬁelds), the meaning of which will become precise in the next sections.
Revised: X t (e.g. density or velocity ﬁelds), the meaning of which will become precise in the next sections.

Original: These variables provide a coarse-grained description of the high-dimensional observables and imbue interpretability in the learned dynamics.
Revised: These variables provide a coarse-grained description of the high-dimensional observables and imbue interpretability in the learned dynamics.

Original: Using X t alone (without z t ) would make it extremely difﬁcult to enforce long-term stability (see Appendix H.2) while ensuring sufﬁcient complexity in the learned dynamics (Felsberger & Koutsourelakis, 2019; Champion et al., 2019).
Revised: Using X t alone (without z t ) would make it extremely difﬁcult to enforce long-term stability (see Appendix H.2) while ensuring sufﬁcient complexity in the learned dynamics (Felsberger & Koutsourelakis, 2019; Champion et al., 2019).

Original: Furthermore and even if the dynamics of x t are ﬁrst-order Markovian, this is not necessarily the case for X t (Chorin & Stinis, 2007).
Revised: Furthermore and even if the dynamics of x t are ﬁrst-order Markovian, this is not necessarily the case for X t (Chorin & Stinis, 2007).

Original: The latent variables z t therefore effectively correspond to a nonlinear coordinate transformation that yields not only Markovian but also stable dynamics (Gin et al., 2019).
Revised: The latent variables z t therefore effectively correspond to a nonlinear coordinate transformation that yields not only Markovian but also stable dynamics (Gin et al., 2019).

Original: The general framework is summarized in Figure 1 and we provide details in the next section.
Revised: The general framework is summarized in Figure 1 and we provide details in the next section.

Original: 2. M ODEL S TRUCTURE
Revised: 2. M ODEL S TRUCTURE

Original: Our model consists of three levels.
Revised: Our model consists of three levels.

Original: At the ﬁrst level, we have the latent variables z t which are connected with X t in the second layer through a probabilistic map G .
Revised: At the ﬁrst level, we have the latent variables z t which are connected with X t in the second layer through a probabilistic map G .

Original: The physical variables X t are ﬁnally connected to the high-dimensional observables through another probabilistic map F .
Revised: The physical variables X t are ﬁnally connected to the high-dimensional observables through another probabilistic map F .

Original: We parametrize F , G with deep neural networks and denote by θ 1 and θ 2 the corresponding parameters (see Appendix D).
Revised: We parametrize F , G with deep neural networks and denote by θ 1 and θ 2 the corresponding parameters (see Appendix D).

Original: In particular, we postulate the following relations:
Revised: In particular, we postulate the following relations:

Original: [Equation]
Revised: [Equation]

Original: We assume that the latent variables z t are complex-valued and a priori independent.
Revised: We assume that the latent variables z t are complex-valued and a priori independent.

Original: Complex variables were chosen as their evolution includes a harmonic components which are observed in many physical systems.
Revised: Complex variables were chosen as their evolution includes a harmonic components which are observed in many physical systems.

Original: In Appendix H.1 we present results with a real-valued latent variables z t,j and illustrate their limitations.
Revised: In Appendix H.1 we present results with a real-valued latent variables z t,j and illustrate their limitations.

Original: We model their dynamics with a discretized Ornstein-Uhlenbeck process on the complex plane with initial conditions z 0 ,j ∼ CN (0 , σ 20 ,j ) 2 .
Revised: We model their dynamics with a discretized Ornstein-Uhlenbeck process on the complex plane with initial conditions z 0 ,j ∼ CN (0 , σ 20 ,j ) 2 .

Original: The parameters associated with this level are denoted summarily by θ 

Revised: The parameters associated with this level are denoted summarily by θ

Original: = { σ 20 ,j , σ 2 j , λ j } hj =1 .
Revised: = { σ 20 ,j , σ 2 j , λ j } hj =1 .

Original: These, along with θ 1 , θ 2 mentioned earlier, and the state variables X t and z t have to be inferred from the data x t .
Revised: These, along with θ 1 , θ 2 mentioned earlier, and the state variables X t and z t have to be inferred from the data x t .

Original: We explain each of the aforementioned components in the sequel.
Revised: We explain each of the aforementioned components in the sequel.

Original: 2.1. S TABLE LOW - DIMENSIONAL D YNAMICS
Revised: 2.1. S TABLE LOW - DIMENSIONAL D YNAMICS

Original: While the physical systems (e.g. particle dynamics) of interest are highly non-stationary, they generally converge to equilibrium in the long-term.
Revised: While the physical systems (e.g. particle dynamics) of interest are highly non-stationary, they generally converge to equilibrium in the long-term.

Original: We enforce long-term stability here by ensuring that the real-part of the λ j ’s in Equation (1) is negative, i.e.:
Revised: We enforce long-term stability here by ensuring that the real-part of the λ j ’s in Equation (1) is negative, i.e.:

Original: [Equation]
Revised: [Equation]

Original: which guarantees ﬁrst and second-order stability i.e. the mean as well as the variance are bounded at all time steps.
Revised: which guarantees ﬁrst and second-order stability i.e. the mean as well as the variance are bounded at all time steps.

Original: The transition density each process z t,j is given by:
Revised: The transition density each process z t,j is given by:

Original: [Equation]
Revised: [Equation]

Original: where the orthogonal matrix R j depends on the imaginary part of λ j :
Revised: where the orthogonal matrix R j depends on the imaginary part of λ j :

Original: [Equation]
Revised: [Equation]

Original: and the decay rate s j depends on the real part of λ j :
Revised: and the decay rate s j depends on the real part of λ j :

Original: [Equation]
Revised: [Equation]

Original: i.e. the closer to zero the latter is, the ”slower” the evolution of the corresponding process is.
Revised: i.e. the closer to zero the latter is, the ”slower” the evolution of the corresponding process is.

Original: As in probabilistic Slow Feature Analysis (SFA) (Turner & Sahani, 2007; Zafeiriou et al., 2015), we set σ 2 j = 1 − exp(2 (cid:60) ( λ j ))
Revised: As in probabilistic Slow Feature Analysis (SFA) (Turner & Sahani, 2007; Zafeiriou et al., 2015), we set σ 2 j = 1 − exp(2 (cid:60) ( λ j ))

Original: = 1 − s 2 j and σ 20 ,j = 1 .
Revised: = 1 − s 2 j and σ 20 ,j = 1 .

Original: As a consequence, a priori, the latent dynamics are stationary 3 and an ordering of the processes z t,j is possible on the basis of (cid:60) ( λ j ) .
Revised: As a consequence, a priori, the latent dynamics are stationary 3 and an ordering of the processes z t,j is possible on the basis of (cid:60) ( λ j ) .

Original: Hence the only independent parameters are the λ j , the imaginary part of which can account for periodic effects in the latent dynamics (see Appendix B).
Revised: Hence the only independent parameters are the λ j , the imaginary part of which can account for periodic effects in the latent dynamics (see Appendix B).

Original: The joint density of z t can ﬁnally be expressed as:
Revised: The joint density of z t can ﬁnally be expressed as:

Original: [Equation]
Revised: [Equation]

Original: The transition density between states at non-neighbouring time-instants is also available analytically and is useful for training on longer trajectories or in cases of missing data.
Revised: The transition density between states at non-neighbouring time-instants is also available analytically and is useful for training on longer trajectories or in cases of missing data.

Original: Details can be found in
Revised: Details can be found in

Original: Appendix B.
Revised: Appendix B.

Original: 2.1. P ROBABILISTIC G ENERATIVE M APPING
Revised: 2.1. P ROBABILISTIC G ENERATIVE M APPING

Original: We employ fully probabilistic maps between the different layers which involve two conditional densities based on Equations (2) and (3), i.e.: 
Revised: We employ fully probabilistic maps between the different layers which involve two conditional densities based on Equations (2) and (3), i.e.: p [Equation] and

Original: [Equation]
Revised: [Equation]

Original: In contrast to the majority of physics-motivated papers (Chorin & Stinis, 2007; Champion et al., 2019) as well as those based on transfer-operators Klus et al.
Revised: In contrast to the majority of physics-motivated papers (Chorin & Stinis, 2007; Champion et al., 2019) as well as those based on transfer-operators Klus et al.

Original: , we note that the generative structure adopted does not require the prescription of a restriction operator (or encoder) and the reduced variables need not be selected a priori but rather are adapted to best reconstruct the observables.
Revised: , we note that the generative structure adopted does not require the prescription of a restriction operator (or encoder) and the reduced variables need not be selected a priori but rather are adapted to best reconstruct the observables.

Original: The splitting of the generative mapping into two parts through the introduction of the intermediate variables X t has several advantages.
Revised: The splitting of the generative mapping into two parts through the introduction of the intermediate variables X t has several advantages.

Original: Firstly, known physical dependencies between the data x and the physical variables X can be taken into account, which reduces the complexity of the associated maps and the total number of parameters.
Revised: Firstly, known physical dependencies between the data x and the physical variables X can be taken into account, which reduces the complexity of the associated maps and the total number of parameters.

Original: For instance, in the case of particle simulations where
Revised: For instance, in the case of particle simulations where

Original: X represents a density or velocity ﬁeld, i.e. it provides a coarsened or averaged description of the ﬁne-scale observables, it can be used to (probabilistically) reconstruct the positions or velocities of the particles.
Revised: X represents a density or velocity ﬁeld, i.e. it provides a coarsened or averaged description of the ﬁne-scale observables, it can be used to (probabilistically) reconstruct the positions or velocities of the particles.

Original: This physical information can be used to compensate for the lack of data when only few training sequences are available (Small data) and can seen as a strong prior to the model order reduction framework.
Revised: This physical information can be used to compensate for the lack of data when only few training sequences are available (Small data) and can seen as a strong prior to the model order reduction framework.

Original: Due to the lower dimension of associated variables, the generative map between z t and X t can be more easily learned even with few training samples.
Revised: Due to the lower dimension of associated variables, the generative map between z t and X t can be more easily learned even with few training samples.

Original: Lastly, the inferred physical variables X can provide insight and interpretability to the analysis of the physical system.
Revised: Lastly, the inferred physical variables X can provide insight and interpretability to the analysis of the physical system.

Original: 2. I NFERENCE AND L EARNING
Revised: 2. I NFERENCE AND L EARNING

Original: Given the probabilistic relations above, our goal is to infer the state variables X (1: n ) 0: T , z (1: n ) 0: T as well as all model parameters θ .
Revised: Given the probabilistic relations above, our goal is to infer the state variables X (1: n ) 0: T , z (1: n ) 0: T as well as all model parameters θ .

Original: We follow a hybrid Bayesian approach in which the posterior of the state variables is approximated using structured Stochastic Variational Inference (Hoffman et al., 2013) and MAP point estimates for θ =
Revised: We follow a hybrid Bayesian approach in which the posterior of the state variables is approximated using structured Stochastic Variational Inference (Hoffman et al., 2013) and MAP point estimates for θ =

Original: { θ 0 , θ 1 , θ 2 } are computed.
Revised: { θ 0 , θ 1 , θ 2 } are computed.

Original: The application of Bayes’ rule leads to the following posterior:
Revised: The application of Bayes’ rule leads to the following posterior:

Original: [Equation]
Revised: [Equation]

Original: where p ( θ ) denotes the prior on the model parameters.
Revised: where p ( θ ) denotes the prior on the model parameters.

Original: In the context of variational inference, we use the following factorization of the approximate posterior 4 :
Revised: In the context of variational inference, we use the following factorization of the approximate posterior 4 :

Original: [Equation]
Revised: [Equation]

Original: We approximate the conditional posterior of z given X with a complex multivariate normal which is parameterized using a tridiagonal precision matrix as proposed in Archer et al. (2015);
Revised: We approximate the conditional posterior of z given X with a complex multivariate normal which is parameterized using a tridiagonal precision matrix as proposed in Archer et al. (2015);

Original: Bamler & Mandt (2017).
Revised: Bamler & Mandt (2017).

Original: This retains dependencies between temporally neighbouring z , but the number of parameters grows linearly with the dimension of z which leads to a highly scalable algorithm.
Revised: This retains dependencies between temporally neighbouring z , but the number of parameters grows linearly with the dimension of z which leads to a highly scalable algorithm.

Original: For the variational posterior of X we employ a Gaussian with a diagonal covariance, i.e.:q φ [Equation] i
Revised: For the variational posterior of X we employ a Gaussian with a diagonal covariance, i.e.:

Original: [Equation]
Revised: [Equation]

Original: We denote summarily with φ the parameters involved and note that deep neural networks are used for the mean µ φ ( X ( i ) 0: T ) as well as the upper bidiagonal matrix B φ ( X ( i ) 0: T ) .
Revised: We denote summarily with φ the parameters involved and note that deep neural networks are used for the mean µ φ ( X ( i ) 0: T ) as well as the upper bidiagonal matrix B φ ( X ( i ) 0: T ) .

Original: Details on the neural net architectures employed are provided in Section 4 and in Appendix D.
Revised: Details on the neural net architectures employed are provided in Section 4 and in Appendix D.

Original: It can be readily shown that the optimal parameter values are found by maximizing the Evidence
Revised: It can be readily shown that the optimal parameter values are found by maximizing the Evidence

Original: Lower Bound (ELBO) F ( q φ ( X (1: n ) 0: T , z (1: n ) 0: T ) , θ ) which is derived in Appendix C.
Revised: Lower Bound (ELBO) F ( q φ ( X (1: n ) 0: T , z (1: n ) 0: T ) , θ ) which is derived in Appendix C.

Original: We compute
Revised: We compute

Original: Monte Carlo estimates of the gradient of the ELBO with respect to φ and θ with the help of the reparametrization trick (Kingma & Welling, 2013) and carry out stochastic optimization with the
Revised: Monte Carlo estimates of the gradient of the ELBO with respect to φ and θ with the help of the reparametrization trick (Kingma & Welling, 2013) and carry out stochastic optimization with the

Original: ADAM algorithm (Kingma & Ba, 2014).
Revised: ADAM algorithm (Kingma & Ba, 2014).

Original: 2. P REDICTIONS
Revised: 2. P REDICTIONS

Original: Once state variables have been inferred and MAP estimates θ MAP for the model parameters have been obtained, the reduced model can be used for probabilistic future predictions.
Revised: Once state variables have been inferred and MAP estimates θ MAP for the model parameters have been obtained, the reduced model can be used for probabilistic future predictions.

Original: In order to do so for a time sequence used in training, we employ the following Monte Carlo scheme to generate a sample x T + P , i.e. P time-steps into the future:
Revised: In order to do so for a time sequence used in training, we employ the following Monte Carlo scheme to generate a sample x T + P , i.e. P time-steps into the future:

Original: Sample X T and z T from the inferred posterior q φ ( z 0: T | X 0: T ) q φ ( X 0: T ) .
Revised: Sample X T and z T from the inferred posterior q φ ( z 0: T | X 0: T ) q φ ( X 0: T ) .

Original: Propagate z T for P time steps forward by using the conditional density in Equation (5).
Revised: Propagate z T for P time steps forward by using the conditional density in Equation (5).

Original: Sample X T + P and x T + P from p ( X T + P | z T + P , θ MAP) and p ( x T + P | X T + P , θ MAP) respectively.
Revised: Sample X T + P and x T + P from p ( X T + P | z T + P , θ MAP) and p ( x T + P | X T + P , θ MAP) respectively.

Original: More importantly perhaps, the trained model can be used for predictions under new initial conditions, e.g. x 0 .
Revised: More importantly perhaps, the trained model can be used for predictions under new initial conditions, e.g. x 0 .

Original: To achieve this, ﬁrst the posterior p ( z 0 | x 0 ) ∝ (cid:82) p ( x 0 | X 0 , θ MAP) p ( X 0 | z 0 , θ MAP)
Revised: To achieve this, ﬁrst the posterior p ( z 0 | x 0 ) ∝ (cid:82) p ( x 0 | X 0 , θ MAP) p ( X 0 | z 0 , θ MAP)

Original: d X 0 must be found before the Monte Carlo steps above can be employed starting at
Revised: d X 0 must be found before the Monte Carlo steps above can be employed starting at

Original: T = 0 .
Revised: T = 0 .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The main theme of our work is the learning of low-dimensional dynamical representations that are stable, interpretable and make use of physical knowledge.
Revised: The main theme of our work is the learning of low-dimensional dynamical representations that are stable, interpretable and make use of physical knowledge.

Original: Linear latent dynamics:
Revised: Linear latent dynamics:

Original: In this context, the line of work that most closely resembles ours pertains to the use of Koopman-operator theory (Koopman, 1931) which attempts to identify appropriate transformations of the original coordinates that yield linear dynamics (Klus et al., 2018).
Revised: In this context, the line of work that most closely resembles ours pertains to the use of Koopman-operator theory (Koopman, 1931) which attempts to identify appropriate transformations of the original coordinates that yield linear dynamics (Klus et al., 2018).

Original: We note that these approaches (Lusch et al., 2018; Champion et al., 2019;
Revised: We note that these approaches (Lusch et al., 2018; Champion et al., 2019;

Original: Gin et al., 2019; Lee & Carlberg, 2020) require additionally the speciﬁcation of an encoder i.e. a map from the original description to the reduced coordinates which we avoid in the generative formulation adopted.
Revised: Gin et al., 2019; Lee & Carlberg, 2020) require additionally the speciﬁcation of an encoder i.e. a map from the original description to the reduced coordinates which we avoid in the generative formulation adopted.

Original: Furthermore only a small fraction are probabilistic and can quantify predictive uncertainties but very often employ restrictive parametrizations for the Koopman matrix in order to ensure long-term stability (Pan &
Revised: Furthermore only a small fraction are probabilistic and can quantify predictive uncertainties but very often employ restrictive parametrizations for the Koopman matrix in order to ensure long-term stability (Pan &

Original: Duraisamy, 2020).
Revised: Duraisamy, 2020).

Original: To the best of our knowledge, none of the works along these lines employ additional, physically-motivated variables and as a result have demonstrated their applicability only in lower-dimensional problems and require very large amounts of training data or some ad hoc preprocessing.
Revised: To the best of our knowledge, none of the works along these lines employ additional, physically-motivated variables and as a result have demonstrated their applicability only in lower-dimensional problems and require very large amounts of training data or some ad hoc preprocessing.

Original: We provide comparative results with Koopman-based deterministic and probabilistic models in Appendix H.3.
Revised: We provide comparative results with Koopman-based deterministic and probabilistic models in Appendix H.3.

Original: Data-driven discovery of nonlinear dynamics:
Revised: Data-driven discovery of nonlinear dynamics:

Original: The data-driven discovery of governing dynamics has received tremendous attention in recent years.
Revised: The data-driven discovery of governing dynamics has received tremendous attention in recent years.

Original: Efforts based on the Mori-Zwanzig formalism can accurately identify dynamics of pre-deﬁned variables, which also account for memory effects, but cannot reconstruct the full ﬁne-grained picture or make predictions about other quantities of interest (Chorin & Stinis, 2007; Kondrashov et al., 2015; Ma et al., 2019).
Revised: Efforts based on the Mori-Zwanzig formalism can accurately identify dynamics of pre-deﬁned variables, which also account for memory effects, but cannot reconstruct the full ﬁne-grained picture or make predictions about other quantities of interest (Chorin & Stinis, 2007; Kondrashov et al., 2015; Ma et al., 2019).

Original: Similar restrictions apply when neural-network-based models are employed as e.g. (Chen et al., 2018; Li et al., 2020).
Revised: Similar restrictions apply when neural-network-based models are employed as e.g. (Chen et al., 2018; Li et al., 2020).

Original: Efforts based on the popular SINDy algorithm (Brunton et al., 2016) require additionally data of the timederivatives of the variables of interest which when estimated with ﬁnite-differences introduce errors and reduce robustness.
Revised: Efforts based on the popular SINDy algorithm (Brunton et al., 2016) require additionally data of the timederivatives of the variables of interest which when estimated with ﬁnite-differences introduce errors and reduce robustness.

Original: Sparse Bayesian learning tools in combination with physically-motivated variables and generative models have been employed by (Felsberger & Koutsourelakis, 2019) but cannot guarantee the long-term stability of the learned dynamics as we also show in Appendix H. and in the context of the systems investigated in section 4.
Revised: Sparse Bayesian learning tools in combination with physically-motivated variables and generative models have been employed by (Felsberger & Koutsourelakis, 2019) but cannot guarantee the long-term stability of the learned dynamics as we also show in Appendix H. and in the context of the systems investigated in section 4.

Original: Infusing domain knowledge from physics: Several efforts have been directed in endowing neural networks with invariances or equivariances arising from physical principles.
Revised: Infusing domain knowledge from physics: Several efforts have been directed in endowing neural networks with invariances or equivariances arising from physical principles.

Original: Usually those pertain to translation or rotation invariance and are domain-speciﬁc as in Sch¨utt et al.
Revised: Usually those pertain to translation or rotation invariance and are domain-speciﬁc as in Sch¨utt et al.

Original: More general formulations such as Hamiltonian (Greydanus et al., 2019; Toth et al., 2019) and Lagrangian
Revised: More general formulations such as Hamiltonian (Greydanus et al., 2019; Toth et al., 2019) and Lagrangian

Original: Dynamics (Lutter et al., 2019) are currently restricted in terms of the dimension of the dynamical system.
Revised: Dynamics (Lutter et al., 2019) are currently restricted in terms of the dimension of the dynamical system.

Original: Physical knowledge has been exploited in conjunction with Gaussian Processes in (CampsValls et al., 2018) as well as in the context of PDEs for constructing reduced-order models as in (Grigo & Koutsourelakis, 2019) or for learning modulated derivatives using Graph Neural Networks as in (Seo et al., 2020).
Revised: Physical knowledge has been exploited in conjunction with Gaussian Processes in (CampsValls et al., 2018) as well as in the context of PDEs for constructing reduced-order models as in (Grigo & Koutsourelakis, 2019) or for learning modulated derivatives using Graph Neural Networks as in (Seo et al., 2020).

Original: Another approach involves using physical laws as regularization terms or for augmenting the loss function as in (Raissi et al., 2019;
Revised: Another approach involves using physical laws as regularization terms or for augmenting the loss function as in (Raissi et al., 2019;

Original: Lusch et al., 2018; Zhu et al., 2019;
Revised: Lusch et al., 2018; Zhu et al., 2019;

Original: Kaltenbach & Koutsourelakis, 2020).
Revised: Kaltenbach & Koutsourelakis, 2020).

Original: In the context of molecular dynamics multiple schemes for coarse-graining which also guarantee long-term stability have been proposed by No´e (2018) and Wu et al.
Revised: In the context of molecular dynamics multiple schemes for coarse-graining which also guarantee long-term stability have been proposed by No´e (2018) and Wu et al.

Original: In our formulation, physically-motivated latent variables are used to facilitate generative maps to very high-dimensional data and serve as the requisite information bottleneck in order to reduce the amount of training data needed.
Revised: In our formulation, physically-motivated latent variables are used to facilitate generative maps to very high-dimensional data and serve as the requisite information bottleneck in order to reduce the amount of training data needed.

Original: Slowness and interpretability: Finally, in contrast to general state-space models for analyzing time-series data such as Karl et al. (2016); Rangapuram et al.
Revised: Slowness and interpretability: Finally, in contrast to general state-space models for analyzing time-series data such as Karl et al. (2016); Rangapuram et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2019), the prior proposed on the complex-valued z t enable the discovery of slow features which are crucial in predicting the evolution of multiscale systems and in combination with the variables X t can provide interpretability and insight into the underlying physical processes.
Revised: (2019), the prior proposed on the complex-valued z t enable the discovery of slow features which are crucial in predicting the evolution of multiscale systems and in combination with the variables X t can provide interpretability and insight into the underlying physical processes.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The high-dimensional, ﬁne-grained model considered consists of f identical particles which can move in the bounded one-dimensional domain s ∈ [ − 1 , 1] (under periodic boundary conditions).
Revised: The high-dimensional, ﬁne-grained model considered consists of f identical particles which can move in the bounded one-dimensional domain s ∈ [ − 1 , 1] (under periodic boundary conditions).

Original: The variables x t consist therefore of the coordinates of the particles at each time instant t and the dimension of the system f is equal to the number of particles.
Revised: The variables x t consist therefore of the coordinates of the particles at each time instant t and the dimension of the system f is equal to the number of particles.

Original: We consider two types of stochastic particle dynamics that correspond to an advection-diffusion-type (section 4.1) and a viscousBurgers’-type (section 4.2) behavior.
Revised: We consider two types of stochastic particle dynamics that correspond to an advection-diffusion-type (section 4.1) and a viscous-Burgers’- type (section 4.2) behavior.

Original: In all experiments, the physically-motivated variables X t relate to a discretization of the particle density into d = 25 equally-sized bins.
Revised: In all experiments, the physically-motivated variables X t relate to a discretization of the particle density into d = 25 equally-sized bins for advection-diffusion-type dynamics and into d = 64 equally-sized bins for viscous-Burgers’-type dynamics.

Original: In order to automatically enforce the conservation of mass at each time instant, we make use of the softmax function i.e. the particle density at a bin k is expressed as
Revised: In order to automatically enforce the conservation of mass at each time instant, we make use of the softmax function i.e. the particle density at a bin k is expressed as

Original: [Equation]
Revised: [Equation]

Original: Given this, the probabilistic map ( F in Equation (3)) corresponds to a multinomial density, i.e.:
Revised: Given this, the probabilistic map ( F in Equation (3)) corresponds to a multinomial density, i.e.:

Original: [Equation]
Revised: [Equation]

Original: where m k ( x t ) is the number of particles in bin k .
Revised: where m k ( x t ) is the number of particles in bin k .

Original: The underlying assumption is that, given X t , the coordinates of the particles x t are conditionally independent.
Revised: The underlying assumption is that, given X t , the coordinates of the particles x t are conditionally independent.

Original: This does not imply that they move independently nor that they cannot exhibit coherent behavior (Felsberger & Koutsourelakis, 2019).
Revised: This does not imply that they move independently nor that they cannot exhibit coherent behavior (Felsberger & Koutsourelakis, 2019).

Original: Furthermore, the aforementioned model automatically satisﬁes permutation-invariance which a central feature of the ﬁne-grained dynamics.
Revised: Furthermore, the aforementioned model automatically satisﬁes permutation-invariance which a central feature of the ﬁne-grained dynamics.

Original: This is another advantage of the physically motivated intermediate variables X as enforcing such symmetries/invariances is not a trivial task even when highly expressive models (e.g. neural networks) are used (Rezende et al., 2019).
Revised: This is another advantage of the physically motivated intermediate variables X as enforcing such symmetries/invariances is not a trivial task even when highly expressive models (e.g. neural networks) are used (Rezende et al., 2019).

Original: The practical consequence of Equation (14) is that no parameters θ 2 need to be inferred.
Revised: The practical consequence of Equation (14) is that no parameters θ 2 need to be inferred.

Original: The second map pertains to p ( X t |
Revised: The second map pertains to p ( X t |

Original: z t , θ 1 ) which we represent with a multivariate normal distribution with a mean and a diagonal covariance matrix modeled by a neural network with parameters θ 1 .
Revised: z t , θ 1 ) which we represent with a multivariate normal distribution with a mean and a diagonal covariance matrix modeled by a neural network with parameters θ 1 .

Original: Details of the parameterization can be found in Appendix D.
Revised: Details of the parameterization can be found in Appendix D.

Original: We assess the performance of the method by computing ﬁrst- and second-order statistics as illustrated in the sequel as well as in Appendix F.
Revised: We assess the performance of the method by computing ﬁrst- and second-order statistics as illustrated in the sequel as well as in Appendix F.

Original: We provide comparative results on the same examples in Appendix H where we report on the performance of various alternatives, such as a formulation without the latent variables z t as well as deterministic and probabilistic Koopman-based models.
Revised: We provide comparative results on the same examples in Appendix H where we report on the performance of various alternatives, such as a formulation without the latent variables z t as well as deterministic and probabilistic Koopman-based models.

Original: Moreover a short study on the effect of the amount of training data is included in Appendix G.
Revised: Moreover a short study on the effect of the amount of training data is included in Appendix G.

Original: 4. P ARTICLE D YNAMICS : A DVECTION -D IFFUSION
Revised: 4. P ARTICLE D YNAMICS : A DVECTION -D IFFUSION

Original: We train the model on N = 64 time-series of the positions of f = 250 × 10 3 particles over
Revised: We train the model on N = 64 time-series of the positions of f = 250 × 10 3 particles over

Original: T = 40 time-steps which were simulated as described in Appendix D.1.
Revised: T = 40 time-steps which were simulated as described in Appendix D.1.

Original: Furthermore, we made use of h = 5 complex, latent processes z t,j .
Revised: Furthermore, we made use of h = 5 complex, latent processes z t,j .

Original: Details regarding the variational posteriors and neural network architectures involved can be found in the Appendix D.1.
Revised: Details regarding the variational posteriors and neural network architectures involved can be found in the Appendix D.1.

Original: In Figure 2, the estimates of the complex-valued parameters λ j are plotted as well as the inferred and predicted time-evolution of 2 associated processes z t,j on the complex plane.
Revised: In Figure 2, the estimates of the complex-valued parameters λ j are plotted as well as the inferred and predicted time-evolution of 2 associated processes z t,j on the complex plane.

Original: We note the clear separation of time-scales in the ﬁrst plot with two slow processes, one intermediate and two fast ones.
Revised: We note the clear separation of time-scales in the ﬁrst plot with two slow processes, one intermediate and two fast ones.

Original: This is also evident in the indicative trajectories on the complex plane.
Revised: This is also evident in the indicative trajectories on the complex plane.

Original: A detailed discussion of the map G learned (Equation (2)) between z t and X t can be found in the Appendix E.
Revised: A detailed discussion of the map G learned (Equation (2)) between z t and X t can be found in the Appendix E.

Original: [Figure]
Revised: [Figure]

Original: In Figure 3 we compare the true particle density with the one predicted by the trained reduced model.
Revised: In Figure 3 we compare the true particle density with the one predicted by the trained reduced model.

Original: We note that the latter is computed by reconstructing the x t futures.
Revised: We note that the latter is computed by reconstructing the x t futures.

Original: We observe that the model is able to accurately track ﬁrst-order statistics well into the future.
Revised: We observe that the model is able to accurately track ﬁrst-order statistics well into the future.

Original: [Figure]
Revised: [Figure]

Revised: The horizontal axis corresponds to time-steps and the vertical to the one-dimensional spatial domain of the problem s ∈ [ − 1 , 1] .

Original: A more detailed view of the predictive estimates with snapshots of the particle density at selected time instances is presented in Figure 4.
Revised: A more detailed view of the predictive estimates with snapshots of the particle density at selected time instances is presented in Figure 4.

Original: Here, not only the posterior mean but also the associated uncertainty is displayed.
Revised: Here, not only the posterior mean but also the associated uncertainty is displayed.

Original: We want to emphasize the last Figure at t = 1000 when the steady state has been reached which clearly shows that our model is capable of converging to stable equilibrium
Revised: We want to emphasize the last Figure at t = 1000 when the steady state has been reached which clearly shows that our model is capable of converging to stable equilibrium.

Original: [Figure]
Revised: [Figure]

Original: Since the proposed model is capable of probabilistically reconstructing the whole ﬁne-grained picture, i.e. x t , predictions with regards to any observable can be obtained.
Revised: Since the proposed model is capable of probabilistically reconstructing the whole ﬁne-grained picture, i.e. x t , predictions with regards to any observable can be obtained.

Original: In Appendix F.1 we assess the accuracy of predictions in terms of second-order statistics, and in particular for the probability of ﬁnding simultaneously a pair of particles at two speciﬁed positions.
Revised: In Appendix F.1 we assess the accuracy of predictions in terms of second-order statistics, and in particular for the probability of ﬁnding simultaneously a pair of particles at two speciﬁed positions.

Original: Finally, we demonstrate the accuracy of the trained in model in producing predictions under unseen initial conditions as described in section 2.3.
Revised: Finally, we demonstrate the accuracy of the trained in model in producing predictions under unseen initial conditions as described in section 2.3.

Original: Figure 5 depicts a new initial condition in terms of the particle density according to which particle positions x 0 were drawn.
Revised: Figure 5 depicts a new initial condition in terms of the particle density according to which particle positions x 0 were drawn.

Original: The posterior on the corresponding z 

Revised: The posterior on the corresponding z

Original: (see section 2.3) can be used to reconstruct the density as shown also on the same
Revised: (see section 2.3) can be used to reconstruct the density as shown also on the same

Original: Figure.
Revised: Figure.

Original: [Figure]
Revised: [Figure]

Original: Figure 6 shows predictions of the particle density (i.e. ﬁrst-order statistics) at various timesteps.
Revised: Figure 6 shows predictions of the particle density (i.e. ﬁrst-order statistics) at various timesteps.

Original: We want to emphasise the frame on the right, for t = 500 which shows the steady state of the system.
Revised: We want to emphasise the frame on the right, for t = 500 which shows the steady state of the system.

Original: We note that even though the initial condition was not contained in the training data, our framework is able to correctly track the system’s evolution and to predict the correct steady state.
Revised: We note that even though the initial condition was not contained in the training data, our framework is able to correctly track the system’s evolution and to predict the correct steady state.

Original: [Figure]
Revised: [Figure]

Original: 4. P ARTICLE D YNAMICS : V ISCOUS B URGERS ’ E QUATION
Revised: 4. P ARTICLE D YNAMICS : V ISCOUS B URGERS ’ E QUATION

Original: In this example, we made use of N = 64 sequences of f = 500 × 10 3 particles over T = 40 timesteps.
Revised: In this example [Figure] we made use of N = 64 sequences of f = 500 × 10 3 particles over T = 40 timesteps.

Original: Details regarding the physical simulator, the stochastic interactions between particles as well as the associated network architectures are contained in Appendix D.2.
Revised: Details regarding the physical simulator, the stochastic interactions between particles as well as the associated network architectures are contained in Appendix D.2.

Original: As in the previous example, we employed the particle density with the softmax transformation for X t and h = 5 complex-valued processes z t at the lowest model level.
Revised: As in the previous example, we employed the particle density with the softmax transformation for X t and h = 5 complex-valued processes z t at the lowest model level.

Original: In Figure 7 the estimated values for λ j are shown where the clear separation of time-scales with three slow and two fast processes can be observed.
Revised: In Figure 7 the estimated values for λ j are shown where the clear separation of time-scales with three slow and two fast processes can be observed.

Original: [Figure]
Revised: [Figure]

Original: In Figure 8 we compare the evolution of the true particle density with the (posterior mean of) the model-predicted one.
Revised: In Figure 8 we compare the evolution of the true particle density with the (posterior mean of) the model-predicted one.

Original: We point out the sharp front at the lower left corner which is characteristic of the Burgers’ equation and which eventually dissipates due to the viscosity.
Revised: We point out the sharp front at the lower left corner which is characteristic of the Burgers’ equation and which eventually dissipates due to the viscosity.

Original: This is captured in the inferred as well as in the predicted solution.
Revised: This is captured in the inferred as well as in the predicted solution.

Original:  A more detailed view on the predictive results with snapshots of the particle density at selected time instances is presented in Figure 9.
Revised: [Figure] A more detailed view on the predictive results with snapshots of the particle density at selected time instances is presented in Figure 9.

Original: We emphasize again the stable convergence of the learned 
Revised: We emphasize again the stable convergence of the learned dynamics to the steady state as well as the accuracy in capturing, propagating and dissipating the shock front

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: ( s , t ) Posterior Mean Reference+/- 1 Standard Deviation 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.012 0. ( s , t ) Posterior Mean Reference+/- 1 Standard Deviation

Original: We compare the accuracy of the predictions for second-order statistics of the ﬁne-grained system in terms of the two-particle probability in Appendix F.2 where excellent agreement with the ground truth, i.e. the one computed by simulating the ﬁne-grained system, is observed.
Revised: We compare the accuracy of the predictions for second-order statistics of the ﬁne-grained system in terms of the two-particle probability in Appendix F.2 where excellent agreement with the ground truth, i.e. the one computed by simulating the ﬁne-grained system, is observed.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We presented a framework for efﬁciently learning a lower-dimensional, dynamical representation of a high-dimensional, ﬁne-grained system that is predictive of its long-term evolution and whose stability is guaranteed.
Revised: We presented a framework for efﬁciently learning a lower-dimensional, dynamical representation of a high-dimensional, ﬁne-grained system that is predictive of its long-term evolution and whose stability is guaranteed.

Original: We infuse domain knowledge with the help of an additional layer of latent variables.
Revised: We infuse domain knowledge with the help of an additional layer of latent variables.

Original: The latent variables at the lowest level provide an interpretabable separation of the timescales and ensure the long-term stability of the learned dynamics.
Revised: The latent variables at the lowest level provide an interpretabable separation of the timescales and ensure the long-term stability of the learned dynamics.

Original: We employed scalable variational inference techniques and applied the proposed model in data generated from very large systems of interacting particles.
Revised: We employed scalable variational inference techniques and applied the proposed model in data generated from very large systems of interacting particles.

Original: In all cases accurate probabilistic predictions were obtained both in terms of ﬁrst- and second-order statistics over a very long time range into the future.
Revised: In all cases accurate probabilistic predictions were obtained both in terms of ﬁrst- and second-order statistics over a very long time range into the future.

Original: More importantly, the ability of the trained model to produce predictions under new, unseen initial conditions was demonstrated.
Revised: More importantly, the ability of the trained model to produce predictions under new, unseen initial conditions was demonstrated.

Original: An obvious limitation for the applicability of the proposed method to general dynamical systems pertains to the physically motivated variables X .
Revised: An obvious limitation for the applicability of the proposed method to general dynamical systems pertains to the physically motivated variables X .

Original: While such variables are available for several classes of physical systems, they need to be re-deﬁned when moving to new problems and expert elicitation might be necessary.
Revised: While such variables are available for several classes of physical systems, they need to be re-deﬁned when moving to new problems and expert elicitation might be necessary.

Original: Furthermore, if an incomplete list of such variables is available from physical insight, this would need to be complemented by additional variables discovered from data.
Revised: Furthermore, if an incomplete list of such variables is available from physical insight, this would need to be complemented by additional variables discovered from data.

Original: To that end, one could envision that some of the abstract latent variables z t or functions thereof, e.g. f
Revised: To that end, one could envision that some of the abstract latent variables z t or functions thereof, e.g. f

Original: ( z t ) , could be employed in a generative map of the form x t = F ( X t , f ( z t )) instead of Equation (3).
Revised: ( z t ) , could be employed in a generative map of the form x t = F ( X t , f ( z t )) instead of Equation (3).

Original: A ﬁnal deﬁciency of the proposed model is the lack of an automated procedure for determining the appropriate number of z .
Revised: A ﬁnal deﬁciency of the proposed model is the lack of an automated procedure for determining the appropriate number of z .

Original: We believe that the ELBO, which provides a lower bound on the model evidence, could be used for this purpose.
Revised: We believe that the ELBO, which provides a lower bound on the model evidence, could be used for this purpose.

Original: A C OMPLEX N ORMAL DISTRIBUTION
Revised: A C OMPLEX N ORMAL DISTRIBUTION

Original: In this Appendix, the complex random normal distribution is reviewed.
Revised: In this Appendix, the complex random normal distribution is reviewed.

Original: The mathematical deﬁnitions introduced follow Andersen et al.
Revised: The mathematical deﬁnitions introduced follow Andersen et al.

Original: A p -variate complex normal random variable Y ∈ C p with Y ∼ CN ( µ C , Σ C ) is deﬁned by a complex mean vector µ C ∈ C p and a complex Covariance Matrix Σ C ∈ C p × p + .
Revised: A p -variate complex normal random variable Y ∈ C p with Y ∼ CN ( µ C , Σ C ) is deﬁned by a complex mean vector µ C ∈ C p and a complex Covariance Matrix Σ C ∈ C p × p + .

Original: The density with respect to Lesbegue measures on C p can be stated as:
Revised: The density with respect to Lesbegue measures on C p can be stated as:

Original: [Equation]
Revised: [Equation]

Original:  where ∗ indicates the conjugate transpose of a matrix.
Revised: ) where ∗ indicates the conjugate transpose of a matrix.

Original: This complex normal random variable has similar properties to the well-known, real-valued counterpart.
Revised: This complex normal random variable has similar properties to the well-known, real-valued counterpart.

Original: For instance, linear transformations of complex random normal variables are again complex random normal variables.
Revised: For instance, linear transformations of complex random normal variables are again complex random normal variables.

Original: These properties directly follow from the fact, that for a complex random normal variable there exists an isomorphic transformation to a real valued 2 p -variate normal random variable W ∈ R 2 p .
Revised: These properties directly follow from the fact, that for a complex random normal variable there exists an isomorphic transformation to a real valued 2 p -variate normal random variable W ∈ R 2 p .

Original: This random normal variable is deﬁned with mean
Revised: This random normal variable is deﬁned with mean

Original: [Equation]
Revised: [Equation]

Original: and covariance
Revised: and covariance

Original: [Equation]
Revised: [Equation]

Original: Therefore: W ∼ N ( µ R , Σ R )
Revised: Therefore: W ∼ N ( µ R , Σ R )

Original: As an example the real valued isomorphic counterpart of the standard complex random normal distribution CN (0 , 1) is the bivariate normal distribution N ( 0 , 12 I ) .
Revised: As an example the real valued isomorphic counterpart of the standard complex random normal distribution CN (0 , 1) is the bivariate normal distribution N ( 0 , 12 I ) .

Original: B C HOICE OF VARIANCE FOR A - PRIORI STEADY STATE
Revised: B C HOICE OF VARIANCE FOR A - PRIORI STEADY STATE

Original: We derive transient and stationary properties of the complex-valued latent processes z t,j and justify our choices for the model parameters.
Revised: We derive transient and stationary properties of the complex-valued latent processes z t,j and justify our choices for the model parameters.

Original: Based on Equation (1) and for each process j , the dynamics can be written in terms of the real and imaginary parts in two dimensions as follows:
Revised: Based on Equation (1) and for each process j , the dynamics can be written in terms of the real and imaginary parts in two dimensions as follows:

Original: [Equation]
Revised: [Equation]

Original: or:
Revised: or:

Original: [Equation]
Revised: [Equation]

Original: where (cid:60) ( (cid:15) t,j ) , (cid:61) ( (cid:15) t,j ) ∼ N (0 , 1 / 2) .
Revised: where (cid:60) ( (cid:15) t,j ) , (cid:61) ( (cid:15) t,j ) ∼ N (0 , 1 / 2) .

Original: The matrix A j is given by (see also Equations (6), (7)):
Revised: The matrix A j is given by (see also Equations (6)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  and can be diagonalized as A j = V P j V ∗ where:
Revised: 20) and can be diagonalized as A j = V P j V ∗ where:

Original: [Equation]
Revised: [Equation]

Original: V ∗ is the conjugate transpose of V and ¯ λ
Revised: V ∗ is the conjugate transpose of V and ¯ λ

Original: j denotes the complex conjugate of λ j .
Revised: j denotes the complex conjugate of λ j .

Original: Due to the linearity of the model and the Gaussian initial conditions, i.e. (cid:60) ( z 0 ,j ) , (cid:61) ( z t,j ) ∼ N (0 , σ 20 ,j / 2) , the marginal will remain Gaussian at all times t .
Revised: Due to the linearity of the model and the Gaussian initial conditions, i.e. (cid:60) ( z 0 ,j ) , (cid:61) ( z 0 ,j ) ∼ N (0 , σ 20 ,j / 2) , the marginal will remain Gaussian at all times t .

Original: A direct consequence of the above is that the mean of real and imaginary parts is always zero, i.e.:
Revised: A direct consequence of the above is that the mean of real and imaginary parts is always zero, i.e.:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, the covariance C ( j ) t is given by:
Revised: Furthermore, the covariance C ( j ) t is given by:

Original: [Equation]
Revised: [Equation]

Original: which upon use of the diagonalized form of A j yields:
Revised: which upon use of the diagonalized form of A j yields:

Original: [Equation]
Revised: [Equation]

Original: λj ) 1 −e
Revised: 1 −

Original: [Equation]
Revised: [Equation]

Original: λj
Revised: λj

Original: [Equation]
Revised: [Equation]

Original: 1 −
Revised: 1 −

Original: [Equation]
Revised: [Equation]

Original: 1 −
Revised: 1 −

Original: [Equation]
Revised: [Equation]

Original: As mentioned earlier, a necessary condition for the long-term stability of the processes is that (cid:60) ( λ j ) < 0 , in which case and as t → ∞ it leads to:
Revised: As mentioned earlier, a necessary condition for the long-term stability of the processes is that (cid:60) ( λ j ) < 0 , in which case and as t → ∞ it leads to:

Original: [Equation]
Revised: [Equation]

Original: This implies that real and imaginary parts are asymptotically uncorrelated with a variance σ 2 j
Revised: This implies that real and imaginary parts are asymptotically uncorrelated with a variance σ 2 j

Original: [Equation]
Revised: [Equation]

Original: By setting σ 2 j = 1 − e 2 (cid:60) ( λ j ) we enable direct comparisons between z t,j solely on the basis of Re ( λ j ) i.e. the degree of slowness (Turner & Sahani, 2007; Zafeiriou et al., 2015)).
Revised: By setting σ 2 j = 1 − e 2 (cid:60) ( λ j ) we enable direct comparisons between z t,j solely on the basis of Re ( λ j ) i.e. the degree of slowness (Turner & Sahani, 2007; Zafeiriou et al., 2015)).

Original: In this case the asymptotic variance of real and imaginary parts becomes 1 / 2 .
Revised: In this case the asymptotic variance of real and imaginary parts becomes 1 / 2 .

Original: Finally by setting σ 20 ,j = 1 we ensure that, a priori, the processes z t,j are stationary , with C ( j ) t =
Revised: Finally by setting σ 20 ,j = 1 we ensure that, a priori, the processes z t,j are stationary , with C ( j ) t =

Original: C ( j ) ∞
Revised: C ( j ) ∞

Original: = 12 I , i.e. no a-priori bias is introduced with regards to their transient characteristics.
Revised: = 12 I , i.e. no a-priori bias is introduced with regards to their transient characteristics.

Original: We ﬁnally note that the autocovariance D ( j ) τ of each of these stationary processes j is given by:
Revised: We ﬁnally note that the autocovariance D ( j ) τ of each of these stationary processes j is given by:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where A j and the covariance C ( j ) t are given above.
Revised: where A j and the covariance C ( j ) t are given above.

Original: By exploiting the diagonalization of A j and that C ( j ) t = C ( j ) ∞ =
Revised: By exploiting the diagonalization of A j and that C ( j ) t = C ( j ) ∞ =

Original: 12 I (for the parameter values discussed earlier), we obtain that:
Revised: 12 I (for the parameter values discussed earlier), we obtain that:

Original: [Equation]
Revised: [Equation]

Original: One can clearly observe harmonic (cross-)correlation terms which depend on the imaginary part of the λ j and can capture persistent periodic effects of the dynamical system in the long-time range.
Revised: One can clearly observe harmonic (cross-)correlation terms which depend on the imaginary part of the λ j and can capture persistent periodic effects of the dynamical system in the long-time range.

Original: C D ERIVATION OF THE ELBO
Revised: C D ERIVATION OF THE ELBO

Original: This section contains details of the derivation of the Evidence-Lower-Bound (ELBO) which serves as the objective function for the determination of the parameters φ and θ during training.
Revised: This section contains details of the derivation of the Evidence-Lower-Bound (ELBO) which serves as the objective function for the determination of the parameters φ and θ during training.

Original: In particular:
Revised: In particular:

Original: [Equation]
Revised: [Equation]

Original: D D ETAILS FOR EXPERIMENTS
Revised: D D ETAILS FOR EXPERIMENTS

Original: This appendix contains details for our experiments involving moving particles that have stochastic interactions corresponding to either an Advection-Diffusion behaviour or viscous Burgers’ type behaviour.
Revised: This appendix contains details for our experiments involving moving particles that have stochastic interactions corresponding to either an Advection-Diffusion behaviour or viscous Burgers’ type behaviour.

Original: D. P ARTICLE D YNAMICS : A DVECTION -D IFFUSION
Revised: D. P ARTICLE D YNAMICS : A DVECTION -D IFFUSION

Original: For the simulations presented f = 250 × 10 3 particles were used, which, at each microscopic time step δt = 2 .
Revised: For the simulations presented f = 250 × 10 3 particles were used, which, at each microscopic time step δt = 2 .

Original: 5 × 10 − 3 performed random, non-interacting, jumps of size δs = 1640 , either to the left with probability p left = 0 .
Revised: 5 × 10 − 3 performed random, non-interacting, jumps of size δs = 1640 , either to the left with probability p left = 0 .

Original: 1875 or to the right with probability p right = 0 .
Revised: 1875 or to the right with probability p right = 0 .

Original: The positions were restricted in [ − 1 , 1] with periodic boundary conditions.
Revised: The positions were restricted in [ − 1 , 1] with periodic boundary conditions.

Original: It is well-known (Cottet & Koumoutsakos, 2000)
Revised: It is well-known (Cottet & Koumoutsakos, 2000)

Original: that in the limit (i.e. f → ∞ ) the particle density ρ ( s, t ) can be modeled with an advectiondiffusion PDE with diffusion constant D
Revised: that in the limit (i.e. f → ∞ ) the particle density ρ ( s, t ) can be modeled with an advectiondiffusion PDE with diffusion constant D

Original: = ( p left + p right ) δs 2 2 δt and velocity v = ( p right − p left ) δsδt :
Revised: = ( p left + p right ) δs 2 2 δt and velocity v = ( p right − p left ) δsδt :

Original: [Equation]
Revised: [Equation]

Original: From this simulation every 800th microscopic time step the particle positions were extracted and used as training data for our system.
Revised: From this simulation every 800th microscopic time step the particle positions were extracted and used as training data for our system.

Original: Sample initial conditions are shown in Figure 10:
Revised: Sample initial conditions are shown in Figure 10:

Original: [Figure]
Revised: [Figure]

Original: The architecture of the neural networks for the generative mappings described above as well as for the variational posteriors introduced in Section 2.2 can be seen in Figure 11.
Revised: The architecture of the neural networks for the generative mappings described above as well as for the variational posteriors introduced in Section 2.2 can be seen in Figure 11.

Original: The neural network used for the generative mapping between the low-dimensional states z t and the mean and covariance for X t consists of only one dense layer, whereas the variational posterior on z 0: T is parameterized using a dense Layer with ReLu activation followed by another dense layer.
Revised: The neural network used for the generative mapping between the low-dimensional states z t and the mean and covariance for X t consists of only one dense layer, whereas the variational posterior on z 0: T is parameterized using a dense Layer with ReLu activation followed by another dense layer.

Original: dynamics corresponding to an advectiondiffusion equation.
Revised: corresponding to an advectiondiffusion equation.

Original: D. P ARTICLE D YNAMICS : V ISCOUS B URGERS ’ EQUATION
Revised: D. P ARTICLE D YNAMICS : V ISCOUS B URGERS ’ EQUATION

Original: The second test-case involved a ﬁne-grained system of f = 500 × 10 3 particles which perform interactive random walks i.e. the jump performed at each ﬁne-scale time-step δt = 2 .
Revised: The second test-case involved a ﬁne-grained system of f = 500 × 10 3 particles which perform interactive random walks i.e. the jump performed at each ﬁne-scale time-step δt = 2 .

Original: 5 × 10 −depends on the positions of the other walkers.
Revised: 5 × 10 −depends on the positions of the other walkers.

Original: In particular we adopted interactions as described in
Revised: In particular we adopted interactions as described in

Original: Roberts (1989); Chertock & Levy (2001); Li et al.
Revised: Roberts (1989); Chertock & Levy (2001); Li et al.

Original: so as, in the limit (i.e. when f → ∞ , δt → 0 , δs → 0 ), the particle density ρ ( s, t ) follows a viscous Burgers’ equation with ν = 0 . 0005 :
Revised: so as, in the limit (i.e. when f → ∞ , δt → 0 , δs → 0 ), the particle density ρ ( s, t ) follows a viscous Burgers’ equation with ν = 0 . 0005 :

Original: [Equation]
Revised: [Equation]

Original: From this simulation every 800th microscopic time step the particle positions were extracted and used as training data for our system.
Revised: From this simulation every 800th microscopic time step the particle positions were extracted and used as training data for our system.

Original: Sample initial conditions are shown in Figure 12.
Revised: Sample initial conditions are shown in Figure 12.

Original: [Figure]
Revised: [Figure]

Original: The architecture of the neural networks for the generative mappings described above as well as for the variational posteriors introduced in Section 2.2 can be seen in Figure 13.
Revised: The architecture of the neural networks for the generative mappings described above as well as for the variational posteriors introduced in Section 2.2 can be seen in Figure 13.

Original: The neural network used for the generative mapping between the low-dimensional states z t and the mean and covariance for X t consists of several dense layers with ReLu activation and Dropout layers to avoid overﬁtting, whereas the variational posterior on z 0: T is parameterized using two dense layers with ReLu activation followed by another dense layer.
Revised: The neural network used for the generative mapping between the low-dimensional states z t and the mean and covariance for X t consists of several dense layers with ReLu activation and Dropout layers to avoid overﬁtting, whereas the variational posterior on z 0: T is parameterized using two dense layers with ReLu activation followed by another dense layer.

Original: E D ETAILED ANALYSIS OF THE GENERATIVE MAPPING AND THE SLOW LATENT VARIABLES
Revised: E D ETAILED ANALYSIS OF THE GENERATIVE MAPPING AND THE SLOW LATENT VARIABLES

Original: In this Appendix we take a closer look at the generative mapping and the (slow) latent variables z learned .
Revised: In this Appendix we take a closer look at the generative mapping and the (slow) latent variables z learned .

Original: For the Advection-Diffusion example, we discovered two slow processes (Section 4.1), zand the marginally faster process z 2 .
Revised: For the Advection-Diffusion example, we discovered two slow processes (Section 4.1), zand the marginally faster process z 2 .

Original: The rest of the processes were very fast in comparison and took values close to the zero point of the complex plane during the inference as well as during the prediction phase.
Revised: The rest of the processes were very fast in comparison and took values close to the zero point of the complex plane during the inference as well as during the prediction phase.

Original: In order to visualize the inﬂuence through the generative mapping of these two slow processes, we set the value of all other processes to zero and then reconstructed the ﬁne-grained state based on different absolute values of z 1 and z 2 .
Revised: In order to visualize the inﬂuence through the generative mapping of these two slow processes, we set the value of all other processes to zero and then reconstructed the ﬁne-grained state based on different absolute values of z 1 and z 2 .

Original: The result are shown in Figure 14 in terms of the reconstructed particle density.
Revised: The result are shown in Figure 14 in terms of the reconstructed particle density.

Original: It is clearly visible that those two processes are responsible for a variety of density proﬁles.
Revised: It is clearly visible that those two processes are responsible for a variety of density proﬁles.

Original: In accordance with their slowness, z 1 (the slightly slower process) is responsible for the most striking changes, whereas the other slow process generates some smaller scale ﬂuctuations.
Revised: In accordance with their slowness, z 1 (the slightly slower process) is responsible for the most striking changes, whereas the other slow process generates some smaller scale ﬂuctuations.

Original: [Equation]
Revised: [Equation]

Original: [Equation] 
Revised: [Equation] [Equation] [Equation] [Equation]

Original: 055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0.
Revised: [Equation] [Equation] [Equation] [Equation]

Revised: [Equation]

Revised: [Equation]

Original: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0.
Revised: [Equation] [Equation] [Equation] [Equation]

Original: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0.
Revised: [Equation] [Equation] [Equation]

Original: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0.
Revised: [Equation] 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0.

Original: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0.
Revised: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0.

Original: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0.
Revised: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0.

Original: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 
Revised: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0.

Original: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 
Revised: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0.

Original: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 
Revised: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0.

Original: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.{+0.
+}
Revised: 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 s 0.025 0.030 0. 0.040 0.045 0.050 0.055 0.060 ( s , t ) 1.00 0.75 0.50 0.25 0.00 {+0.
+}

Original: [Figure]
Revised: [Figure]

Original: F T WO - POINT P ROBABILITY
Revised: F T WO - POINT P ROBABILITY

Original: This appendix contains the predictive estimates for the two-point probability, i.e. the probability of ﬁnding two particles simultaneously in two bins ( b 1 , b 2 ) .
Revised: This appendix contains the predictive estimates for the two-point probability, i.e. the probability of ﬁnding two particles simultaneously in two bins ( b 1 , b 2 ) .

Original: This two-point probability can be computed based on the reconstructed ﬁne-grained system and corresponds to a second order statistic.
Revised: This two-point probability can be computed based on the reconstructed ﬁne-grained system and corresponds to a second order statistic.

Original: F. P ARTICLE D YNAMICS : A DVECTION -D IFFUSION
Revised: F. P ARTICLE D YNAMICS : A DVECTION -D IFFUSION

Original: The estimated two-point probability as well as the comparison to test data is shown for two indicative time-instants in Figure 15 and 16.
Revised: The estimated two-point probability as well as the comparison to test data is shown for two indicative time-instants in Figure 15 and 16.

Original: We note a very good agreement with the ground truth.
Revised: We note a very good agreement with the ground truth.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: F. P ARTICLE D YNAMICS : V ISCOUS B URGERS ’ EQUATION
Revised: F. P ARTICLE D YNAMICS : V ISCOUS B URGERS ’ EQUATION

Original: The estimated two-point probability as well as the comparison to test data is shown for two indicative time-instants in Figure 17 and 18.
Revised: The estimated two-point probability as well as the comparison to test data is shown for two indicative time-instants in Figure 17 and 18.

Original: We note a very good agreement with the ground truth.
Revised: We note a very good agreement with the ground truth.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: G E FFECT OF THE AMOUNT OF TRAINING DATA
Revised: G E FFECT OF THE AMOUNT OF TRAINING DATA

Original: This section contains a study on the inﬂuence of the amount of training data.
Revised: This section contains a study on the inﬂuence of the amount of training data.

Original: We illustrate this in the context of the Advection-Diffusion example (section 4.1) by using 16 time sequences instead of the 64 employed earlier.
Revised: We illustrate this in the context of the Advection-Diffusion example (section 4.1) by using 16 time sequences instead of the 64 employed earlier.

Original: In the ﬁgures below we note that the proposed model is capable capturing the main features of the system’s dynamics as well as the correct steady state even with fewer training data.
Revised: In the ﬁgures below we note that the proposed model is capable capturing the main features of the system’s dynamics as well as the correct steady state even with fewer training data.

Original: We believe that this is due to the intermediate layer of physically-motivated variables X t which introduces an information bottleneck.
Revised: We believe that this is due to the intermediate layer of physically-motivated variables X t which introduces an information bottleneck.

Original: Finally, and as one would expect, fewer training data leads to increased predictive uncertainty
Revised: Finally, and as one would expect, fewer training data leads to increased predictive uncertainty

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: H C OMPARISON WITH OTHER APPROACHES
Revised: H C OMPARISON WITH OTHER APPROACHES

Original: This appendix contains results obtained by other methods for the test cases discussed in the main text.
Revised: This appendix contains results obtained by other methods for the test cases discussed in the main text.

Original: The simulation data is identical to the one used for the proposed method and details of the speciﬁc algorithms are described in the following.
Revised: The simulation data is identical to the one used for the proposed method and details of the speciﬁc algorithms are described in the following.

Original: H. R EAL - VALUED LATENT SPACE
Revised: H. R EAL - VALUED LATENT SPACE

Original: To demonstrate the utility of a complex-valued latent space
Revised: To demonstrate the utility of a complex-valued latent space

Original: [Figure]
Revised: [Figure]

Original: the two examples were also solved with a real-valued z t,j .
Revised: the two examples were also solved with a real-valued z t,j .

Original: The only difference here is the restriction of the latent variables z t,j and the λ j to real values.
Revised: The only difference here is the restriction of the latent variables z t,j and the λ j to real values.

Original: A model with real-valued latent space is also capable of ensuring the stability but it is not capable of capturing periodic components of the dynamics.
Revised: A model with real-valued latent space is also capable of ensuring the stability but it is not capable of capturing periodic components of the dynamics.

Original: As most physical systems (including the two examples) contain such components, the algorithm is not able to accurately model the dynamics and fails in generating reliable predictions.
Revised: As most physical systems (including the two examples) contain such components, the algorithm is not able to accurately model the dynamics and fails in generating reliable predictions.

Original: This can be readily observed in the extrapolative predictions of Figures 21 and 22 where, apart for the biased results, one can also note increased predictive uncertainty.
Revised: This can be readily observed in the extrapolative predictions of Figures 21 and 22 where, apart for the biased results, one can also note increased predictive uncertainty.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: H. N O STABLE LATENT SPACE
Revised: H. N O STABLE LATENT SPACE

Original: Another possibility is to employ only the physically motivated latent variables X t and remove completely the latent variables z t in the ﬁrst layer.
Revised: Another possibility is to employ only the physically motivated latent variables X t and remove completely the latent variables z t in the ﬁrst layer.

Original: This approach is similar to the one investigated in
Revised: This approach is similar to the one investigated in

Original: Felsberger & Koutsourelakis (2019) and Kaltenbach & Koutsourelakis (2020) as well as to the idea of neural ODEs (Chen et al., 2018).
Revised: Felsberger & Koutsourelakis (2019) and Kaltenbach & Koutsourelakis (2020) as well as to the idea of neural ODEs (Chen et al., 2018).

Original: In this case, one must learn directly the dynamics of X t and for this purpose we employed a three-layer, fully-connected neural network NN as follows:
Revised: In this case, one must learn directly the dynamics of X t and for this purpose we employed a three-layer, fully-connected neural network NN as follows:

Original: [Equation]
Revised: [Equation]

Original: The learned dynamics are in general non-linear and stability is not guaranteed.
Revised: The learned dynamics are in general non-linear and stability is not guaranteed.

Original: As it can be observed in Figures 23 and 24, the trained model is capable of producing accurate predictions for some timesteps but eventually in both cases predictions become unstable.
Revised: As it can be observed in Figures 23 and 24, the trained model is capable of producing accurate predictions for some timesteps but eventually in both cases predictions become unstable.

Original: This could also be problematic when the trained model is used to make predictions with new initial conditions as the chaotic nature of the nonlinear dynamics can lead to signiﬁcant errors even for shorter time horizons.
Revised: This could also be problematic when the trained model is used to make predictions with new initial conditions as the chaotic nature of the nonlinear dynamics can lead to signiﬁcant errors even for shorter time horizons.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: H. K OOPMAN - BASED MODELS
Revised: H. K OOPMAN - BASED MODELS

Original: The ﬁnal alternative explored involved probabilistic and deterministic Koopman-based models for the latent dynamics.
Revised: The ﬁnal alternative explored involved probabilistic and deterministic Koopman-based models for the latent dynamics.

Original: We kept the generative framework of our model and did not use an encoder as for instance in Gin et al.
Revised: We kept the generative framework of our model and did not use an encoder as for instance in Gin et al.

Original: (2019) in order to remove the effect of the associated model choice.
Revised: (2019) in order to remove the effect of the associated model choice.

Original: For the same reason, we retained the intermediate variables
Revised: For the same reason, we retained the intermediate variables

Original: X t even though these do not appear in any known Koopman-operator implementations.
Revised: X t even though these do not appear in any known Koopman-operator implementations.

Original: We replaced our complex-valued dynamics of the latent processes z t,j with the models described in the sequel.
Revised: We replaced our complex-valued dynamics of the latent processes z t,j with the models described in the sequel.

Original: H.3. P ROBABILISTIC K OOPMAN - BASED MODEL
Revised: H.3. P ROBABILISTIC K OOPMAN - BASED MODEL

Original: We used real-valued latent variables z t which are not a-priori independent and whose dynamics are parameterized with a Koopman matrix K and a diagonal noise matrix W :
Revised: We used real-valued latent variables z t which are not a-priori independent and whose dynamics are parameterized with a Koopman matrix K and a diagonal noise matrix W :

Original: [Equation]
Revised: [Equation]

Original: The learned matrix K is not guaranteed to be stable in the absence of additional constraints but in both cases examined the eigenvalues of the learned K were real smaller than one.
Revised: The learned matrix K is not guaranteed to be stable in the absence of additional constraints but in both cases examined the eigenvalues of the learned K were real smaller than one.

Original: Long-term predictions were stable and for the Burgers’ case in Figure 26 we were also able to reach the true steady state.
Revised: Long-term predictions were stable and for the Burgers’ case in Figure 26 we were also able to reach the true steady state.

Original: For the (simpler) Advection-Diffusion example in Figure 25 an incorrect steady state was reached and the predictive quality started to deteriorate after some time-steps.
Revised: For the (simpler) Advection-Diffusion example in Figure 25 an incorrect steady state was reached and the predictive quality started to deteriorate after some time-steps.

Original: In comparison to our framework, the probabilistic Koopman-based model does not provide a direct separation between slow and fast processes and therefore its interpretability is reduced.
Revised: In comparison to our framework, the probabilistic Koopman-based model does not provide a direct separation between slow and fast processes and therefore its interpretability is reduced.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: H.3. N ON - PROBABILISTIC K OOPMAN L EARNING
Revised: H.3. N ON - PROBABILISTIC K OOPMAN L EARNING

Original: We also used real-valued latent variables z t with deterministic dynamics which were parameterized as follows:
Revised: We also used real-valued latent variables z t with deterministic dynamics which were parameterized as follows:

Original: [Equation]
Revised: [Equation]

Original: The absence of noise in comparison to Equation (32), led in both cases to an estimate for the Koopman matrix K that did not yield stable predictions (each of the learned K matrices had at least one eigenvalue which was larger than 1 ).
Revised: The absence of noise in comparison to Equation (32), led in both cases to an estimate for the Koopman matrix K that did not yield stable predictions (each of the learned K matrices had at least one eigenvalue which was larger than 1 ).

Original: We speculate that the lack of stochasticity made the model less capable of dealing with the information loss.
Revised: We speculate that the lack of stochasticity made the model less capable of dealing with the information loss.

Original: We also note in Figures 27 and 28 that the predictions obtained are, with an exception of a few time-steps, highly inaccurate.
Revised: We also note in Figures 27 and 28 that the predictions obtained are, with an exception of a few time-steps, highly inaccurate.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]