Original: Learning Dissipative Dynamics in Chaotic Systems
Revised: Learning Dissipative Dynamics in Chaotic Systems

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Chaotic systems are notoriously challenging to predict because of their sensitivity toperturbations and errors due to time stepping.
Revised: Chaotic systems are notoriously challenging to predict because of their sensitivity toperturbations and errors due to time stepping.

Original: Despite this unpredictable behavior,for many dissipative systems the statistics of the long term trajectories are governedby an invariant measure supported on a set, known as the global attractor; for manyproblems this set is finite dimensional, even if the state space is infinite dimensional.
Revised: Despite this unpredictable behavior,for many dissipative systems the statistics of the long term trajectories are governedby an invariant measure supported on a set, known as the global attractor; for manyproblems this set is finite dimensional, even if the state space is infinite dimensional.

Original: For Markovian systems, the statistical properties of long-term trajectories areuniquely determined by the solution operator that maps the evolution of the systemover arbitrary positive time increments.
Revised: For Markovian systems, the statistical properties of long-term trajectories areuniquely determined by the solution operator that maps the evolution of the systemover arbitrary positive time increments.

Original: In this work, we propose a machinelearning framework to learn the underlying solution operator for dissipative chaoticsystems, showing that the resulting learned operator accurately captures short-timetrajectories and long-time statistical behavior.
Revised: In this work, we propose a machinelearning framework to learn the underlying solution operator for dissipative chaoticsystems, showing that the resulting learned operator accurately captures short-timetrajectories and long-time statistical behavior.

Original: Using this framework, we are ableto predict various statistics of the invariant measure for the turbulent Kolmogorov
Revised: Using this framework, we are ableto predict various statistics of the invariant measure for the turbulent Kolmogorov

Original: Flow dynamics with Reynolds numbers up to
Revised: Flow dynamics with Reynolds numbers up to

Original:  Introduction
Revised:  Introduction

Original: Machine learning methods for chaotic systems.
Revised: Machine learning methods for chaotic systems.

Original: Chaotic systems are characterized by stronginstabilities.
Revised: Chaotic systems are characterized by stronginstabilities.

Original: Small perturbations to the system, such as the initialization, lead to errors whichaccumulate during time-evolution, due to positive Lyapunov exponents.
Revised: Small perturbations to the system, such as the initialization, lead to errors whichaccumulate during time-evolution, due to positive Lyapunov exponents.

Original: Such instability makeschaotic systems challenging, both for mathematical analysis and numerical simulation.
Revised: Such instability makeschaotic systems challenging, both for mathematical analysis and numerical simulation.

Original: Because ofthe intrinsic instability, it is infeasible for any method to capture the exact trajectory of a chaoticsystem for long periods.
Revised: Because ofthe intrinsic instability, it is infeasible for any method to capture the exact trajectory of a chaoticsystem for long periods.

Original: Therefore, prior works mainly focus on fitting recurrent neural networks(RNN) on extremely short trajectories or learning a step-wise projection from a randomly generatedevolution using reservoir computing (RC)
Revised: Therefore, prior works mainly focus on fitting recurrent neural networks(RNN) on extremely short trajectories or learning a step-wise projection from a randomly generatedevolution using reservoir computing (RC)

Original: Another approach to this problem is to work in thespace of probability densities propagated by, or invariant under, the dynamics [6, 7], the FrobeniusPerron approach; a second, related, approach is to consider the adjoint of this picture and studythe propagation of functionals on the state space of the system, the Koopman operator approach[8, 9].
Revised: Another approach to this problem is to work in thespace of probability densities propagated by, or invariant under, the dynamics [6, 7], the FrobeniusPerron approach; a second, related, approach is to consider the adjoint of this picture and studythe propagation of functionals on the state space of the system, the Koopman operator approach[8, 9].

Original: However, even when the state space is finite dimensional, the Frobenius-Perron and Koopmanapproaches require approximation of a linear operator on a function space; when the state spaceis infinite-dimensional approximation of this operator is particularly challenging.
Revised: However, even when the state space is finite dimensional, the Frobenius-Perron and Koopmanapproaches require approximation of a linear operator on a function space; when the state spaceis infinite-dimensional approximation of this operator is particularly challenging.

Original: These previousattempts are able to push the limits of faithful prediction to moderate periods on low dimensionalordinary differential equations (ODEs), e.g. the Lorenz-63 system, or on one-dimensional partialdifferential equations (PDEs), e.g. the Kuramoto-Sivashinsky (KS) equation.
Revised: These previousattempts are able to push the limits of faithful prediction to moderate periods on low dimensionalordinary differential equations (ODEs), e.g. the Lorenz-63 system, or on one-dimensional partialdifferential equations (PDEs), e.g. the Kuramoto-Sivashinsky (KS) equation.

Original: However, they are lesseffective at modeling more complicated turbulent systems such as the Navier-Stokes equation (NS),especially over long time periods.
Revised: However, they are lesseffective at modeling more complicated turbulent systems such as the Navier-Stokes equation (NS),especially over long time periods.

Original: Indeed, because of positive Lyapunov exponents, we cannot expectpredictions of long-time trajectories of such chaotic systems to be successful.
Revised: Indeed, because of positive Lyapunov exponents, we cannot expectpredictions of long-time trajectories of such chaotic systems to be successful.

Original: Instead, we take a new perspective: we capture statistical properties of long trajectories by accurately learning the solution operator, mapping initial condition to solution at a short time later
Revised: Instead, we take a new perspective: we capture statistical properties of long trajectories by accurately learning the solution operator, mapping initial condition to solution at a short time later

Original: Invariants in chaos.
Revised: Invariants in chaos.

Original: Despite their instability, many chaotic systems exhibit certain reproducible statistical properties, such as the auto-correlation and, for PDEs, the energy spectrum.
Revised: Despite their instability, many chaotic systems exhibit certain reproducible statistical properties, such as the auto-correlation and, for PDEs, the energy spectrum.

Original: Such properties remain the same for different realizations of the initial condition [10].
Revised: Such properties remain the same for different realizations of the initial condition [10].

Original: This is provably the casefor the Lorenz-63 model [11, 12] and empirically holds for many dissipative PDEs, such as the KSequation and the two-dimensional Navier-Stokes equation (Kolmogorov flows) [13].
Revised: This is provably the casefor the Lorenz-63 model [11, 12] and empirically holds for many dissipative PDEs, such as the KSequation and the two-dimensional Navier-Stokes equation (Kolmogorov flows) [13].

Original: Dissipativity isa physical property of many natural systems.
Revised: Dissipativity isa physical property of many natural systems.

Original: Dissipativity may be encoded mathematically as theexistence of a compact set into which all bounded sets of initial conditions evolve in a finite time,and therafter remain inside; the global attractor captures all possible long time behaviour of thesystem by mapping this compact set forward in time [13, 14].
Revised: Dissipativity may be encoded mathematically as theexistence of a compact set into which all bounded sets of initial conditions evolve in a finite time,and therafter remain inside; the global attractor captures all possible long time behaviour of thesystem by mapping this compact set forward in time [13, 14].

Original: There is strong empirical evidence thatmany dissipative systems are ergodic, i.e., there exists an invariant measure which is supported on theglobal attractor.
Revised: There is strong empirical evidence thatmany dissipative systems are ergodic, i.e., there exists an invariant measure which is supported on theglobal attractor.

Original: For autonomous systems, our focus here, the solution operator for the dynamicalsystem maps the initial condition to the solution at any positive time.
Revised: For autonomous systems, our focus here, the solution operator for the dynamicalsystem maps the initial condition to the solution at any positive time.

Original: By fixing an arbitrary positivetime and composing this map with itself many times, predictions can be made far into the future.
Revised: By fixing an arbitrary positivetime and composing this map with itself many times, predictions can be made far into the future.

Original: Thiscompositional property follows from the Markov semigroup property of the solution operator [13, 14].
Revised: This compositional property follows from the Markov semigroup property of the solution operator[13, 14].

Original: It is natural to ask that approximations of the solution operator inherit dissipativity of the true solutionoperator, and that invariant sets and the invariant measure, which characterize the global attractor, arewell-approximated; this problem is well-studied in classical numerical analysis [15, 16] and here weinitiate the study of inheriting dissipativity for machine-learned solution operators.
Revised: It is natural to ask that approximations of the solution operator inherit dissipativity of thetrue solution operator, and that invariant sets and the invariant measure, which characterize the globalattractor, are well-approximated; this problem is well-studied in classical numerical analysis [15, 16]and here we initiate the study of inheriting dissipativity for machine-learned solution operators.

Original: By learning asolution operator, we are able to quickly and accurately generate an approximate attractor and estimateits invariant measure for a variety of chaotic systems that are of interest to the physics and appliedmathematics communities [17–23].
Revised: Bylearning a solution operator, we are able to quickly and accurately generate an approximate attractorand estimate its invariant measure for a variety of chaotic systems that are of interest to the physicsand applied mathematics communities [17–23].

Original: Prior work in learning complex dynamics from data [24–27]has labeled long-term stability a desirable feature, but all lack a principled data-driven modelingparadigm to enforce it.
Revised: Prior work in learning complex dynamics fromdata [24–27] has labeled long-term stability a desirable feature (e.g., [28, 29] learning the attractorsof dynamical systems), but all lack a principled data-driven modeling paradigm to enforce it.

Original: In this paper we work within a precise mathematical setting encompassing awide variety of practical problems [13], and we demonstrate two principled approaches to enforcedissipativity.
Revised: Inthis paper we work within a precise mathematical setting encompassing a wide variety of practicalproblems [13], and we demonstrate two principled approaches to enforce dissipativity.

Original: One of these leads to provably dissipative models, something not previously achieved inthe literature for our definition of dissipativity.
Revised: One of theseleads to provably dissipative models, something not previously achieved in the literature for ourdefinition of dissipativity.

Original: Neural operators.
Revised: Neural operators.

Original: To learn the solution operators for PDEs, one must model the time-evolutionof functions in infinite-dimensional function spaces.
Revised: To learn the solution operators for PDEs, one must model the time-evolutionof functions in infinite-dimensional function spaces.

Original: This is especially challenging when we needto generate long trajectories since even a small error accumulates over multiple compositions ofthe learned operator, potentially causing an exponential blow-up or a collapse due to the space’shigh dimensionality.
Revised: This is especially challenging when we needto generate long trajectories since even a small error accumulates over multiple compositions ofthe learned operator, potentially causing an exponential blow-up or a collapse due to the space’shigh dimensionality.

Original: Because we study the evolution of functions in time, we propose to use neuraloperators [28, 29], a recently developed operator learning method.
Revised: Because we study the evolution of functions in time, we propose to use neuraloperators [30, 31], a recently developed operator learning method.

Original: The neural operator remedies themesh-dependent nature of finite-dimensional operator methods such as RNNs, CNNs, and RC.
Revised: The neural operator remedies themesh-dependent nature of finite-dimensional operator methods such as RNNs, CNNs, and RC.

Original: Neuraloperators are guaranteed to universally approximate any operator in a mesh independent manner [29],and hence, can capture the solution operator of chaotic systems.
Revised: Neuraloperators are guaranteed to universally approximate any operator in a mesh independent manner [31],and hence, can capture the solution operator of chaotic systems.

Original: This approximation guarantee andthe absorption of trajectories by the global attractor makes it possible to accurately follow it over longtime horizons, allowing access to the invariant measure of chaotic systems.
Revised: This approximation guarantee andthe absorption of trajectories by the global attractor makes it possible to accurately follow it over longtime horizons, allowing access to the invariant measure of chaotic systems.

Original: [Figure]
Revised: [Figure]

Original: Our contributions.
Revised: Our contributions.

Original: In this work, we formulate a machine learning framework for chaotic systemsexploiting their dissipativity and Markovian properties.
Revised: In this work, we formulate a machine learning framework for chaotic systemsexploiting their dissipativity and Markovian properties.

Original: We propose the Markov neural operator(MNO) and train it given only one-step evolution data from a chaotic system.
Revised: We propose the Markov neural operator(MNO) and train it given only one-step evolution data from a chaotic system.

Original: By composing thelearned operator over a long horizon, we accurately approximate the global attractor of the system [30].
Revised: By composing thelearned operator over a long horizon, we accurately approximate the global attractor of the system [32].

Original: Our architecture is outlined in Figure 2.
Revised: Our architecture is outlined in Figure 2.

Original: In order to assess its performance, we study the statistics ofthe associated invariant measure such as the Fourier spectrum, the spectrum of the proper orthogonaldecomposition (POD), the point-wise distribution, the auto-correlation, and other domain-specificstatistics such as the turbulence kinetic energy and the dissipation rate.
Revised: In order to assess its performance, we study the statistics ofthe associated invariant measure such as the Fourier spectrum, the spectrum of the proper orthogonaldecomposition (POD), the point-wise distribution, the auto-correlation, and other domain-specificstatistics such as the turbulence kinetic energy and the dissipation rate.

Original: Furthermore we study thebehavior of our leaned operator over long horizons and ensure that it does not blow up or collapse butrather accurately follows the global attractor.
Revised: Furthermore we study thebehavior of our leaned operator over long horizons and ensure that it does not blow up or collapse butrather accurately follows the global attractor.

Original: In this work:
Revised: In this work:

Original: • We theoretically prove that, under suitable conditions, the MNO can approximate the underlying
Revised: • We theoretically prove that, under suitable conditions, the MNO can approximate the underlying

Original: solution operator of chaotic PDEs, while conventional neural networks lack such strong guarantees.• We impose dissipativity by augmenting the data on an outer shell to enforce that the dynamicevolution stays close to the attractor.
Revised: solution operator of chaotic PDEs, while conventional neural networks lack such strong guarantees.• We impose dissipativity by augmenting the data on an outer shell to enforce that the dynamicevolution stays close to the attractor.

Original: As an additional safeguard, we impose a hard dissipativity
Revised: As an additional safeguard, we impose a hard dissipativity

Original: constraint on MNO predictions far from the attractor and the data augmentation shell.
Revised: constraint on MNO predictions far from the attractor and the data augmentation shell.

Original: We show
Revised: We show

Original: this is crucial for learning in a chaotic regime as demonstrated by Figure 1.
Revised: this is crucial for learning in a chaotic regime as demonstrated by Figure 1.

Original: The resulting systemremains stable against
Revised: The resulting systemremains stable against

Original: large perturbations.•
Revised: large perturbations.•

Original: We study the choice of time steps for training the MNO, demonstrating that the error follows avalley-shaped phenomenon (cf.
Revised: We study the choice of time steps for training the MNO, demonstrating that the error follows avalley-shaped phenomenon (cf.

Original: This gives rise to a recipe for choosing the optimal timestep for accurate learning.•
Revised: This gives rise to a recipe for choosing the optimal timestep for accurate learning.•

Original: We show that standard mean square error (MSE) type losses for training are not adequate, and themodels often fail to capture the higher frequency information induced from the derivatives.
Revised: We show that standard mean square error (MSE) type losses for training are not adequate, and themodels often fail to capture the higher frequency information induced from the derivatives.

Original: Weinvestigate various Sobolev losses in operator learning.
Revised: Weinvestigate various Sobolev losses in operator learning.

Original: We show that using Sobolev norms fortraining captures higher-order derivatives and moments, as well as high frequency details [33, 34].This is similar in spirit to pre-multiplying the spectrum by the wavenumber, an approach commonlyused in the study of fluid systems [35].
Revised: We show that using Sobolev norms fortraining captures higher-order derivatives and moments, as well as high frequency details [35, 36].This is similar in spirit to pre-multiplying the spectrum by the wavenumber, an approach commonlyused in the study of fluid systems [37].

Original: • We investigate multiple existing deep learning architectures, including U-Net [36], long short-termmemory convolution neural networks (LSTM-CNN) [37]
Revised: • We investigate multiple existing deep learning architectures, including U-Net [38], long short-termmemory convolution neural networks (LSTM-CNN) [39]

Original: and gated recurrent units (GRU) [38], inplace of the neural operator to learn the solution operator.
Revised: and gated recurrent units (GRU) [40], inplace of the neural operator to learn the solution operator.

Original: We show MNO provides an order ofmagnitude lower error on all loss functions studied.
Revised: We show MNO provides an order ofmagnitude lower error on all loss functions studied.

Original: Furthermore, we show that MNO outperformsthe aforementioned neural network architectures on all statistics studied.
Revised: Furthermore, we show that MNO outperformsthe aforementioned neural network architectures on all statistics studied.

Original: In summary, we propose a principled approach to the learning dissipative chaotic systems.
Revised: In summary, we propose a principled approach to the learning dissipative chaotic systems.

Original: Dissipativity is enforced through data augmentation and through post-processing of the learned model.
Revised: Dissipativity is enforced through data augmentation and through post-processing of the learned model.

Original: Themethodology allows for treatment of infinite dimensional dynamical systems (PDEs) and leads tomethods with short term trajectory accuracy and desirable long-term statistical properties.
Revised: Themethodology allows for treatment of infinite dimensional dynamical systems (PDEs) and leads tomethods with short term trajectory accuracy and desirable long-term statistical properties.

Original:  Problem setting
Revised:  Problem setting

Original: We consider potentially infinite dimensional dynamical systems where the phase space U is a Banachspace and, in particular, a function space on a Lipschitz domain D ⊂ R d (for finite dimensionalsystems, U will be a Euclidean space).
Revised: We consider potentially infinite dimensional dynamical systems where the phase space U is a Banachspace and, in particular, a function space on a Lipschitz domain D ⊂ R d (for finite dimensionalsystems, U will be a Euclidean space).

Original: We are interested in the initial-value problemdu dt ( t ) =
Revised: We are interested in the initial-value problemdu dt ( t ) =

Original: F ( u ( t )) , u (0) =
Revised: F ( u ( t )) , u (0) =

Original: u 0 , t ∈ (0 , ∞ ) (1) for initial conditions u 0 ∈ U where F is usually a non-linear operator.
Revised: u 0 , t ∈ (0 , ∞ ) (1) for initial conditions u 0 ∈ U where F is usually a non-linear operator.

Original: We will assume, given someappropriate boundary conditions on ∂D when applicable, the solution u ( t )
Revised: We will assume, given someappropriate boundary conditions on ∂D when applicable, the solution u ( t )

Original: ∈
Revised: ∈

Original: U exists and is uniquefor all times t ∈ (0 , ∞ ) .
Revised: U exists and is uniquefor all times t ∈ (0 , ∞ ) .

Original: When making the spatial dependence explicit, if it is present, we willwrite u ( x, t ) to indicate the evaluation u ( t ) | x for any x ∈ D .
Revised: When making the spatial dependence explicit, if it is present, we willwrite u ( x, t ) to indicate the evaluation u ( t ) | x for any x ∈ D .

Original: We define the family of operators
Revised: We define the family of operators

Original: S t : U → U as mapping u 

Revised: S t : U → U as mapping u

Original: (cid:55)→ u ( t ) for any t ≥ 0 , and note that, since (1) is autonomous, S t satisfiesthe Markov property i.e. S t ( S s ( u 0 ))
Revised: (cid:55)→ u ( t ) for any t ≥ 0 , and note that, since (1) is autonomous, S t satisfiesthe Markov property i.e. S t ( S s ( u 0 ))

Original: =
Revised: =

Original: u ( s + t ) for any s, t ≥ 0 .
Revised: u ( s + t ) for any s, t ≥ 0 .

Original: We adopt the viewpoint of castingtime-dependent PDEs into function space ODEs (1), as this leads to the semigroup approach toevolutionary PDEs which underlies our learning methodology.
Revised: We adopt the viewpoint of castingtime-dependent PDEs into function space ODEs (1), as this leads to the semigroup approach toevolutionary PDEs which underlies our learning methodology.

Original: One approach to this problem is to learn F itself (e.g., [39–42]).
Revised: One approach to this problem is to learn F itself (e.g., [41–44]).

Original: However, in infinite dimensions,
Revised: However, in infinite dimensions,

Original: F is an unbounded operator and learning unbounded operators is considerably more challengingthan learning bounded operators; this is proven in [43] for linear operators.
Revised: F is an unbounded operator and learning unbounded operators is considerably more challengingthan learning bounded operators; this is proven in [45] for linear operators.

Original: We avoid these issues byinstead learning the solution operator S t for some fixed t > 0 ; this map is bounded as an operatoron U .
Revised: We avoid these issues byinstead learning the solution operator S t for some fixed t > 0 ; this map is bounded as an operatoron U .

Original: Furthermore, large time statistical behaviour can be obtained more efficiently by using thesolution operator and not resolving dynamics at timescales smaller than time t .
Revised: Furthermore, large time statistical behaviour can be obtained more efficiently by using thesolution operator and not resolving dynamics at timescales smaller than time t .

Original: Dissipativity.
Revised: Dissipativity.

Original: Systems for which there exists some fixed, bounded, positively-invariant set E suchthat for any bounded B ⊂ U , there is some time t ∗ =
Revised: Systems for which there exists some fixed, bounded, positively-invariant set E suchthat for any bounded B ⊂ U , there is some time t ∗ =

Original: t ∗ ( B ) beyond which the dynamics of anytrajectory starting in B enters and remains in E are known as dissipative systems [13, 14].
Revised: t ∗ ( B ) beyond which the dynamics of anytrajectory starting in B enters and remains in E are known as dissipative systems [13, 14].

Original: The set
Revised: The set

Original: E is known as the absorbing set of the system.
Revised: E is known as the absorbing set of the system.

Original: For such systems, the global attractor A , definedsubsequently, is characterized as the ω -limit set of E .
Revised: For such systems, the global attractor A , definedsubsequently, is characterized as the ω -limit set of E .

Original: In particular, for any initial condition u 0 ∈ U ,the trajectory u ( t ) approaches A as t → ∞ .
Revised: In particular, for any initial condition u 0 ∈ U ,the trajectory u ( t ) approaches A as t → ∞ .

Original: In this work, we consider dissipative dynamical systemswhere there exist some α ≥ 0 and β > 0 such that
Revised: In this work, we consider dissipative dynamical systemswhere there exist some α ≥ 0 and β > 0 such that

Original: [Equation]
Revised: [Equation]

Original: for all u ∈ U .
Revised: for all u ∈ U .

Original: It can be shown that systems which satisfy this inequality are dissipative [14] withthe absorbing set E , an open ball of radius (cid:112) α/β + ε for any ε > 0 .
Revised: It can be shown that systems which satisfy this inequality are dissipative [14] withthe absorbing set E , an open ball of radius (cid:112) α/β + ε for any ε > 0 .

Original: There are several well-knownexamples of dynamical systems that satisfy the above inequality.
Revised: There are several well-knownexamples of dynamical systems that satisfy the above inequality.

Original: In this paper we consider thefinite-dimensional Lorenz-63 system and the infinite-dimensional cases of the Kuramoto-Sivashinskyand 2D incompressible Navier-Stokes equations, in the form of Kolmogorov flows [13].
Revised: In this paper we consider thefinite-dimensional Lorenz-63 system and the infinite-dimensional cases of the Kuramoto-Sivashinskyand 2D incompressible Navier-Stokes equations, in the form of Kolmogorov flows [13].

Original: Global Attractors.
Revised: Global Attractors.

Original: A compact, invariant set A is called a global attractor if, for any bounded set
Revised: A compact, invariant set A is called a global attractor if, for any bounded set

Original: B ⊂ U and any ϵ > 0 there exists a time t ∗ =
Revised: B ⊂ U and any ϵ > 0 there exists a time t ∗ =

Original: t ∗ ( ϵ, B ) such that S t ( B ) is contained within an ϵneighborhood of A for all t ≥ t ∗ .
Revised: t ∗ ( ϵ, B ) such that S t ( B ) is contained within an ϵneighborhood of A for all t ≥ t ∗ .

Original: Formally it is the ω -limit set of any absorbing set; this characterizesit uniquely [13, 14] under assumption (2).
Revised: Formally it is the ω -limit set of any absorbing set; this characterizesit uniquely [13, 14] under assumption (2).

Original: Although unique, the ω -limit set of a single initial conditionmay differ across initial conditions [14].
Revised: Although unique, the ω -limit set of a single initial conditionmay differ across initial conditions [14].

Original: In this paper we concentrate on the ergodic setting where thesame ω -limit set is seen for almost all initial conditions with respect to the invariant measure.
Revised: In this paper we concentrate on the ergodic setting where thesame ω -limit set is seen for almost all initial conditions with respect to the invariant measure.

Original: Many
Revised: Many

Original: PDEs arising in physics such as reaction-diffusion equations (dynamics of biochemical systems)
Revised: PDEs arising in physics such as reaction-diffusion equations (dynamics of biochemical systems)

Original: orthe Kuramoto-Sivashinky and Navier-Stokes equation (fluid mechanics) are dissipative and possessa global attractor which is often finite-dimensional [13].
Revised: orthe Kuramoto-Sivashinky and Navier-Stokes equation (fluid mechanics) are dissipative and possessa global attractor which is often finite-dimensional [13].

Original: Therefore, numerically characterizing theattractor is an important problem in scientific computing with many potential applications.
Revised: Therefore, numerically characterizing theattractor is an important problem in scientific computing with many potential applications.

Original: Data distribution.
Revised: Data distribution.

Original: For many applications, an exact form for the possible initial conditions to (1)is not available; it is therefore convenient to use a stochastic model to describe the initial states.
Revised: For many applications, an exact form for the possible initial conditions to (1)is not available; it is therefore convenient to use a stochastic model to describe the initial states.

Original: Tothat end, let µ 0 be a probability measure on U and assume that all possible initial conditions to (1)come as samples from µ 0 i.e. u 0 ∼ µ 0 .
Revised: Tothat end, let µ 0 be a probability measure on U and assume that all possible initial conditions to (1)come as samples from µ 0 i.e. u 0 ∼ µ 0 .

Original: Then any possible state of the dynamic (1) after some timet > 0 can be thought of as being distributed according to the pushforward measure µ t :=
Revised: Then any possible state of the dynamic (1) after some timet > 0 can be thought of as being distributed according to the pushforward measure µ t :=

Original: S ♯t µ 0 i.e.u ( t ) ∼ µ t .
Revised: S ♯t µ 0 i.e.u ( t ) ∼ µ t .

Original: Therefore as the dynamic evolves, so does the type of likely functions that result.
Revised: Therefore as the dynamic evolves, so does the type of likely functions that result.

Original: Thisfurther complicates the problem of long time predictions since training data may only be obtainedup to finite time horizons hence the model will need the ability to predict not only on data that isout-of-sample but also out-of-distribution.
Revised: Thisfurther complicates the problem of long time predictions since training data may only be obtainedup to finite time horizons hence the model will need the ability to predict not only on data that isout-of-sample but also out-of-distribution.

Original: Ergodic systems.
Revised: Ergodic systems.

Original: To alleviate some of the previously presented challenges, we consider ergodicsystems.
Revised: To alleviate some of the previously presented challenges, we consider ergodicsystems.

Original: Roughly speaking, a system is ergodic if there exists an invariant measure µ which isunchanged by pushforward under S t and time averages of functionals on state space U convergeto averages against µ .
Revised: Roughly speaking, a system is ergodic if there exists an invariant measure µ which isunchanged by pushforward under S t and time averages of functionals on state space U convergeto averages against µ .

Original: For dissipative systems the invariant measure is supported on the globalattractor A and together the pair ( A, µ ) capture the large time dynamics and their statistics.
Revised: For dissipative systems the invariant measure is supported on the globalattractor A and together the pair ( A, µ ) capture the large time dynamics and their statistics.

Original: Provingergodicity is hard for deterministic systems.
Revised: Provingergodicity is hard for deterministic systems.

Original: Rigorous proofs have been developed for the Lorenz-63model [12]; empirical evidence of ergodicity may be seen much more widely, including for the
Revised: Rigorous proofs have been developed for the Lorenz-63model [12]; empirical evidence of ergodicity may be seen much more widely, including for the

Original: Kuramoto-Sivashinsky and Navier-Stokes equations studied in this paper.
Revised: Kuramoto-Sivashinsky and Navier-Stokes equations studied in this paper.

Original: For stochastic differentialequations ergodicity is easier to establish; see [44] for details.
Revised: For stochastic differentialequations ergodicity is easier to establish; see [46] for details.

Original: Ergodicity mitigates learning a model that is able to predict out-of-distribution since both the inputand the output of ˆ S h , an approximation to S h , will approximately be distributed according to µ .
Revised: Ergodicity mitigates learning a model that is able to predict out-of-distribution since both the inputand the output of ˆ S h , an approximation to S h , will approximately be distributed according to µ .

Original: Furthermore, we may use ˆ S h to learn about µ since sampling it simply corresponds to running thedynamic forward.
Revised: Furthermore, we may use ˆ S h to learn about µ since sampling it simply corresponds to running thedynamic forward.

Original: Indeed, we need only generate data on a finite time horizon in order to learn ˆ S h ,and, once learned, we may use it to sample µ indefinitely by repeatedly composing ˆ S h with itself.
Revised: Indeed, we need only generate data on a finite time horizon in order to learn ˆ S h ,and, once learned, we may use it to sample µ indefinitely by repeatedly composing ˆ S h with itself.

Original: Having samples of µ then allows us to compute statistics which characterize the long term behavior ofthe system and therefore the global attractor A .
Revised: Having samples of µ then allows us to compute statistics which characterize the long term behavior ofthe system and therefore the global attractor A .

Original: This strategy avoids the issue of accumulating errorsin long term trajectory predictions since we are only interested in the property that ˆ S h ( u ( t )) ∼ µ .
Revised: This strategy avoids the issue of accumulating errorsin long term trajectory predictions since we are only interested in the property that ˆ S h ( u ( t )) ∼ µ .

Original:  Learning the Markov neural operator in chaotic dynamics
Revised:  Learning the Markov neural operator in chaotic dynamics

Original: We propose the Markov neural operator (MNO), a method for learning the underlying solutionoperators of autonomous, dissipative, chaotic dynamical systems.
Revised: We propose the Markov neural operator (MNO), a method for learning the underlying solutionoperators of autonomous, dissipative, chaotic dynamical systems.

Original: In particular, we approximate theoperator mapping the solution from the current to the next step ˆ S h : u ( t ) (cid:55)→ u ( t + h ) .
Revised: In particular, we approximate theoperator mapping the solution from the current to the next step ˆ S h : u ( t ) (cid:55)→ u ( t + h ) .

Original: We approximatethe solution operator S h , an element of the underlying continuous time semigroup { S t : t ∈ [0 , ∞ ) } ,using a neural operator as in Figure 2; in the finite dimensional case we use a feedforward neuralnetwork.
Revised: We approximatethe solution operator S h , an element of the underlying continuous time semigroup { S t : t ∈ [0 , ∞ ) } ,using a neural operator as in Figure 2; in the finite dimensional case we use a feedforward neuralnetwork.

Original: See Appendix B.1 for background on the solution operator and semigroup.
Revised: See Appendix B.1 for background on the solution operator and semigroup.

Original: Long-term predictions.
Revised: Long-term predictions.

Original: Having access to the map ˆ S h , its semigroup properties allow for approximating long time trajectories of (1) by repeatedly composing ˆ S h with its own output.
Revised: Having access to the map ˆ S h , its semigroup properties allow for approximating long time trajectories of (1) by repeatedly composing ˆ S h with its own output.

Original: Therefore, forany n ∈ N , we compute u ( nh ) as follows,
Revised: Therefore, forany n ∈ N , we compute u ( nh ) as follows,

Original: [Equation]
Revised: [Equation]

Original: The above semigroup formulation can be applied with various choices of the backbone model for ˆ S h .
Revised: The above semigroup formulation can be applied with various choices of the backbone model for ˆ S h .

Original: In general, we prefer models that can be evaluated quickly and have approximation guarantees so theper-step error can be controlled.
Revised: In general, we prefer models that can be evaluated quickly and have approximation guarantees so theper-step error can be controlled.

Original: Therefore, we choose the standard feed-forward neural network [45]for ODE systems, and the Fourier neural operator [46] for infinite dimensional PDE systems.
Revised: Therefore, we choose the standard feed-forward neural network [47]for ODE systems, and the Fourier neural operator [48] for infinite dimensional PDE systems.

Original: For the the neural operator parametric class, we prove the following theorem regarding the MNO.
Revised: For the the neural operator parametric class, we prove the following theorem regarding the MNO.

Original: The result states that our construction can approximate trajectories of infinite-dimensional dynamicalsystems arbitrary well.
Revised: The result states that our construction can approximate trajectories of infinite-dimensional dynamicalsystems arbitrary well.

Original: The proof is given in Appendix B.3.
Revised: The proof is given in Appendix B.3.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let K ⊂ U be a compact set and assume that, for some h > 0 , the solution operator
Revised: Let K ⊂ U be a compact set and assume that, for some h > 0 , the solution operator

Original: S h : U → U associated to the dynamic (1) is locally Lipschitz.
Revised: S h : U → U associated to the dynamic (1) is locally Lipschitz.

Original: Then, for any n ∈ N and ϵ > 0 thereexists a neural operator ˆ S h : U → U such that
Revised: Then, for any n ∈ N and ϵ > 0 thereexists a neural operator ˆ S h : U → U such that

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 indicates that if the backbone model is rich enough, it can approximate many chaoticsystems for arbitrarily long periods.
Revised: Theorem 1 indicates that if the backbone model is rich enough, it can approximate many chaoticsystems for arbitrarily long periods.

Original: For finite-dimensional systems, the same theorem holds withfeed-forward neural networks instead of neural operators.
Revised: For finite-dimensional systems, the same theorem holds withfeed-forward neural networks instead of neural operators.

Original: We note that standard neural networks suchas RNNs and CNNs do not possess such approximation theorems in the infinite-dimensional setting.
Revised: We note that standard neural networks suchas RNNs and CNNs do not possess such approximation theorems in the infinite-dimensional setting.

Original: Invariant statistics.
Revised: Invariant statistics.

Original: A useful application of the solution operators is to estimate statistics of theinvariant measure of a chaotic system.
Revised: A useful application of the solution operators is to estimate statistics of theinvariant measure of a chaotic system.

Original: Assume the target system is ergodic and there exists aninvariant measure µ as discussed in Section 2.
Revised: Assume the target system is ergodic and there exists aninvariant measure µ as discussed in Section 2.

Original: An invariant statistic is defined as
Revised: An invariant statistic is defined as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for any functional G : U → R d .
Revised: for any functional G : U → R d .

Original: Examples include the L 2 norm, any spectral coefficients, andthe spatial correlation, as well as problem-specific statistics such as the turbulence kinetic energyand dissipation rates in fluid flow problems.
Revised: Examples include the L 2 norm, any spectral coefficients, andthe spatial correlation, as well as problem-specific statistics such as the turbulence kinetic energyand dissipation rates in fluid flow problems.

Original: Given the property (3) and using the ergodicity from(4), the approximate model ˆ S h can be used to estimate any invariant statistic simply by computing
Revised: Given the property (3) and using the ergodicity from(4), the approximate model ˆ S h can be used to estimate any invariant statistic simply by computing

Original: T G ≈ 1 n (cid:80) nk =1 G ( S kh ( u 0 )) where nh = T and T > 0 large enough.
Revised: T G ≈ 1 n (cid:80) nk =1 G ( S kh ( u 0 )) where nh = T and T > 0 large enough.

Original: Encouraging dissipativity via regularization.
Revised: Encouraging dissipativity via regularization.

Original: In practice, training data for dissipative systemsis typically drawn from trajectories close to the global attractor, so a priori there is no guarantee ofa learned model’s behavior far from the attractor.
Revised: In practice, training data for dissipative systemsis typically drawn from trajectories close to the global attractor, so a priori there is no guarantee ofa learned model’s behavior far from the attractor.

Original: Thus, if we seek to learn the global attractor andinvariant statistics of a dynamical system, it is crucial that we encourage learning dissipativity.
Revised: Thus, if we seek to learn the global attractor andinvariant statistics of a dynamical system, it is crucial that we encourage learning dissipativity.

Original: Tolearn ˆ S h : u ( t ) (cid:55)→ u ( t + h ) we propose the following loss function, found by supplementing the
Revised: Tolearn ˆ S h : u ( t ) (cid:55)→ u ( t + h ) we propose the following loss function, found by supplementing the

Original: Sobolev loss (see Appendix B.2) with a dissipativity-inducing regularization term:
Revised: Sobolev loss (see Appendix B.2) with a dissipativity-inducing regularization term:

Original: [Equation]
Revised: [Equation]

Original: Here µ is the underlying distribution of the training data, α is a loss weighting hyperparameter, and0 < λ < 1 is some constant factor for scaling down (i.e., enforcing dissipativity) inputs u drawnfrom a probability measure ν .
Revised: Here µ is the underlying distribution of the training data, α is a loss weighting hyperparameter, and0 < λ < 1 is some constant factor for scaling down (i.e., enforcing dissipativity) inputs u drawnfrom a probability measure ν .

Original: We choose ν to be a uniform probability distribution supported onsome shell with a fixed inner and outer radii from the origin in U ; these are chosen so that the supportof ν is disjoint from the attractor.
Revised: We choose ν to be a uniform probability distribution supported onsome shell with a fixed inner and outer radii from the origin in U ; these are chosen so that the supportof ν is disjoint from the attractor.

Original: Our dissipative regularization term scales down u by constant λ ,but in principle alternative dissipative cost functionals can be used.
Revised: Our dissipative regularization term scales down u by constant λ ,but in principle alternative dissipative cost functionals can be used.

Original: Enforcing dissipativity via post-processing.
Revised: Enforcing dissipativity via post-processing.

Original: While the previous dissipativity regularization enforces a dissipative map in practice, it does not readily yield to theoretical guarantees of dissipativity.
Revised: While the previous dissipativity regularization enforces a dissipative map in practice, it does not readily yield to theoretical guarantees of dissipativity.

Original: For an additional safeguard against model instability, we enforce a hard dissipativity constraint farfrom the attractor and from the shell where ν is supported, resulting in provably dissipative dynamics.
Revised: For an additional safeguard against model instability, we enforce a hard dissipativity constraint farfrom the attractor and from the shell where ν is supported, resulting in provably dissipative dynamics.

Original: In detail, we post-process the model: whenever the dynamic moves out of an a priori defined stableregion, we switch to the second model Ψ that pushes the dynamic back.
Revised: In detail, we post-process the model: whenever the dynamic moves out of an a priori defined stableregion, we switch to the second model Ψ that pushes the dynamic back.

Original: The new model combinesthe learned model ˆ S h and the safety model Ψ , via a threshold function ρ :
Revised: The new model combinesthe learned model ˆ S h and the safety model Ψ , via a threshold function ρ :

Original: [Equation]
Revised: [Equation]

Original: where Ψ is some dissipative map and ρ is a partition of unity.
Revised: where Ψ is some dissipative map and ρ is a partition of unity.

Original: For simplicity we define
Revised: For simplicity we define

Original: [Equation]
Revised: [Equation]

Original: 7) where α is the effective transition radius between ˆ S h and Ψ and β controls the transition rate.
Revised: 7) where α is the effective transition radius between ˆ S h and Ψ and β controls the transition rate.

Original: Notethat this choice of Ψ is consistent with the regularization term in the loss (5).
Revised: Notethat this choice of Ψ is consistent with the regularization term in the loss (5).

Original: By its construction, the post-processed model is dissipative.
Revised: By its construction, the post-processed model is dissipative.

Original: However, its dynamics (in particularthe unlearnable transition between ˆ S h and Ψ ) are not as smooth as the dissipative regularizationmodel, which learn the whole dynamic together, as shown in Figure 3.
Revised: However, its dynamics (in particularthe unlearnable transition between ˆ S h and Ψ ) are not as smooth as the dissipative regularizationmodel, which learn the whole dynamic together, as shown in Figure 3.

Original: Combining regularization andpost-processing results in a dissipative model with both smooth dynamics and theoretical guarantees.
Revised: Combining regularization andpost-processing results in a dissipative model with both smooth dynamics and theoretical guarantees.

Original:  Experiments
Revised:  Experiments

Original: We evaluate our approach on the finite-dimensional, chaotic Lorenz-63 system as well as the chaotic Kuramoto-Sivashinsky and 2D Navier-Stokes equations.
Revised: We evaluate our approach on the finite-dimensional, chaotic Lorenz-63 system as well as the chaotic1D Kuramoto-Sivashinsky and 2D Navier-Stokes equations.

Original: In all cases we show that encouragingdissipativity is crucial for capturing the global attractor and evaluating statistics of the invariantmeasure.
Revised: In all cases we show that encouragingdissipativity is crucial for capturing the global attractor and evaluating statistics of the invariantmeasure.

Original: To the best of our knowledge, we showcase the first machine learning method able to predictthe statistical behavior of the incompressible Navier-Stokes equation in a strongly chaotic regime.
Revised: To the best of our knowledge, we showcase the first machine learning method able to predictthe statistical behavior of the incompressible Navier-Stokes equation in a strongly chaotic regime.

Original: 4.1 Lorenz-63 system
Revised: 4.1 Lorenz-63 system

Original: To motivate and justify our framework for learning chaotic systems in the infinite-dimensionalsetting, we first apply our framework on the simple yet still highly chaotic Lorenz-63 ODE system, awidely-studied [47] simplified model for atmospheric dynamics given by
Revised: To motivate and justify our framework for learning chaotic systems in the infinite-dimensionalsetting, we first apply our framework on the simple yet still highly chaotic Lorenz-63 ODE system, awidely-studied [49] simplified model for atmospheric dynamics given by

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We use the canonical parameters ( α, b, r ) =
Revised: We use the canonical parameters ( α, b, r ) =

Original: Since the solution operator ofthe Lorenz-63 system is finite-dimensional, we learn it by training a feedforward neural networkon a single trajectory with h = 0 .
Revised: Since the solution operator ofthe Lorenz-63 system is finite-dimensional, we learn it by training a feedforward neural networkon a single trajectory with h = 0 .

Original: 05 s on the Lorenz attractor.
Revised: 05 s on the Lorenz attractor.

Original: Figure 3 shows that dissipativityregularization produces predictions that isotropically point towards the attractor.
Revised: Figure 3 shows that dissipativityregularization produces predictions that isotropically point towards the attractor.

Original: Observe that the ournetwork is also dissipative outside the regularization shell.
Revised: Observe that the ournetwork is also dissipative outside the regularization shell.

Original: We empirically find that dissipativity can be encouraged without significantly affecting the model’sstep-wise relative L 2 error and the learned statistical properties of the attractor.
Revised: We empirically find that dissipativity can be encouraged without significantly affecting the model’sstep-wise relative L 2 error and the learned statistical properties of the attractor.

Original: We also find thatvarying the dissipativity hyperparameters (loss weight α , scaling factor λ , and shell radius) does notsignificantly affect the step-wise or dissipative error.
Revised: We also find thatvarying the dissipativity hyperparameters (loss weight α , scaling factor λ , and shell radius) does notsignificantly affect the step-wise or dissipative error.

Original: Details in Appendix A.1.
Revised: Details in Appendix A.1.

Original: 4.2 Kuramoto-Sivashinsky equation
Revised: 4.2 Kuramoto-Sivashinsky equation

Original: We consider the following one-dimensional KS equation,∂u
Revised: We consider the following one-dimensional KS equation,∂u

Original: [Equation]
Revised: [Equation]

Original: initial condition
Revised: initial condition

Original: [Equation]
Revised: [Equation]

Original: where the spatial domain [0 , L ] is equipped with periodic boundary conditions.
Revised: where the spatial domain [0 , L ] is equipped with periodic boundary conditions.

Original: We study the impactthe time step h has on learning.
Revised: We study the impactthe time step h has on learning.

Original: Our study shows that when the time steps are too large, the correlationis chaotic and hard to capture.
Revised: Our study shows that when the time steps are too large, the correlationis chaotic and hard to capture.

Original: But counter-intuitively, when the time steps are too small, the evolutionis also hard to capture.
Revised: But counter-intuitively, when the time steps are too small, the evolutionis also hard to capture.

Original: In this case, the model’s input and output are very close, and the identity map
Revised: In this case, the model’s input and output are very close, and the identity map

Original: [Figure]
Revised: [Figure]

Original: will be a local minimum.
Revised: will be a local minimum.

Original: We thus use the MNO to also learn the time-derivative or residual.
Revised: We thus use the MNO to also learn the time-derivative or residual.

Original: Figure 4ashows
Revised: Figure 4ashows

Original: the results for varying h and when MNO is used to learn either the identity residual or thesolution operator.
Revised: the results for varying h and when MNO is used to learn either the identity residual or thesolution operator.

Original: We observe that the residual model has a better per-step error and accumulatederror at smaller h .
Revised: We observe that the residual model has a better per-step error and accumulatederror at smaller h .

Original: When the time step is large, there is no difference in modeling the residual.
Revised: When the time step is large, there is no difference in modeling the residual.

Original: As shown in Figures 4b and 4c, we compare the performance of the MNO model against LSTMand GRU that we use to model the evolution operator of the KS equation with h = 1 s .
Revised: As shown in Figures 4b and 4c, we compare the performance of the MNO model against LSTMand GRU that we use to model the evolution operator of the KS equation with h = 1 s .

Original: We observethat MNO model accurately recovers the Fourier spectrum of KS equation.
Revised: We observethat MNO model accurately recovers the Fourier spectrum of KS equation.

Original: For all other statistics(see Appendix A.2) all models perform similarly, except on the velocity distribution where MNOperforms best.
Revised: For all other statistics(see Appendix A.2) all models perform similarly, except on the velocity distribution where MNOperforms best.

Original: We emphasize that some of these statistics are very challenging to capture and mostmachine learning approaches in the literature thus far fail to do so (see Appendix A.2 for details).
Revised: We emphasize that some of these statistics are very challenging to capture and mostmachine learning approaches in the literature thus far fail to do so (see Appendix A.2 for details).

Original: 4.3 Kolmogorov Flow
Revised: 4.3 Kolmogorov Flow

Original: We consider two-dimensional Kolmogorov flow (a form of the Navier-Stokes equations) for a viscous,incompressible fluid,
Revised: We consider two-dimensional Kolmogorov flow (a form of the Navier-Stokes equations) for a viscous,incompressible fluid,

Original: [Equation]
Revised: [Equation]

Original: with initial condition u ( · , 0) =
Revised: with initial condition u ( · , 0) =

Original: u 0 where u denotes the velocity, p the pressure, and Re > 0 is the
Revised: u 0 where u denotes the velocity, p the pressure, and Re > 0 is the

Original: Reynolds number.
Revised: Reynolds number.

Original: We encourage dissipativity during training with the criterion described in eq.
Revised: We encourage dissipativity during training with the criterion described in eq.

Original: (5),with λ = 0 .
Revised: (5),with λ = 0 .

Original: 5 and ν being a uniform probability distribution supported on a shell around the origin.
Revised: 5 and ν being a uniform probability distribution supported on a shell around the origin.

Original: We test the effect of encouraging dissipativity in the turbulent (and blow-up prone) Re = 500 setting,where we observe that a non-dissipative model blows up when composed with itself multiple times ifthe initial condition is perturbed slightly from the attractor (Figure 1), even though the model achievesrelatively low L 2 error.
Revised: We test the effect of encouraging dissipativity in the turbulent (and blow-up prone) Re = 500 setting,where we observe that a non-dissipative model blows up when composed with itself multiple times ifthe initial condition is perturbed slightly from the attractor (Figure 1), even though the model achievesrelatively low L 2 error.

Original: In contrast, we empirically observe that the dissipative MNO does not blowup and its composed predictions returns to the attractor even when the initial condition is perturbed.
Revised: In contrast, we empirically observe that the dissipative MNO does not blowup and its composed predictions returns to the attractor even when the initial condition is perturbed.

Original: Simulation on turbulent Re = 5000 case.
Revised: Simulation on turbulent Re = 5000 case.

Original: The proposed dissipative MNO model can stably learncomplex chaotic dynamics at high Reynolds numbers.
Revised: The proposed dissipative MNO model can stably learncomplex chaotic dynamics at high Reynolds numbers.

Original: As shown in Figure 6, the model captures theenergy spectrum that converges to the Kolmogorov energy cascade rate of k − 5 / 3 .
Revised: As shown in Figure 6, the model captures theenergy spectrum that converges to the Kolmogorov energy cascade rate of k − 5 / 3 .

Original: The learned modelis dissipative with the dissipative regularization, without compromising the accuracy as shown in
Revised: The learned modelis dissipative with the dissipative regularization, without compromising the accuracy as shown in

Original: Table 5.
Revised: Table 5.

Original: Even given a O (10 4 ) -scaled perturbation, the dynamic quickly returns to the attractor of thesystem.
Revised: Even given a O (10 4 ) -scaled perturbation, the dynamic quickly returns to the attractor of thesystem.

Original: In contrast, the model without dissipative regularization will stay outside the attractor giveneven a O (10) -scale perturbation.
Revised: In contrast, the model without dissipative regularization will stay outside the attractor giveneven a O (10) -scale perturbation.

Original: Accuracy with respect to various norms.
Revised: Accuracy with respect to various norms.

Original: We study performance of MNO along with otherpredictive architectures, U-Net [36] and LSTM-CNN [37], on modeling the vorticity w in therelatively non-turbulent Re = 40 case 3.
Revised: We study performance of MNO along with otherpredictive architectures, U-Net [38] and LSTM-CNN [39], on modeling the vorticity w in therelatively non-turbulent Re = 40 case 3.

Original: The MNO achieves one order of magnitude better accuracycompared to this architectures.
Revised: The MNO achieves one order of magnitude better accuracycompared to this architectures.

Original: As shown in Table 3, we train each model using the balanced
Revised: As shown in Table 3, we train each model using the balanced

Original: L 2 (= H 0 ) , H 1 , and H 2 losses, defined as the sum of the relative L 2 loss grouped by each order ofderivative.
Revised: L 2 (= H 0 ) , H 1 , and H 2 losses, defined as the sum of the relative L 2 loss grouped by each order ofderivative.

Original: We measure the error with respect to the standard (unbalanced) norms.
Revised: We measure the error with respect to the standard (unbalanced) norms.

Original: The MNO with
Revised: The MNO with

Original: H 2 loss consistently achieves the smallest error on vorticity for all of the L 2 , H 1 , and H 2 norms.
Revised: H 2 loss consistently achieves the smallest error on vorticity for all of the L 2 , H 1 , and H 2 norms.

Original: However, the L 2 loss achieves the smallest error on the turbulence kinetic energy (TKE), while the
Revised: However, the L 2 loss achieves the smallest error on the turbulence kinetic energy (TKE), while the

Original: H 1 loss achieves the smallest error on the dissipation ϵ .
Revised: H 1 loss achieves the smallest error on the dissipation ϵ .

Original: Vorticity fields for NS equation ( Re = 40 )are shown in Figure 11.
Revised: Vorticity fields for NS equation ( Re = 40 )are shown in Figure 11.

Original: The figure indicates that the predicted trajectories (b) (c) (d) share the samebehaviors as in the ground truth (a), indicating that the MNO model is stable.
Revised: The figure indicates that the predicted trajectories (b) (c) (d) share the samebehaviors as in the ground truth (a), indicating that the MNO model is stable.

Original: Estimating the attractor and invariant statistics.
Revised: Estimating the attractor and invariant statistics.

Original: We compose MNO 10000 times to obtain theglobal attractor, and we compute the PCA (POD) basis of these 10000 snapshots and project themonto the first two components.
Revised: We compose MNO 10000 times to obtain theglobal attractor, and we compute the PCA (POD) basis of these 10000 snapshots and project themonto the first two components.

Original: As shown in Figure 5a, we obtain a cycle-shaped attractor.
Revised: As shown in Figure 5a, we obtain a cycle-shaped attractor.

Original: The trueattractor has dimension on the order of O (100) [13].
Revised: The trueattractor has dimension on the order of O (100) [13].

Original: If the attractor is a high-dimensional sphere,then most of its mass concentrates around the equator.
Revised: If the attractor is a high-dimensional sphere,then most of its mass concentrates around the equator.

Original: Therefore, when projected to low-dimension,the attractor will have a ring shape.
Revised: Therefore, when projected to low-dimension,the attractor will have a ring shape.

Original: We see that most points are located on the ring, while few pointsare located in the center.
Revised: We see that most points are located on the ring, while few pointsare located in the center.

Original: The points in the center have high dissipation, implying they are intermittentstates.
Revised: The points in the center have high dissipation, implying they are intermittentstates.

Original: In Figure 5b we add the time axis.
Revised: In Figure 5b we add the time axis.

Original: While the trajectory jumps around the cycle, we observethere is a rough period of 2000 s. We perform the same PCA on the training data, showing the samebehavior.
Revised: While the trajectory jumps around the cycle, we observethere is a rough period of 2000 s. We perform the same PCA on the training data, showing the samebehavior.

Original: Furthermore, in Figure 12, we present invariant statistics for the NS equation ( Re = 40 ),computed from a large number of samples from our approximation of the invariant measure.
Revised: Furthermore, in Figure 12, we present invariant statistics for the NS equation ( Re = 40 ),computed from a large number of samples from our approximation of the invariant measure.

Original: We findthat the MNO consistently outperforms all other models in accurately capturing these statistics.
Revised: We findthat the MNO consistently outperforms all other models in accurately capturing these statistics.

Original: Derivative orders.
Revised: Derivative orders.

Original: Roughly speaking, vorticity is the derivative of velocity; velocity is the derivativeof the stream function.
Revised: Roughly speaking, vorticity is the derivative of velocity; velocity is the derivativeof the stream function.

Original: Therefore we can denote the order of derivative of vorticity, velocity, andstream function as 2 , 1 , and 0 respectively.
Revised: Therefore we can denote the order of derivative of vorticity, velocity, andstream function as 2 , 1 , and 0 respectively.

Original: Combining vorticity, velocity, and stream function, with
Revised: Combining vorticity, velocity, and stream function, with

Original: L 2 , H 1 , and H 2 loss, we have in total the order of derivatives ranging from 0 to 4 .
Revised: L 2 , H 1 , and H 2 loss, we have in total the order of derivatives ranging from 0 to 4 .

Original: We observe, ingeneral, it is best practice to keep the order of derivatives in the model at a number slightly higherthan
Revised: We observe, ingeneral, it is best practice to keep the order of derivatives in the model at a number slightly higherthan

Original: that of the target quantity.
Revised: that of the target quantity.

Original: For example, as shown in Figure 5d, when querying the velocity(first-order quantity), it is best to use second-order (modeling velocity plus H 1 loss or modelingvorticity plus L 2 loss).
Revised: For example, as shown in Figure 5d, when querying the velocity(first-order quantity), it is best to use second-order (modeling velocity plus H 1 loss or modelingvorticity plus L 2 loss).

Original: This is further illustrated in Table 4 for the Re = 500 case.
Revised: This is further illustrated in Table 4 for the Re = 500 case.

Original: In general, usinga higher order of derivatives as the loss will increase the power of the model and capture the invariantstatistics more accurately.
Revised: In general, usinga higher order of derivatives as the loss will increase the power of the model and capture the invariantstatistics more accurately.

Original: However, a higher-order of derivative means higher irregularity.
Revised: However, a higher-order of derivative means higher irregularity.

Original: It in turnrequires a higher resolution for the model to resolve and for computing the discrete Fourier transform.
Revised: It in turnrequires a higher resolution for the model to resolve and for computing the discrete Fourier transform.

Original: This trade-off again suggests it is best to pick a Sobolev norm not too low or too high.
Revised: This trade-off again suggests it is best to pick a Sobolev norm not too low or too high.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes]
Revised: [Yes]

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data areproprietary.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section C(c)
Revised: [Yes] See Section C(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Statement ofTheorem 1(b) Did you include complete proofs of all theoretical results?
Revised: [Yes] Statement ofTheorem 1(b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix3.
Revised: [Yes] See Appendix3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] we plan to doso.
Revised: [Yes] we plan to do so.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Specified in Section
Revised: [Yes] Specified in Section

Original: (c) Did you report error bars (e.g., with respect to the random seed after running ex-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running ex-

Original: periments multiple times)?
Revised: periments multiple times)?

Original: [No] The error is consistent in the operator learning for
Revised: [No] The error is consistent in the operator learning for

Original: PDEs.
Revised: PDEs.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] Specified in Section 
Revised: [Yes] Specified in Section 44.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] Supplement
Revised: [Yes] Supplement

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: Supplement
Revised: Supplement

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Full Experiments
Revised: A Full Experiments

Original: [Figure]

Original: In this section, we present the full numerical experiments on the Lorenz-63 system, the KuramotoSivashinsky equation, and the Kolmogorov flow.
Revised: In this section, we present the full numerical experiments on the Lorenz-63 system, the KuramotoSivashinsky equation, and the Kolmogorov flow.

Original: A.1 Lorenz-63
Revised: A.1 Lorenz-63

Original: The Lorenz-63 system is three dimensional ODE system:
Revised: The Lorenz-63 system is three dimensional ODE system:

Original: [Equation]
Revised: [Equation]

Original: In this paper, we use the canonical parameters ( α, b, r ) =
Revised: In this paper, we use the canonical parameters ( α, b, r ) =

Original: Learning the solution operator.
Revised: Learning the solution operator.

Original: We use a simple feedforward neural network with 6 hidden layersand 150 neurons per layer to learn the solution operator for the Lorenz-63 system.
Revised: We use a simple feedforward neural network with 6 hidden layersand 150 neurons per layer to learn the solution operator for the Lorenz-63 system.

Original: We discretize thetraining trajectory into time-steps of 0.05 seconds.
Revised: We discretize thetraining trajectory into time-steps of 0.05 seconds.

Original: Dissipativity regularization.
Revised: Dissipativity regularization.

Original: We enforce dissipativity during training with the criterion describedin eq.
Revised: We enforce dissipativity during training with the criterion describedin eq.

Original: 5, with λ = 0 .
Revised: 5, with λ = 0 .

Original: 5 and ν being a uniform probability distribution supported on a shell around theorigin with inner radius 90 and outer radius 130 .
Revised: 5 and ν being a uniform probability distribution supported on a shell around theorigin with inner radius 90 and outer radius 130 .

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We evaluate the effectiveness of our technique for enforcing dissipativity inthree ways:
Revised: We evaluate the effectiveness of our technique for enforcing dissipativity inthree ways:

Original: • Relative L2 error:
Revised: • Relative L2 error:

Original: We evaluate the learned solution operators on relative (i
Revised: We evaluate the learned solution operators on relative (i

Original: e., normalized)
Revised: e., normalized)

Original: L2 error over 1 second by composing the model 20 times to gauge their ability to predict
Revised: L2 error over 1 second by composing the model 20 times to gauge their ability to predict

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: longer trajectories.
Revised: longer trajectories.

Original: As seen in Table 1, enforcing dissipativity does not cause a decrease inrelative L2 error when compared to the baseline network, even though both models have thesame number of parameters.
Revised: As seen in Table 1, enforcing dissipativity does not cause a decrease inrelative L2 error when compared to the baseline network, even though both models have thesame number of parameters.

Original: • Vector field far from the attractor: We also qualitatively evaluate the learned vector
Revised: • Vector field far from the attractor: We also qualitatively evaluate the learned vector

Original: fields far from the attractor both with and without enforced dissipativity.
Revised: fields far from the attractor both with and without enforced dissipativity.

Original: We observe that
Revised: We observe that

Original: enforcing dissipativity produces predictions that isotropically point towards the attractor,
Revised: enforcing dissipativity produces predictions that isotropically point towards the attractor,

Original: implying that the attractive properties of the Lorenz attractor are learned in the process.
Revised: implying that the attractive properties of the Lorenz attractor are learned in the process.

Original: See
Revised: See

Original: Figure8.
Revised: Figure

Original: Observe that the dissipative network is also dissipative outside the shell in which
Revised: Observe that the dissipative network is also dissipative outside the shell in which

Original: dissipativity was enforced during training.
Revised: dissipativity was enforced during training.

Original: • Invariant statistics of the attractor: To justify learning the Markovian map between
Revised: • Invariant statistics of the attractor: To justify learning the Markovian map between

Original: time-steps, we also compare the coordinate histograms of the ground-truth and the learned
Revised: time-steps, we also compare the coordinate histograms of the ground-truth and the learned

Original: attractors (with and without dissipativity enforced) after composing for 200000 time steps.
Revised: attractors (with and without dissipativity enforced) after composing for 200000 time steps.

Original: Both models match the ground-truth coordinate histograms.
Revised: Both models match the ground-truth coordinate histograms.

Original: Since the coordinate-wise
Revised: Since the coordinate-wise

Original: histograms
Revised: histograms

Original: of the models matches the ground-truth, this indicates that the models arelearning the distribution of the attractor and thus the invariant measure of the system.
Revised: of the models matches the ground-truth, this indicates that the models arelearning the distribution of the attractor and thus the invariant measure of the system.

Original: Choice of dissipativity hyperparameters.
Revised: Choice of dissipativity hyperparameters.

Original: We also study the choice of dissipativity hyperparameters in eq.
Revised: We also study the choice of dissipativity hyperparameters in eq.

Original: In particular, we investigate the choice of α, λ and the radius between the two
Revised: In particular, we investigate the choice of α, λ and the radius between the two

Original: Model
Revised: Model

Original: Per-step error
Revised: Per-step error

Original: Per-second error
Revised: Per-second error

Original: Dissipativity error
Revised: Dissipativity error

Original: [Table]
Revised: [Table]

Original: as in Table
Revised: as in Table

Original: = 50 s .
Revised: = 50 s .

Original: shells on which measure ν is supported.
Revised: shells on which measure ν is supported.

Original: We conduct this study on the Lorenz system because it isfinite-dimensional and simpler to visualize and interpret.
Revised: We conduct this study on the Lorenz system because it isfinite-dimensional and simpler to visualize and interpret.

Original: In our ablation experiments we vary each hyperparameter while holding the other ones constant atdefault values of a radius of 40 , α = 1 , and λ = 0 .
Revised: In our ablation experiments we vary each hyperparameter while holding the other ones constant atdefault values of a radius of 40 , α = 1 , and λ = 0 .

Original: Further, we define the inner radius to be sup ∥ x
Revised: Further, we define the inner radius to be sup ∥ x

Original: ∥ λover all x in the training set.
Revised: ∥ λover all x in the training set.

Original: The results are presented in Table 2.
Revised: The results are presented in Table 2.

Original: The choice of the dissipativity shell radius, λ , and α empirically does not have a significant impacton the model’s capacity to learn the dynamics on the attractor and the enforced dissipative dynamics.
Revised: The choice of the dissipativity shell radius, λ , and α empirically does not have a significant impacton the model’s capacity to learn the dynamics on the attractor and the enforced dissipative dynamics.

Original: As such, we select the smallest inner radius of r i = sup
Revised: As such, we select the smallest inner radius of r i = sup

Original: ∥ x ∥ λ (over all x in the training set) on whichdissipativity remains well-defined.
Revised: ∥ x ∥ λ (over all x in the training set) on whichdissipativity remains well-defined.

Original: Note that if ∥ x ∥ < r i , then λx would lie inside the smallest ballcontaining the attractor.
Revised: Note that if ∥ x ∥ < r i , then λx would lie inside the smallest ballcontaining the attractor.

Original: A.2 Kuramoto-Sivashinsky equation
Revised: A.2 Kuramoto-Sivashinsky equation

Original: We consider the following one-dimensional Kuramoto-Sivashinsky equation,
Revised: We consider the following one-dimensional Kuramoto-Sivashinsky equation,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where L = 32 π or 64 π and the spatial domain [0 , L ] is equipped with periodic boundary conditions.
Revised: where L = 32 π or 64 π and the spatial domain [0 , L ] is equipped with periodic boundary conditions.

Original: We assume the initial condition u 0 ∈ ˙
Revised: We assume the initial condition u 0 ∈ ˙

Original: L 2 per ([0 , L ]; R ) , where ˙
Revised: L 2 per ([0 , L ]; R ) , where ˙

Original: L 2 per ([0 , L ]; R ) is the space of all meanzero L 2 -functions that are periodic on [0 , L ] .
Revised: L 2 per ([0 , L ]; R ) is the space of all meanzero L 2 -functions that are periodic on [0 , L ] .

Original: Existence of the semigroup S t : ˙
Revised: Existence of the semigroup S t : ˙

Original: L 2 per ([0 , L ]; R ) →˙
Revised: L 2 per ([0 , L ]; R ) →˙

Original: L 2 per ([0 , L ]; R ) is established in [13, Theorem 3.1].
Revised: L 2 per ([0 , L ]; R ) is established in [13, Theorem 3.1].

Original: Data is obtained by solving the equation using theexponential time-differencing fourth-order Runge-Kutta method from [49].
Revised: Data is obtained by solving the equation using theexponential time-differencing fourth-order Runge-Kutta method from [51].

Original: Random initial conditionsare generated according to a mean zero Gaussian measure with covariance L − 2 /α τ 12 (2 α − 1) ( − ∆ +( τ 2 /L 2 ) I ) − α where α = 2 , τ = 7 , and periodic boundary conditions on [0 , L ] ; for details see [50].
Revised: Random initial conditionsare generated according to a mean zero Gaussian measure with covariance L − 2 /α τ 12 (2 α − 1) ( − ∆ +( τ 2 /L 2 ) I ) − α where α = 2 , τ = 7 , and periodic boundary conditions on [0 , L ] ; for details see [52].

Original: Benchmarks for Kuramoto-Sivashinsky.
Revised: Benchmarks for Kuramoto-Sivashinsky.

Original: We compare MNO with common choices of recurrentneural networks including the long short-term memory network (LSTM)[51] and gated recurrent unit(GRU)[38].
Revised: We compare MNO with common choices of recurrentneural networks including the long short-term memory network (LSTM)[53] and gated recurrent unit(GRU)[40].

Original: All models use the time-discretization h = 1 s .
Revised: All models use the time-discretization h = 1 s .

Original: The training dataset consists of 1000different realizations of trajectories on the time interval t ∈ [50 , 200] (the first 50 s is truncated sothe dynamics reach the ergodic state), which adds up to 1000 × 150 = 150 , 000 snapshots in total.
Revised: The training dataset consists of 1000different realizations of trajectories on the time interval t ∈ [50 , 200] (the first 50 s is truncated sothe dynamics reach the ergodic state), which adds up to 1000 × 150 = 150 , 000 snapshots in total.

Original: Another 200 realizations are generated for testing.
Revised: Another 200 realizations are generated for testing.

Original: Every single snapshot has the resolution 2048 .
Revised: Every single snapshot has the resolution 2048 .

Original: Weuse Adam optimizer to minimize the relative L 2 loss with learning rate = 0 .
Revised: Weuse Adam optimizer to minimize the relative L 2 loss with learning rate = 0 .

Original: 001 , and step learningrate scheduler that decays by half every 10 epochs for 50 epochs in total.
Revised: 001 , and step learningrate scheduler that decays by half every 10 epochs for 50 epochs in total.

Original: LSTM and GRU: havingtested many different configurations, we choose the best hyper-parameters: the number of layers = 1 ,width = 1000 .
Revised: LSTM and GRU: havingtested many different configurations, we choose the best hyper-parameters: the number of layers = 1 ,width = 1000 .

Original: During the evaluation, we additionally provide 1000 snapshots as a warm-up of thememory.
Revised: During the evaluation, we additionally provide 1000 snapshots as a warm-up of thememory.

Original: MNO: we choose 1-d Fourier neural operator as our base model with four Fourier layerswith 20 frequencies per channel and width = 64 .
Revised: MNO: we choose 1-d Fourier neural operator as our base model with four Fourier layerswith 20 frequencies per channel and width = 64 .

Original: Experiments run on Nvidia V100 GPUs.
Revised: Experiments run on Nvidia V100 GPUs.

Original: Accuracy with respect to the true trajectory.
Revised: Accuracy with respect to the true trajectory.

Original: In general, MNO has a smaller per-step errorcompared to RNN.
Revised: In general, MNO has a smaller per-step errorcompared to RNN.

Original: As shown in Figure 9, the MNO model captures a longer period of the exacttrajectory compared to LSTM and GRU.
Revised: As shown in Figure 9, the MNO model captures a longer period of the exacttrajectory compared to LSTM and GRU.

Original: LSTM and GRU start to diverge at t = 20 s while FNO isable to keep up with the exact trajectory until t = 50 s .
Revised: LSTM and GRU start to diverge at t = 20 s while FNO isable to keep up with the exact trajectory until t = 50 s .

Original: Invariant statistics for the KS equations As shown in Figure 10, we present enormous invariantstatistics for the KS equation.
Revised: Invariant statistics for the KS equations As shown in Figure 10, we present enormous invariantstatistics for the KS equation.

Original: We use 150000 snapshots to train the MNO, LSTM, and GRU to modelthe evolution operator of the KS equation with h = 1 s .
Revised: We use 150000 snapshots to train the MNO, LSTM, and GRU to modelthe evolution operator of the KS equation with h = 1 s .

Original: We compose each model for T = 1000 timesteps to obtain a long trajectory (attractor), and estimate various invariant statistics from them.
Revised: We compose each model for T = 1000 timesteps to obtain a long trajectory (attractor), and estimate various invariant statistics from them.

Original: • (a) Fourier spectrum : the Fourier spectrum of the predicted attractor.
Revised: • (a) Fourier spectrum : the Fourier spectrum of the predicted attractor.

Original: All models are able
Revised: All models are able

Original: to capture the Fourier modes with magnitude larger than O (1) , while MNO is more accurate
Revised: to capture the Fourier modes with magnitude larger than O (1) , while MNO is more accurate

Original: on the tail.
Revised: on the tail.

Original: • (b) Spatial correlation : the spatial correlation of the attractor, averaged in the time di-
Revised: • (b) Spatial correlation : the spatial correlation of the attractor, averaged in the time di-

Original: mension.
Revised: mension.

Original: MNO is more accurate on the near-range correlation, but all models miss the
Revised: MNO is more accurate on the near-range correlation, but all models miss the

Original: long-range correlation.
Revised: long-range correlation.

Original: • (c) Auto-correlation of the Fourier mode : the auto-correlation of the 10 th Fourier mode.
Revised: • (c) Auto-correlation of the Fourier mode : the auto-correlation of the 10 th Fourier mode.

Original: Since the Fourier modes are nearly constant, the auto -correlation is constant too
Revised: Since the Fourier modes are nearly constant, the auto-correlation is constant too.

Original: • (d) Auto-correlation of the PCA mode : 
Revised: • (d) Auto-correlation of the PCA mode : the auto correlation

Original: the
Revised: the

Original: respect 
Revised: respect to the

Revised: basis

Original: the
Revised: the

Original:  • (e) Distribution of kinetic energy : the distribution of kinetic energy with respect to the
Revised: data • (e) Distribution of kinetic energy : the distribution of kinetic energy with respect to the

Original: time dimension.
Revised: time dimension.

Original: MNO captures the distribution most accurately.
Revised: MNO captures the distribution most accurately.

Original: • (f) Pixelwise distribution of velocity : since the KS equation is homogeneous, we can
Revised: • (f) Pixelwise distribution of velocity : since the KS equation is homogeneous, we can

Original: compute the distribution of velocity with respect to pixels.
Revised: compute the distribution of velocity with respect to pixels.

Original: MNO captures the pixelwise
Revised: MNO captures the pixelwise

Original: distribution most accurately too.
Revised: distribution most accurately too.

Original: Choice of time discretization.
Revised: Choice of time discretization.

Original: We further study the choice of time steps h .
Revised: We further study the choice of time steps h .

Original: As shown in Figure 4a,when the time steps are too large, the correlation is chaotic and hard to capture.
Revised: As shown in Figure 4a,when the time steps are too large, the correlation is chaotic and hard to capture.

Original: But counter-intuitively,when the time steps are too small, the evolution is also hard to capture.
Revised: But counter-intuitively,when the time steps are too small, the evolution is also hard to capture.

Original: In this case, the input andoutput of the network will be very close, and the identity map will be a local minimum.
Revised: In this case, the input andoutput of the network will be very close, and the identity map will be a local minimum.

Original: An easy fixis to use MNO to learn the time-derivative or residual.
Revised: An easy fixis to use MNO to learn the time-derivative or residual.

Original: This is shown in the figure, where the residualmodel (blue line) has a better per-step error and accumulated error at smaller h .
Revised: This is shown in the figure, where the residualmodel (blue line) has a better per-step error and accumulated error at smaller h .

Original: When the time stepis large, there is no difference in modeling the residual.
Revised: When the time stepis large, there is no difference in modeling the residual.

Original: This idea can generalize to other integratorsas an extension of Neural ODEs to PDEs [42].
Revised: This idea can generalize to other integratorsas an extension of Neural ODEs to PDEs [44].

Original: A.3 Kolmogorov Flow
Revised: A.3 Kolmogorov Flow

Original: We consider the two-dimensional Navier-Stokes equation for a viscous, incompressible fluid,
Revised: We consider the two-dimensional Navier-Stokes equation for a viscous, incompressible fluid,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where u denotes the velocity, p the pressure, and Re > 0 is the Reynolds number.
Revised: where u denotes the velocity, p the pressure, and Re > 0 is the Reynolds number.

Original: The domain[0 , 2 π ] 2 is equipped with periodic boundary conditions.
Revised: The domain[0 , 2 π ] 2 is equipped with periodic boundary conditions.

Original: The specific choice of forcing sin( ny )ˆ xconstitutes a Kolmogorov flow; we choose n = 4 in all experiments.
Revised: The specific choice of forcing sin( ny )ˆ xconstitutes a Kolmogorov flow; we choose n = 4 in all experiments.

Original: We define U to be the closedsubspace of L 2 ([0 , 2 π ] 2 ; R 2 ) , U = (cid:8) u ∈ ˙
Revised: We define U to be the closedsubspace of L 2 ([0 , 2 π ] 2 ; R 2 ) , U = (cid:8) u ∈ ˙

Original: L 2 per ([0 , 2 π ] 2 ; R 2 ) : ∇ · u = 0 (cid:9) and assume u 0 ∈ U .
Revised: L 2 per ([0 , 2 π ] 2 ; R 2 ) : ∇ · u = 0 (cid:9) and assume u 0 ∈ U .

Original: Wedefine the vorticity w = ( ∇ × u )ˆ z and the stream function f as the solution to the Poisson equation− ∆ f = w .
Revised: Wedefine the vorticity w = ( ∇ × u )ˆ z and the stream function f as the solution to the Poisson equation− ∆ f = w .

Original: Existence of the semigroup S t : U → U is established in [13, Theorem 2.1].
Revised: Existence of the semigroup S t : U → U is established in [13, Theorem 2.1].

Original: Wedenote turbulence kinetic energy (TKE) ⟨ ( u − ¯ u ) 2 ⟩ , and dissipation ϵ =
Revised: Wedenote turbulence kinetic energy (TKE) ⟨ ( u − ¯ u ) 2 ⟩ , and dissipation ϵ =

Original: ⟨ w 2 ⟩ /Re .
Revised: ⟨ w 2 ⟩ /Re .

Original: Data is obtainedby solving the equation in vorticity form using the pseudo-spectral split step method from [19].
Revised: Data is obtainedby solving the equation in vorticity form using the pseudo-spectral split step method from [19].

Original: Random initial conditions are generated according to a mean zero Gaussian measure with covariance7 3 / 2 ( − ∆ + 49 I ) − 2 .
Revised: Random initial conditions are generated according to a mean zero Gaussian measure with covariance7 3 / 2 ( − ∆ + 49 I ) − 2 .

Original: 5 with periodic boundary conditions on [0 , 2 π ] 2 .
Revised: 5 with periodic boundary conditions on [0 , 2 π ] 2 .

Original: Benchmarks for the 2d Kolmogorov flow.
Revised: Benchmarks for the 2d Kolmogorov flow.

Original: We compare MNO with common standard twodimensional dynamic models including U-Net[36] and LSTM-CNN[37] on modeling the vorticityw in the relatively non-turbulent Re = 40 case.
Revised: We compare MNO with common standard twodimensional dynamic models including U-Net[38] and LSTM-CNN[39] on modeling the vorticityw in the relatively non-turbulent Re = 40 case.

Original: We choose the discretization h
Revised: We choose the discretization h

Original: = 1 s .
Revised: = 1 s .

Original: The trainingdataset consists of 180 realizations of trajectories on time interval t ∈ [100 , 500] (the first 100 secondsare discarded) which adds up to 180 × 400 = 72 , 000 snapshots in total.
Revised: The trainingdataset consists of 180 realizations of trajectories on time interval t ∈ [100 , 500] (the first 100 secondsare discarded) which adds up to 180 × 400 = 72 , 000 snapshots in total.

Original: Another 20 realizationsare generated for testing.
Revised: Another 20 realizationsare generated for testing.

Original: Each single snapshot has resolution 64 × 64 .
Revised: Each single snapshot has resolution 64 × 64 .

Original: We use the Adam optimizerto minimize the relative L 2 loss with learning rate = 0 . 0005 , and step learning rate scheduler thatdecays by half every 10 epochs for 50 epochs in total.
Revised: We use the Adam optimizerto minimize the relative L 2 loss with learning rate = 0 . 0005 , and step learning rate scheduler thatdecays by half every 10 epochs for 50 epochs in total.

Original: U-Net: we use five layers of convolution anddeconvolution with width from 64 to 1024 .
Revised: U-Net: we use five layers of convolution anddeconvolution with width from 64 to 1024 .

Original: LSTM-CNN: we use one layer of LSTM with width= 64 .
Revised: LSTM-CNN: we use one layer of LSTM with width= 64 .

Original: MNO: we parameterize the 2-d Fourier neural operator consists of four Fourier layers with 20frequencies per channel and width = 64 .
Revised: MNO: we parameterize the 2-d Fourier neural operator consists of four Fourier layers with 20frequencies per channel and width = 64 .

Original: Enforcing dissipativity.
Revised: Enforcing dissipativity.

Original: We enforce dissipativity during training with the criterion described in eq.5, with λ = 0 .
Revised: We enforce dissipativity during training with the criterion described in eq.5, with λ = 0 .

Original: 5 and ν being a uniform probability distribution supported on a shell around the origin.
Revised: 5 and ν being a uniform probability distribution supported on a shell around the origin.

Original: Accuracy with respect to various norms.
Revised: Accuracy with respect to various norms.

Original: MNO shows near one order of magnitude better accuracycompared to U-Net and LSTM-CNN.
Revised: MNO shows near one order of magnitude better accuracycompared to U-Net and LSTM-CNN.

Original: As shown in Table 3, we train each model using the balanced
Revised: As shown in Table 3, we train each model using the balanced

Original: L 2 (= H 0 ) , H 1 , and H 2 losses, defined as the sum of the relative L 2 loss grouped by each order ofderivative.
Revised: L 2 (= H 0 ) , H 1 , and H 2 losses, defined as the sum of the relative L 2 loss grouped by each order ofderivative.

Original: And we measure the error with respect to the standard (unbalanced) norms.
Revised: And we measure the error with respect to the standard (unbalanced) norms.

Original: The MNOwith H 2 loss consistently achieves the smallest error on vorticity on all of the L 2 , H 1 , and H 2norms.
Revised: The MNOwith H 2 loss consistently achieves the smallest error on vorticity on all of the L 2 , H 1 , and H 2norms.

Original: However, L 2 loss achieves the smallest error on the turbulence kinetic energy (TKE); H 1 lossachieves the smallest error on the dissipation ϵ .
Revised: However, L 2 loss achieves the smallest error on the turbulence kinetic energy (TKE); H 1 lossachieves the smallest error on the dissipation ϵ .

Original: The NS equation with Re 40 is shown in Figure 11.
Revised: The NS equation with Re 40 is shown in Figure 11.

Original: (a) show the ground truth data of vorticityfield, each column represents a snapshot at t = 100 s with a different initial condition.
Revised: (a) show the ground truth data of vorticityfield, each column represents a snapshot at t = 100 s with a different initial condition.

Original: (b), (c),(d) show the predicted trajectory of MNO on vorticity, using L 2 , H 1 , and H 2 losses respectively.
Revised: (b), (c),(d) show the predicted trajectory of MNO on vorticity, using L 2 , H 1 , and H 2 losses respectively.

Original: We are able to generate a long trajectory with the MNO model.
Revised: We are able to generate a long trajectory with the MNO model.

Original: The five columns represent t =1000 s, 2000 s, 3000 s, 4000 s, 5000 s respectively.
Revised: The five columns represent t =1000 s, 2000 s, 3000 s, 4000 s, 5000 s respectively.

Original: As shown in the figure, the predicted trajectories(b) (c) (d) share the same behaviors as in the ground truth (a).
Revised: As shown in the figure, the predicted trajectories(b) (c) (d) share the same behaviors as in the ground truth (a).

Original: It indicates the MNO model is stable.
Revised: It indicates the MNO model is stable.

Original: Visualizing the attractor generated by MNO.
Revised: Visualizing the attractor generated by MNO.

Original: We compose MNO 10000 times to obtain theglobal attractor, and we compute the PCA (POD) basis of these 10000 snapshots and project themonto the first two components.
Revised: We compose MNO 10000 times to obtain theglobal attractor, and we compute the PCA (POD) basis of these 10000 snapshots and project themonto the first two components.

Original: As shown in Figure (a), we obtain a cycle-shaped attractor.
Revised: As shown in Figure (a), we obtain a cycle-shaped attractor.

Original: The trueattractor has a degree of freedom around O (100) [13].
Revised: The trueattractor has a degree of freedom around O (100) [13].

Original: If the attractor is a high-dimensional sphere,then most of the mass concentrates around its equator.
Revised: If the attractor is a high-dimensional sphere,then most of the mass concentrates around its equator.

Original: Therefore, when projected to low-dimension,the attractor will have the shape of a ring.
Revised: Therefore, when projected to low-dimension,the attractor will have the shape of a ring.

Original: Most of the points are located on the ring, while a fewother points are located in the center.
Revised: Most of the points are located on the ring, while a fewother points are located in the center.

Original: The points in the center have high dissipation, implying they areintermittent states.
Revised: The points in the center have high dissipation, implying they areintermittent states.

Original: In Figure (b) we add the time axis.
Revised: In Figure (b) we add the time axis.

Original: While the trajectory jumps around the cycle,we observe there is a rough period of 2000 s. We perform the same PCA analysis on the training data,which shows the same behavior.
Revised: While the trajectory jumps around the cycle,we observe there is a rough period of 2000 s. We perform the same PCA analysis on the training data,which shows the same behavior.

Original: Invariant statistics.
Revised: Invariant statistics.

Original: Similarly, we present enormous invariant statistics for the NS equation (Re 40 ),as shown in Figure 12,.
Revised: Similarly, we present enormous invariant statistics for the NS equation (Re 40 ),as shown in Figure 12,.

Original: We use 72000 snapshots to train the MNO, UNet, and ConvLSTM to modelthe evolution operator of the KS equation with h = 1 s .
Revised: We use 72000 snapshots to train the MNO, UNet, and ConvLSTM to modelthe evolution operator of the KS equation with h = 1 s .

Original: We compose each model for T = 10000 timesteps to obtain a long trajectory (attractor), and estimate various invariant statistics from them.
Revised: We compose each model for T = 10000 timesteps to obtain a long trajectory (attractor), and estimate various invariant statistics from them.

Original: • (a, d) Fourier spectrum of velocity and vorticity : the Fourier spectrum of the predicted
Revised: • (a, d) Fourier spectrum of velocity and vorticity : the Fourier spectrum of the predicted

Original: attractor.
Revised: attractor.

Original: Again, all models are able to capture the Fourier modes with magnitude larger
Revised: Again, all models are able to capture the Fourier modes with magnitude larger

Original: than O (1) , while MNO is more accurate on the tail
Revised: than O (1) , while MNO is more accurate on the tail.

Original: Using the Sobolev norm further helpsto capture the tail.
Revised: Using the Sobolev norm further helpsto capture the tail.

Original: • (b, e) Pixelwise distribution of velocity and vorticity : All models preserve the pixelwisedistribution.
Revised: • (b, e) Pixelwise distribution of velocity and vorticity : All models preserve the pixelwise distribution.

Original: • (c, f) Distribution of kinetic energy and dissipation rate : the distribution of kinetic energy
Revised: • (c, f) Distribution of kinetic energy and dissipation rate : the distribution of kinetic energy

Original: with respect to the time dimension.
Revised: with respect to the time dimension.

Original: MNO captures the distribution most accurately. • (g) Auto-correlation of the Fourier mode : the auto-correlation of the 10 th Fourier mode.
Revised: MNO captures the distribution most accurately.• (g) Auto-correlation of the Fourier mode : the auto-correlation of the 10 th Fourier mode.

Original: Since the Fourier modes are nearly constant, the auto-correlation is constant too.
Revised: Since the Fourier modes are nearly constant, the auto-correlation is constant too.

Original: Notice it is very expensive to generate long-time ground truth data, so the figure does not include the
Revised: Notice it isvery expensive to generate long-time ground truth data, so the figure does not include the

Original: ground truth.
Revised: ground truth.

Original: However, it is easy to obtain the auto-correlation by MNO.
Revised: However, it is easy to obtain the auto-correlation by MNO.

Original: • (h) Auto-correlation of the PCA mode : the auto-correlation of the first PCA mode (with respect to the PCA basis of the ground truth data).
Revised: • (h) Auto-correlation of the PCA mode : the auto-correlation of the first PCA mode (withrespect to the PCA basis of the ground truth data).

Original: The PCA mode oscillates around[ − 1000 , 1000] , showing an ergodic state.
Revised: The PCA mode oscillates around[ − 1000 , 1000] , showing an ergodic state.

Original: The UNet oscillates around [0 , 2000] .
Revised: The UNet oscillates around [0 , 2000] .

Original: • (i) Spatial correlation : the spatial correlation of the attractor, averaged in the time dimension.
Revised: • (i) Spatial correlation : the spatial correlation of the attractor, averaged in the time dimension.

Original: The four columns represent the truth and MNO with different losses.
Revised: The four columns represent the truth and MNO with different losses.

Original: As seen from thefigure, there is a wave pattern matching the force term sin(4 y ) .
Revised: As seen from thefigure, there is a wave pattern matching the force term sin(4 y ) .

Original: Order of derivatives.
Revised: Order of derivatives.

Original: Roughly speaking, vorticity is the derivative of velocity; velocity is thederivative of the stream function.
Revised: Roughly speaking, vorticity is the derivative of velocity; velocity is thederivative of the stream function.

Original: Therefore we can denote the order of derivative of vorticity, velocity,and stream function as 2 , 1 , and 0 respectively.
Revised: Therefore we can denote the order of derivative of vorticity, velocity,and stream function as 2 , 1 , and 0 respectively.

Original: Combining vorticity, velocity, and stream function,with L 2 , H 1 , and H 2 loss, we have in total the order of derivatives ranging from 0 to 4 .
Revised: Combining vorticity, velocity, and stream function,with L 2 , H 1 , and H 2 loss, we have in total the order of derivatives ranging from 0 to 4 .

Original: We observe,in general, it is best practice to keep the order of derivatives in the model at a number slightly higherthan
Revised: We observe,in general, it is best practice to keep the order of derivatives in the model at a number slightly higherthan

Original: that of the target quantity.
Revised: that of the target quantity.

Original: For example, as shown in Figure 5d, when querying the velocity(first-order quantity), it is best to use second-order (modeling velocity plus H 1 loss or modelingvorticity plus L 2 loss).
Revised: For example, as shown in Figure 5d, when querying the velocity(first-order quantity), it is best to use second-order (modeling velocity plus H 1 loss or modelingvorticity plus L 2 loss).

Original: This is further illustrated in Table 4.
Revised: This is further illustrated in Table 4.

Original: In general, using a higher order ofderivatives as the loss will increase the power of the model and capture the invariant statistics moreaccurately.
Revised: In general, using a higher order ofderivatives as the loss will increase the power of the model and capture the invariant statistics moreaccurately.

Original: However, a higher-order of derivative means higher irregularity.
Revised: However, a higher-order of derivative means higher irregularity.

Original: It in turn requires ahigher resolution for the model to resolve and for computing the discrete Fourier transform.
Revised: It in turn requires ahigher resolution for the model to resolve and for computing the discrete Fourier transform.

Original: Thistrade-off again suggests it is best to pick a Sobolev norm not too low or too high.
Revised: Thistrade-off again suggests it is best to pick a Sobolev norm not too low or too high.

Original: [Table]
Revised: [Table]

Original: B Theoretical background
Revised: B Theoretical backgroundB.1 Solution operators and the semigroup

Original: B.1 Solution operators and the semigroup

Original: Since the system (1) is autonomous, that is, F does not explicitly depend on time, under the wellposedness assumption, we may define, for any t ∈ [0 , ∞ ) , a solution operator S t : U
Revised: Since the system (1) is autonomous, that is, F does not explicitly depend on time, under the wellposedness assumption, we may define, for any t ∈ [0 , ∞ ) , a solution operator S t : U

Original: → U such thatu ( t ) =
Revised: → U such thatu ( t ) =

Original: S t u (0) .
Revised: S t u (0) .

Original: This map satisfies the properties
Revised: This map satisfies the properties

Original: S 

Revised: S

Original: = I ,2.
Revised: = I ,

Original: S t ( u 0 ) =
Revised: S t ( u 0 ) =

Original: u ( t ) ,3.
Revised: u ( t ) ,3.

Original: S t ( S s ( u 0 )) =
Revised: S t ( S s ( u 0 )) =

Original: u ( t + s ) ,
Revised: u ( t + s ) ,

Original: [Table]
Revised: [Table]

Original: on f error on u error on w
Revised: on f error on u error on w

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: for any s, t ∈ [0 , ∞ ) and any u 0 ∈ U where I denotes the identity operator on U .
Revised: for any s, t ∈ [0 , ∞ ) and any u 0 ∈ U where I denotes the identity operator on U .

Original: In particular, thefamily { S t : t ∈ [0 , ∞ ) } defines a semigroup of operators acting on U .
Revised: In particular, thefamily { S t : t ∈ [0 , ∞ ) } defines a semigroup of operators acting on U .

Original: Our goal is to approximate aparticular element of this semigroup associated to some fixed time step h > 0 given observations ofthe trajectory from (1).
Revised: Our goal is to approximate aparticular element of this semigroup associated to some fixed time step h > 0 given observations ofthe trajectory from (1).

Original: We build an approximation ˆ S h : U → U such that
Revised: We build an approximation ˆ S h : U → U such that

Original: [Equation]
Revised: [Equation]

Original: B.2 Sobolev norm loss
Revised: B.2 Sobolev norm loss

Original: We incorporate Sobolev norms in the training process to better capture invariant statistics of the learnedoperator.
Revised: We incorporate Sobolev norms in the training process to better capture invariant statistics of the learnedoperator.

Original: Given the ground truth operator S h and the learned operator ˆ S h and f = ˆ S h ( u ) − S h ( u ) ,we compute the step-wise loss in the Sobolev norm
Revised: Given the ground truth operator S h and the learned operator ˆ S h and f = ˆ S h ( u ) − S h ( u ) ,we compute the step-wise loss in the Sobolev norm

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In particular, we use p = 2 , where the Sobolev norm can be easily computed in Fourier space:
Revised: In particular, we use p = 2 , where the Sobolev norm can be easily computed in Fourier space:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where ˆ f is the Fourier series of f .
Revised: 14) where ˆ f is the Fourier series of f .

Original: In practice, we use k = 0 , 1 , 2 .
Revised: In practice, we use k = 0 , 1 , 2 .

Original: B.3 Proof of Theorem 1
Revised: B.3 Proof of Theorem 1

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Since S h is continuous and K is compact, the set
Revised: Since S h is continuous and K is compact, the set

Original: [Equation]
Revised: [Equation]

Original: is compact.
Revised: is compact.

Original: Therefore, there exist a set of representers φ 1 , φ 2 , · · · ∈ R such that
Revised: Therefore, there exist a set of representers φ 1 , φ 2 , · · · ∈ R such that

Original: [Equation]
Revised: [Equation]

Original: where R m = span { φ 1 , .
Revised: where R m = span { φ 1 , .

Original: φ m } .
Revised: φ m } .

Original: For any m ∈ N , let P m : U → R m denote a projection of U to
Revised: For any m ∈ N , let P m : U → R m denote a projection of U to

Original: R m .
Revised: R m .

Original: Since R is compact, the set
Revised: Since R is compact, the set

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: R ) (cid:33) is compact [29, Lemma 14].
Revised: R ) (cid:33) is compact [31, Lemma 14].

Original: Since S h is locally Lipschitz and P is compact, there exists a constant
Revised: Since S h is locally Lipschitz and P is compact, there exists a constant

Original: C = C ( P ) > 

Revised: C = C ( P ) >

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, assume C ̸ = 1 and define
Revised: Without loss of generality, assume C ̸ = 1 and define

Original: [Equation]
Revised: [Equation]

Original: By the universal approximation theorem for neural operators [29, Theorem 4] or [52, Theorem 2.5],there exists a neural operator ˆ S h : U → U such that
Revised: By the universal approximation theorem for neural operators [31, Theorem 4] or [54, Theorem 2.5],there exists a neural operator ˆ S h : U → U such that

Original: [Equation]
Revised: [Equation]

Original: Perusal of the proof of the universal approximation theorem for neural operators shows that ˆ S h canbe chosen so that ˆ S h ( P ) ⊆
Revised: Perusal of the proof of the universal approximation theorem for neural operators shows that ˆ S h canbe chosen so that ˆ S h ( P ) ⊆

Original: R m for some m ∈ N large enough, therefore ˆ S h ( P ) ⊆
Revised: R m for some m ∈ N large enough, therefore ˆ S h ( P ) ⊆

Original: P .
Revised: P .

Original: Let u 0 ∈ K ,then the triangle inequality implies
Revised: Let u 0 ∈ K ,then the triangle inequality implies

Original: [Equation]
Revised: [Equation]

Original: By the discrete time Grönwall lemma,
Revised: By the discrete time Grönwall lemma,

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: C Discussion and future work
Revised: C Discussion and future work

Original: In this work, we learn MNO from only local data and compose it to obtain the global attractor ofchaotic systems, and by explicitly enforcing dissipativity, we empirically show that MNO predictionsdo not blow up or collapse even in the long-time horizon, while still achieving relatively low errorrunning several orders of magnitude faster than traditional methods.
Revised: In this work, we learn MNO from only local data and compose it to obtain the global attractor ofchaotic systems, and by explicitly enforcing dissipativity, we empirically show that MNO predictionsdo not blow up or collapse even in the long-time horizon, while still achieving relatively low errorrunning several orders of magnitude faster than traditional methods.

Original: Furthermore, the MNO shows great potential for modeling partially observed systems or studyingsystems that exhibit bifurcations, both of which are of great interest for engineering applications.
Revised: Furthermore, the MNO shows great potential for modeling partially observed systems or studyingsystems that exhibit bifurcations, both of which are of great interest for engineering applications.

Revised: Thestudy of non-ergodic dissipative systems and their basins of attraction is also an important directionfor future study.

Original: This work provides a method for fast computation with applications to many scientific computingproblems.
Revised: This work provides a method for fast computation with applications to many scientific computingproblems.

Original: Our methods have two main long-term impacts beyond the immediate interests of thescientific computing community.
Revised: Our methods have two main long-term impacts beyond the immediate interests of thescientific computing community.

Original: Since they are orders of magnitude faster than the traditional solverscurrently employed in supercomputers, edge devices, and servers, the deployment of our methodssignificantly reduces the carbon footprint caused by scientific studies.
Revised: Since they are orders of magnitude faster than the traditional solverscurrently employed in supercomputers, edge devices, and servers, the deployment of our methodssignificantly reduces the carbon footprint caused by scientific studies.

Original: Furthermore, the proposedmethods are extremely flexible.
Revised: Furthermore, the proposedmethods are extremely flexible.

Original: Off-the-shelf usage of our methods allows scientists from a varietyof disciplines, e.g. chemistry, biology, ecology, epidemiology, physics, and applied mathematics, todeploy them on their complex systems without the need to build elaborate numerical methods.
Revised: Off-the-shelf usage of our methods allows scientists from a varietyof disciplines, e.g. chemistry, biology, ecology, epidemiology, physics, and applied mathematics, todeploy them on their complex systems without the need to build elaborate numerical methods.

Original: MNO has two major limitations.
Revised: MNO has two major limitations.

Original: First, it assumes the target system is approximately Markovian.
Revised: First, it assumes the target system is approximately Markovian.

Original: Ifthe system is heavily path-dependent, then the MNO framework does not directly apply.
Revised: Ifthe system is heavily path-dependent, then the MNO framework does not directly apply.

Original: Second,although we develop an approximation theorem for finite period, it does not hold for infinite timehorizon.
Revised: Second,although we develop an approximation theorem for finite period, it does not hold for infinite timehorizon.

Original: As discussed previously, it is infeasible to track the exact trajectory of chaotic systems on an infinitetime horizon.
Revised: As discussed previously, it is infeasible to track the exact trajectory of chaotic systems on an infinitetime horizon.

Original: Even very small errors will accumulate in each step, and eventually cause the simulation to diverge from the true trajectory.
Revised: Even very small errors will accumulate in each step, and eventually cause the simulationto diverge from the true trajectory.

Original: However, it is possible to track the attractor of the system.
Revised: However, it is possible to track the attractor of the system.

Original: Anattractor is absorbing.
Revised: Anattractor is absorbing.

Original: If the simulated trajectory only makes a small error, the attractor will absorbit back, so that the simulated trajectory will never diverge from the true attractor.
Revised: If the simulated trajectory only makes a small error, the attractor will absorbit back, so that the simulated trajectory will never diverge from the true attractor.

Original: Therefore, it ispossible to have the simulated trajectory capture the true attractor.
Revised: Therefore, it ispossible to have the simulated trajectory capture the true attractor.

Original: To obtain an infinite-time approximation error bound is non-trivial.
Revised: To obtain an infinite-time approximation error bound is non-trivial.

Original: Previously, [14, 15] (cf. Theorem3.12) show a result for (finite-dimensional) ODE systems.
Revised: Previously, [14, 15] (cf. Theorem3.12) show a result for (finite-dimensional) ODE systems.

Original: If the system is Lipschitz then there existsa numerical simulation that forms a dissipative dynamical system that does not blow up or collapse.
Revised: If the system is Lipschitz then there existsa numerical simulation that forms a dissipative dynamical system that does not blow up or collapse.

Original: And the the simulated attractor A h approximates the true attractor A with the time step hdist ( A h , A ) → 0 , as h →
Revised: And the the simulated attractor A h approximates the true attractor A with the time step hdist ( A h , A ) → 0 , as h →

Original: To generalize such theorem to Markov neural operator (MNO), we need to overcome two difficulties(1) generalize the formulation from (finite-dimensional) ODE systems to (infinite-dimensional) PDEsystems, and (2) show MNO can obtain a sufficient error rate with respect to the time step h .
Revised: To generalize such theorem to Markov neural operator (MNO), we need to overcome two difficulties(1) generalize the formulation from (finite-dimensional) ODE systems to (infinite-dimensional) PDEsystems, and (2) show MNO can obtain a sufficient error rate with respect to the time step h .

Original: The first aspect requires extending the theory from finite dimension to infinite dimension, whichis non-trivial since the operator F in (1) is not compact or bounded.
Revised: The first aspect requires extending the theory from finite dimension to infinite dimension, whichis non-trivial since the operator F in (1) is not compact or bounded.

Original: This makes it hard to boundthe error with respect to the attractor [53].
Revised: This makes it hard to boundthe error with respect to the attractor [55].

Original: The second aspect requires to formulate MNO slightlydifferently.
Revised: The second aspect requires to formulate MNO slightlydifferently.

Original: In the current formulation, the evolution operator is chosen for a fixed time step h .
Revised: In the current formulation, the evolution operator is chosen for a fixed time step h .

Original: Toachieve O ( h ) error we need to formulate the evolution operator continuously for infinitesimal h .
Revised: Toachieve O ( h ) error we need to formulate the evolution operator continuously for infinitesimal h .

Original: Especially, for a semi-linear PDE system
Revised: Especially, for a semi-linear PDE system

Original: [Equation]
Revised: [Equation]

Original: where A is a linear, self-adjoint operator and F is a continuous but nonlinear operator (this formulationincludes the KS and NS equations).
Revised: where A is a linear, self-adjoint operator and F is a continuous but nonlinear operator (this formulationincludes the KS and NS equations).

Original: The evolution can be written as
Revised: The evolution can be written as

Original: [Equation]
Revised: [Equation]

Original: Where Φ( u ( t ) , A, t ) := (cid:82) h 0 e − A ( h − s ) F ( u ( s ))
Revised: Where Φ( u ( t ) , A, t ) := (cid:82) h 0 e − A ( h − s ) F ( u ( s ))

Original: ds is bounded despite F is not.
Revised: ds is bounded despite F is not.

Original: If one can approximateΦ( u ( t ) , A, h ) by a neural operator, then MNO can potentially achieve the needed error rate.
Revised: If one can approximateΦ( u ( t ) , A, h ) by a neural operator, then MNO can potentially achieve the needed error rate.

Original: Thisshows hope to obtain an approximation error bound for infinite time zero.
Revised: Thisshows hope to obtain an approximation error bound for infinite time zero.

Original: We leave this as a promisingfuture direction.
Revised: We leave this as a promisingfuture direction.