Original: L EARNING PDE S OLUTION O PERATOR FOR C ONTINUOUS M ODELING OF T IME -S ERIES
Revised: L EARNING PDE S OLUTION O PERATOR FOR C ONTINUOUS M ODELING OF T IME -S ERIES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning underlying dynamics from data is important and challenging in many realworld scenarios.
Revised: Learning underlying dynamics from data is important and challenging in many realworld scenarios.

Original: Incorporating differential equations (DEs) to design continuous networks has drawn much attention recently, the most prominent of which is
Revised: Incorporating differential equations (DEs) to design continuous networks has drawn much attention recently, the most prominent of which is

Original: Neural ODE.
Revised: Neural ODE.

Original: Most prior works make specific assumptions on the type of DEs or restrict them to first or second-order DEs, making the model specialized for certain problems.
Revised: Most prior works make specific assumptions on the type of DEs or restrict them to first or second-order DEs, making the model specialized for certain problems.

Original: Furthermore, due to the use of numerical integration, they suffer from computational expensiveness and numerical instability.
Revised: Furthermore, due to the use of numerical integration, they suffer from computational expensiveness and numerical instability.

Original: Building upon recent
Revised: Building upon recent

Original: Fourier neural operator (FNO), this work proposes a partial differential equation (PDE) based framework which improves the dynamics modeling capability and circumvents the need for costly numerical integration.
Revised: Fourier neural operator (FNO), this work proposes a partial differential equation (PDE) based framework which improves the dynamics modeling capability and circumvents the need for costly numerical integration.

Original: FNO is hard to be directly applied to real applications because it is mainly confined to physical PDE problems.
Revised: FNO is hard to be directly applied to real applications because it is mainly confined to physical PDE problems.

Original: To fill this void, we propose a continuous-in-time FNO to deal with irregularlysampled time series and provide a theoretical result demonstrating its universality.
Revised: To fill this void, we propose a continuous-in-time FNO to deal with irregularlysampled time series and provide a theoretical result demonstrating its universality.

Original: Moreover, we reveal an intrinsic property of PDEs that increases the stability of the model.
Revised: Moreover, we reveal an intrinsic property of PDEs that increases the stability of the model.

Original: Several numerical evidence shows that our method represents a broader range of problems, including synthetic, image classification, and irregular timeseries.
Revised: Several numerical evidence shows that our method represents a broader range of problems, including synthetic, image classification, and irregular timeseries.

Original: Our framework opens up a new way for a continuous representation of neural networks that can be readily adopted for real-world applications.
Revised: Our framework opens up a new way for a continuous representation of neural networks that can be readily adopted for real-world applications.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The modeling of time-series data plays an important role in various applications in our everyday lives including climate forecasting (Schneider, 2001; Mudelsee, 2019), medical sciences (Stoffer & Ombao, 2012; Jensen et al., 2014), and finance (Chatigny et al., 2020; Andersen et al., 2005).
Revised: The modeling of time-series data plays an important role in various applications in our everyday lives including climate forecasting (Schneider, 2001; Mudelsee, 2019), medical sciences (Stoffer & Ombao, 2012; Jensen et al., 2014), and finance (Chatigny et al., 2020; Andersen et al., 2005).

Original: Numerous deep learning architectures (Connor et al., 1994; Hochreiter & Schmidhuber, 1997; Cho et al., 2014) have been developed to learn sequential patterns from diverse time-series datasets.
Revised: Numerous deep learning architectures (Connor et al., 1994; Hochreiter & Schmidhuber, 1997; Cho et al., 2014) have been developed to learn sequential patterns from diverse time-series datasets.

Original: In recent years, leveraging differential equations to design continuous networks has attracted increasing attention, first sparked by neural ordinary differential equations (ODEs) (Chen et al., 2018).
Revised: In recent years, leveraging differential equations to design continuous networks has attracted increasing attention, first sparked by neural ordinary differential equations (ODEs) (Chen et al., 2018).

Original: Differential equations that characterize the rates of change and interaction of continuously varying quantities have become indispensable mathematical language to describe time-evolving real-world phenomena (Cannon &
Revised: Differential equations that characterize the rates of change and interaction of continuously varying quantities have become indispensable mathematical language to describe time-evolving real-world phenomena (Cannon &

Original: Dostrovsky, 2012; Sundén & Fu, 2016; Black & Scholes, 2019).
Revised: Dostrovsky, 2012; Sundén & Fu, 2016; Black & Scholes, 2019).

Original: By virtue of their ability to represent and predict the world around us, incorporaFdting differential equations into neural networks has reinvigorated research in continuous deep learning, offering new theoretical perspectives on neural networks.
Revised: By virtue of their ability to represent and predict the world around us, incorporaFdting differential equations into neural networks has reinvigorated research in continuous deep learning, offering new theoretical perspectives on neural networks.

Original: Moreover, they provide memory efficiency, invertibility, and the ability to handle irregular time-series data (Rubanova et al., 2019;
Revised: Moreover, they provide memory efficiency, invertibility, and the ability to handle irregular time-series data (Rubanova et al., 2019;

Original: Chen et al., 2019; Dong et al., 2020).
Revised: Chen et al., 2019; Dong et al., 2020).

Original: Despite their eminent success, Neural ODEs have yet to be successfully applied to complex and large-scale tasks due to the limitation of expressiveness of ODEs.
Revised: Despite their eminent success, Neural ODEs have yet to be successfully applied to complex and large-scale tasks due to the limitation of expressiveness of ODEs.

Original: To respond to this limitation, there have been several  attempts to introduce more diverse differential equations, such as controlled differential equations (Kidger et al., 2020), delay differential equations (Zhu et al., 2020; Anumasa & PK, 2021), and integro-differential equations (Zappala et al., 2022).
Revised: To respond to this limitation, there are several works that enhance the expressiveness of Neural ODEs (Gholami et al., 2019; Gu et al., 2021). Another line of works attempts to introduce more diverse differential equations, such as controlled differential equations (Kidger et al., 2020), delay differential equations (Zhu et al., 2020; Anumasa & PK, 2021), and integro-differential equations (Zappala et al., 2022).

Original: In real applications, however, we usually know little about the underlying dynamics of the time evolution system.
Revised: In real applications, however, we usually know little about the underlying dynamics of the time evolution system.

Original: In general, we are hard to knowledge about how the temporal states evolve, which kind of differential equation it follows, how variables depend on each other, and how high derivatives it contains.
Revised: In general, we are hard to knowledge about how the temporal states evolve, which kind of differential equation it follows, how variables depend on each other, and how high derivatives it contains.

Original: Therefore, it is necessary to develop a model that can learn an extended class of differential equations that is able to cover more diverse applications, in a data-driven manner (Holt et al., 2022).
Revised: Therefore, it is necessary to develop a model that can learn an extended class of differential equations that is able to cover more diverse applications, in a data-driven manner (Holt et al., 2022).

Original: In this work, we propose a partial differential equation (PDE) based novel framework that can learn a broad range of dynamics without prior knowledge of governing equations.
Revised: In this work, we propose a partial differential equation (PDE) based novel framework that can learn a broad range of dynamics without prior knowledge of governing equations.

Original: PDEs that enjoy relations between the various partial derivatives of multivariable states represent much general dynamics, including ODEs as a special case.
Revised: PDEs that enjoy relations between the various partial derivatives of multivariable states represent much general dynamics, including ODEs as a special case.

Original: There have been several attempts to design neural networks through the lens of PDEs (Eliasof et al., 2021; Ruthotto & Haber, 2020; Ben-Yair et al., 2021;
Revised: There have been several attempts to design neural networks through the lens of PDEs (Eliasof et al., 2021; Ruthotto & Haber, 2020; Ben-Yair et al., 2021;

Original: Sun et al., 2020; Kim et al., 2020b).
Revised: Sun et al., 2020; Kim et al., 2020b).

Original: Most of the prior works have been designed under specific assumptions about the type or structure of the PDEs.
Revised: Most of the prior works have been designed under specific assumptions about the type or structure of the PDEs.

Original: As the underlying dynamics are unknown in real-world data, however, it should be oblivious to the knowledge of the underlying PDE structure and needs to be learned from data.
Revised: As the underlying dynamics are unknown in real-world data, however, it should be oblivious to the knowledge of the underlying PDE structure and needs to be learned from data.

Original: Moreover, because the appropriate properties of PDE differ for each given problem (Eliasof et al., 2021), it is necessary to be able to represent a wide range of PDEs as possible.
Revised: Moreover, because the appropriate properties of PDE differ for each given problem (Eliasof et al., 2021), it is necessary to be able to represent a wide range of PDEs as possible.

Original: To this end, we adopt Fourier neural operator (FNO) (Li et al., 2021), an emerging PDE solver that directly parametrizes the  solution operator without prior information on the governing PDE.
Revised: To this end, we adopt Fourier neural operator (FNO) (Li et al., 2021), an emerging model that directly parametrizes the PDE solution operator without prior information on the governing PDE.

Original: By learning the solution operator, the model automatically learns diverse equations in a completely data-driven way, as well as attains the solution in a single call without using numerical integration that invokes computational expensiveness and numerical instability.
Revised: By learning the solution operator, the model automatically learns diverse equations in a completely data-driven way, as well as attains the solution in a single call without using numerical integration that invokes computational expensiveness and numerical instability.

Original: Because FNO has been scrutinized on mathematically well-posed PDE problems, adapting FNO to practical applications confronts several drawbacks.
Revised: Because FNO has been scrutinized on mathematically well-posed PDE problems, adapting FNO to practical applications confronts several drawbacks.

Original: First, due to its notion of discrete time, FNO is difficult to directly transfer to irregularly-sampled time-series commonly arising in real-world problems.
Revised: First, due to its notion of discrete time, FNO is difficult to directly transfer to irregularly-sampled time-series commonly arising in real-world problems.

Original: To render it more suitable for continuous time-series, we propose a continuous-in-time
Revised: To render it more suitable for continuous time-series, we propose a continuous-in-time

Original: FNO, termed CTFNO , that can be evaluated at arbitrary time points.
Revised: FNO, termed CTFNO , that can be evaluated at arbitrary time points.

Original: By learning continuous-time solution operator, CTFNO can flexibly capture diverse time-series data.
Revised: By learning continuous-time solution operator, CTFNO can flexibly capture diverse time-series data.

Original: Moreover, we demonstrate the representational power of CTFNO via a rigorous theoretical proof of the universal approximation theorem.
Revised: Moreover, we demonstrate the representational power of CTFNO via a rigorous theoretical proof of the universal approximation theorem.

Original: Secondly, we develop a network architecture that guarantees stability and leads to well-posed learning problems.
Revised: Secondly, we develop a network architecture that guarantees stability and leads to well-posed learning problems.

Original: By ensuring the stability of the model, it can defend against noisy observations and alleviate overfitting.
Revised: By ensuring the stability of the model , it can defend against noisy observations and alleviate overfitting.

Original: We also verify that it enhances the adversarial robustness for image classification.
Revised: We also verify that it enhances the adversarial robustness for image classification.

Original: The results of various experiments show that our model provides superior performance on wide array of real-world data with applications in time-series and image classification
Revised: The results of various experiments show that our model provides superior performance on wide array of real-world data with applications in time-series and image classification

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Fourier Neural Operator
Revised: Fourier Neural Operator

Original: In contrast to conventional neural networks, in which maps are learned between finite-dimensional vector spaces, Fourier neural operator (FNO) (Li et al., 2021) learns a map G :
Revised: In contrast to conventional neural networks, in which maps are learned between finite-dimensional vector spaces, Fourier neural operator (FNO) (Li et al., 2021) learns a map G :

Original: A → U between infinite-dimensional function spaces A and U , with a special focus on
Revised: A → U between infinite-dimensional function spaces A and U , with a special focus on

Original: PDEs.
Revised: PDEs.

Original: An input a ∈ A is any of source or initial functions, and u ∈ U is the corresponding solution.
Revised: An input a ∈ A is any of source or initial functions, and u ∈ U is the corresponding solution.

Original: The solution to fairly general PDEs is represented as an integral operator of Green’s function G in a bounded domain Ω ⊂ R n :
Revised: The solution to fairly general PDEs is represented as an integral operator of Green’s function G in a bounded domain Ω ⊂ R n , i.e, convolution of the input function a with the Green’s function G ;

Original: [Equation]
Revised: [Equation]

Original: This solution formula elucidates an elegant way to design FNO.
Revised: This solution formula elucidates an elegant way to design FNO.

Original: The overall computational flow of
Revised: The overall computational flow of

Original: FNO for approximating the integral operator (1) is given as a P −→ v 0 L 1 −→ v 1 L 2 −→ · · · L L −→ v L Q −→ u, for a given depth L .
Revised: FNO for approximating the integral operator (1) is given as a P −→ v 0 L 1 −→ v 1 L 2 −→ · · · L L −→ v L Q −→ u, for a given depth L .

Original: To recover the sorts of universality, the input function a is lifted to a higher dimensional representation v 0 ( x ) ∈ R d v by a Nemytskii operator P( a ) ( x ) = P ( a ( x ) , x ) .
Revised: To recover the sorts of universality, the input function a is lifted to a higher dimensional representation v 0 ( x ) ∈ R d v by a lifting operator P .

Original: Fourier layers L l iteratively construct functions v l as follows.
Revised: Fourier layers L l iteratively construct functions v l as follows.

Original: Under sufficient regularity conditions, the kernel G is square-integrable (Grüter & Widman, 1982; Stakgold & Holst, 2011; Dong & Kim, 2009).
Revised: Under sufficient regularity conditions, the kernel G is squareintegrable (Grüter & Widman, 1982; Stakgold & Holst, 2011; Dong & Kim, 2009).

Original: Then, the kernel convolution can be efficiently computed  in Fourier domain.
Revised: Then, the kernel convolution can be efficiently computed as a product of the Fourier transforms of the functions in Fourier domain.

Original: FNO formulates the integral kernel directly in Fourier space as follows:
Revised: FNO formulates the integral kernel directly in Fourier space as follows:

Original: [Equation]
Revised: [Equation]

Original:  where R θ ℓ ( ξ ) ∈ C d v × d v is a Fourier transformed kernel matrix indexed by frequency ξ , and F denotes Fourier transform.
Revised: ) ( x ) (cid:1) , where R θ ℓ ( ξ ) ∈ C d v × d v is a Fourier transformed kernel matrix indexed by frequency ξ , and F denotes Fourier transform.

Original: Finally,  Q condenses features to reconstruct u .
Revised: Finally, the projection operator Q condenses features to reconstruct u .

Original: Treatment of time-varying problems When applied to time-dependent PDEs, the vanilla FNO can only learn an operator that maps the initial function to a solution for a single fixed time.
Revised: Treatment of time-varying problems When applied to time-dependent PDEs, the vanilla FNO can only learn an operator that maps the initial function to a solution for a single fixed time.

Original: To deal with time-varying problems, two methods are suggested in (Li et al., 2021): (i) It poses the time-dependent problem as a sequence-to-sequence task.
Revised: To deal with time-varying problems, two methods are suggested in (Li et al., 2021): (i) It poses the time-dependent problem as a sequence-to-sequence task.

Original: (ii) Treats it as ( n + 1) − dimensional problem by adding one more dimension and apply FNO layers to convolve in space-time domain.
Revised: (ii) Treats it as ( n + 1) − dimensional problem by adding one more dimension and apply FNO layers to convolve in space-time domain.

Original: In this case, FNO can only predict the solution at times on a fixed equispaced temporal mesh.
Revised: In this case, FNO can only predict the solution at times on a fixed equispaced temporal mesh.

Original:  M ETHOD 3. M OTIVATION
Revised:  M ETHOD 3. M OTIVATION

Original: Starting with Neural ODEs, leveraging differential equations (DEs) has been found to be effective in modeling time-series data.
Revised: Starting with Neural ODEs, leveraging differential equations (DEs) has been found to be effective in modeling time-series data.

Original: They have shown promising results, however, they confront some limitations.
Revised: They have shown promising results, however, they confront some limitations.

Original: Their model architectures and inference schemes are specialized to the DEs on which they are based, such as first-order ODE for Neural ODE.
Revised: Their model architectures and inference schemes are specialized to the DEs on which they are based, such as first-order ODE for Neural ODE.

Original: These bespoke model structures rule out their generalization ability to other classes of DEs.
Revised: These bespoke model structures rule out their generalization ability to other classes of DEs.

Original: Consequently, they fail on many tasks close to real applications or large-scale datasets.
Revised: Consequently, they fail on many tasks close to real applications or large-scale datasets.

Original: This drawback is further exacerbated in real-world applications.
Revised: This drawback is further exacerbated in real-world applications.

Original: Several approaches have been proposed to improve upon it (Holt et al., 2022), but they still leave much to be desired.
Revised: Several approaches have been proposed to improve upon it (Holt et al., 2022), but they still leave much to be desired.

Original: This is confirmed by the experimental results in Figure 1 conducted on three synthetic examples proposed by (Biloš et al., 2021).
Revised: This is confirmed by the experimental results in Figure 1 conducted on three synthetic examples proposed by (Biloš et al., 2021).

Original: The results show that baselines fail to capture the true trajectories or produce a degraded solution.
Revised: The results show that baselines fail to capture the true trajectories or produce a degraded solution.

Original: Therefore, we need a model that can learn a broad range of dynamics to represent a wide variety of applications.
Revised: Therefore, we need a model that can learn a broad range of dynamics to represent a wide variety of applications.

Original: In this paper, we propose a novel framework for modeling complex systems through the lens of PDEs, a richer family of differential equations.
Revised: In this paper, we propose a novel framework for modeling complex systems through the lens of PDEs, a richer family of differential equations.

Original: Among recent advances in the field to learn PDE solvers, we harness FNO, which has shown promise ability to learn complex PDEs.
Revised: Among recent advances in the field to approximate PDEs, we harness FNO, which has shown promise ability to learn complex PDEs.

Original: In contrast to existing works that require the knowledge of the PDE a priori (Raissi et al., 2019; Long et al., 2019), FNO automatically learns the underlying dynamics from data.
Revised: In contrast to existing works that require the knowledge of the PDE a priori (Raissi et al., 2019; Long et al., 2019), FNO automatically learns the underlying dynamics from data.

Original: Because it can represent a wide range of dynamics in a data-driven manner, it is able to cover more diverse applications.
Revised: Because it can represent a wide range of dynamics in a data-driven manner, it is able to cover more diverse applications.

Original: The results depicted in Figure 1 demonstrates the ability of our model to represent more general temporal dynamics.
Revised: The results depicted in Figure 1 demonstrates the ability of our model to represent more general temporal dynamics.

Original: It can be seen that our model consistently achieves superior performance in all examples.
Revised: It can be seen that our model consistently achieves superior performance in all examples.

Original: Please consult Section 4.1 for additional results and experimental details.
Revised: Please consult Section 4.1 for additional results and experimental details.

Original: 3. C ONTINUOUS -T IME FNO
Revised: 3. C ONTINUOUS -T IME FNO

Original: In a wide variety of real-world problems, the system of interest evolves continuously over time.
Revised: In a wide variety of real-world problems, the system of interest evolves continuously over time.

Original: FNO has recently shown the promise ability to learn complex PDEs, however, the notion of discrete time of FNO hinders its wider applicability to time-varying systems.
Revised: FNO has recently shown the promise ability to learn complex PDEs, however, the notion of discrete time of FNO hinders its wider applicability to time-varying systems.

Original: To ameliorate this limitation, we introduce a continuous-in-time Fourier neural operator (CTFNO).
Revised: To ameliorate this limitation, we introduce a continuous-in-time Fourier neural operator (CTFNO).

Original: Consider a time-dependent PDE
Revised: Consider a time-dependent PDE

Original: [Equation]
Revised: [Equation]

Original: where Ω ⊂ R n and u : [0 , ∞ ) × Ω → R n is the solution with initial condition u 0 ( x ) at time t = 0 .
Revised: where Ω ⊂ R n and u : [0 , ∞ ) × Ω → R n is the solution with initial condition u 0 ( x ) at time t = 0 .

Original: The Green’s function formula for (2)
Revised: The Green’s function formula for (2)

Original: [Equation]
Revised: [Equation]

Original: shows that the weights of an FNO layer should be conditioned on time, to learn an operator ( t, u 0 ( x ))
Revised: shows that the weights of an FNO layer should be conditioned on time, to learn an operator ( t, u 0 ( x ))

Original: (cid:55)→ u ( t, x ) for an arbitrary time t and initial condition u 0 .
Revised: (cid:55)→ u ( t, x ) for an arbitrary time t and initial condition u 0 .

Original: For example, the Fourier transform of the Green’s kernel of the heat equation is given by R ( t, ξ ) =
Revised: For example, the Fourier transform of the Green’s kernel of the heat equation is given by R ( t, ξ ) =

Original: e − 4 ctξ 2 .
Revised: e − 4 ctξ 2 .

Original: To this end, we propose a time-aware Fourier layer that updates the solution based on time-dependent Green’s function as follows:
Revised: To this end, we propose a time-aware Fourier layer that updates the solution based on time-dependent Green’s function as follows:

Original: 𝐴𝐵𝑅𝑊𝜎
Revised: 𝐴𝐵𝑅𝑊𝜎

Original: F
Revised: F

Original: F − 𝑃 𝑢 0 (𝑥) 𝑠ℎ𝑎𝑟𝑖𝑛𝑔 𝑛𝑒𝑡 1 𝜑(𝑡) 𝜓(𝑡) 𝑠ℎ𝑎𝑟𝑖𝑛𝑔 𝑛𝑒𝑡𝑢(𝑡,𝑥) 𝑄 𝑅𝑊𝜎
Revised: F − 𝑃 𝑢 0 (𝑥) 𝑠ℎ𝑎𝑟𝑖𝑛𝑔 𝑛𝑒𝑡 1 𝜑(𝑡) 𝜓(𝑡) 𝑠ℎ𝑎𝑟𝑖𝑛𝑔 𝑛𝑒𝑡𝑢(𝑡,𝑥) 𝑄 𝑅𝑊𝜎

Original: F
Revised: F

Original: F − 𝐴𝐵𝑡 𝑣 0 (𝑥) 𝑣 1 (𝑡,𝑥) 𝑣 2 (𝑡,𝑥) 𝑣 3 (𝑡,𝑥) 𝑅𝑊𝜎
Revised: F − 𝐴𝐵𝑡 𝑣 0 (𝑥) 𝑣 1 (𝑡,𝑥) 𝑣 2 (𝑡,𝑥) 𝑣 3 (𝑡,𝑥) 𝑅𝑊𝜎

Original: F
Revised: F

Original: F − 𝐵𝐴
Revised: F − 𝐵𝐴

Original: [Equation]
Revised: [Equation]

Original: Time Modulating To equip the FNO network with the ability to capture information on the time of observations, a time-dependent layer is constructed as follows.
Revised: Time Modulating To equip the FNO network with the ability to capture information on the time of observations, a time-dependent layer is constructed as follows.

Original: Two sharing networks, parametrized by two-layer fully connected networks, first convert the input time t into multidimensional representations φ ( t ) , ψ ( t ) ∈ R c for a hidden dimension c .
Revised: Two sharing networks, parametrized by two-layer fully connected networks, first convert the input time t into multi-dimensional representations φ ( t ) , ψ ( t ) ∈ R c for a hidden dimension c by sinusoidal embeddings (Vaswani et al., 2017).

Original: For each Fourier layer L ℓ , affine transforms A ℓ and B ℓ produce temporal information φ ℓ ( t, ξ )
Revised: For each Fourier layer L ℓ , affine transforms A ℓ and B ℓ produce temporal information φ ℓ ( t, ξ )

Original: =
Revised: =

Original: A ℓ ( ξ ) φ ( t ) ∈ C and ψ ℓ ( t ) =
Revised: A ℓ ( ξ ) φ ( t ) ∈ C and ψ ℓ ( t ) =

Original: diag ( B ℓ ψ ( t ))
Revised: diag ( B ℓ ψ ( t ))

Original: ∈
Revised: ∈

Original: R d v × d v .
Revised: R d v × d v .

Original: Then, thesetwo time-dependent representations φ ℓ ( t, ξ ) and ψ ℓ ( t ) scale each weight in frequency domain R ℓ and in spatial domain W ℓ , respectively.
Revised: Then, these time-dependent representations φ ℓ ( t, ξ ) and ψ ℓ ( t ) scale each weight in frequency domain R ℓ and in spatial domain W ℓ , respectively.

Original: This is somewhat reminiscent of weight modulation in StyleGAN2 (Karras et al., 2020), but applied per-frequency in the Fourier domain and per-channel in the spatial domain.
Revised: This is somewhat reminiscent of weight modulation in StyleGAN2 (Karras et al., 2020), but applied per-frequency in the Fourier domain and per-channel in the spatial domain.

Original: [Table]
Revised: [Table]

Original: Ablation study for time-aware network We provideadditional insights into our model by investigating how useful the time modulating is for learning time-dependent PDEs.
Revised: Ablation study for time-aware network We provide insights into our model by investigating how useful the structure of the CTFNO designed based on the theoretical time-dependent Green’s formula (3) is for learning time-dependent PDEs.

Original: We study three alternative ways to equip the temporal information into the FNO network:
Revised: We study three alternative ways to equip the temporal information into the FNO network:

Original: Baseline 1: concatenating time t with an input function u 0 ( x ) .
Revised: Baseline 1: concatenating time t with an input function u 0 ( x ) .

Original: Baseline 2: concatenating encoded time φ 0 ( t ) with a lifted input function v 0 ( x ) .
Revised: Baseline 2: concatenating encoded time φ 0 ( t ) with a lifted input function v 0 ( x ) .

Original: Baseline 3: concatenating encoded times φ 0 ( t ) , φ 1 ( t ) , . .
Revised: Baseline 3: concatenating encoded times φ 0 ( t ) , φ 1 ( t ) , . .

Original: . , φ L ( t ) with intermediate features v 0 ( x ) , v 1 ( x ) , . .
Revised: . , φ L ( t ) with intermediate features v 0 ( x ) , v 1 ( x ) , . .

Original: . , v L ( x ) , respectively.
Revised: . , v L ( x ) , respectively.

Original: See Appendix D.1 for more details.
Revised: See Appendix D.1 for more details.

Original: we test the ability of these models to learn canonical linear and nonlinear PDEs: one-dimensional heat and Burgers’ equations.
Revised: we test the ability of these models to learn canonical linear and nonlinear PDEs: one-dimensional heat and Burgers’ equations.

Original: Table 1 show that, in all cases, the three ablation study models, which are structures that can continuously handle arbitrary time t , are similar to or worse than FNO2d (proposed in the original FNO paper) that can only evaluate the solution value at times on a specific grid.
Revised: Table 1 show that, in all cases, the three ablation study models, which are structures that can continuously handle arbitrary time t , are similar to or worse than FNO2d (proposed in the original FNO paper) that can only evaluate the solution value at times on a specific grid.

Original: However, CTFNO significantly outperforms other models.
Revised: However, CTFNO significantly outperforms other models.

Original: The rationale of the way of imposing temporal information on CTFNO is based on Green’s formula of time-dependent PDEs.
Revised: The rationale of the way of imposing temporal information on CTFNO is based on Green’s formula of time-dependent PDEs.

Original: The results validate that the  structure of CTFNO designed based on thetheoretical time-dependent Green’s formula is much more expressive in actually learning the PDE.
Revised: The results validate that the time modulating structure of CTFNO designed based on the time-dependent Green’s formula is much more expressive in actually learning the PDE.

Original: 3. U NIVERSAL A PPROXIMATION
Revised: 3. U NIVERSAL A PPROXIMATION

Original: The universality of FNO has been proven by Kovachki et al.
Revised: The universality of FNO has been proven by Kovachki et al.

Original: Based on this, we prove the universality of the proposed CTFNO, which is condensed in the following informal statement.
Revised: Based on this, we prove the universality of the proposed CTFNO, which is condensed in the following informal statement.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: (Informal) CTFNO can approximate any time-dependent continuous operator, of arbitrary accuracy.
Revised: (Informal) CTFNO can approximate any time-dependent continuous operator, of arbitrary accuracy.

Original: A formal statement and details of proof of Theorem 3.1 are provided in Appendix A.
Revised: A formal statement and details of proof of Theorem 3.1 are provided in Appendix A.

Original: 3. S TABILITY
Revised: 3. S TABILITY

Original: When we learn a model for the time evolution of dynamical systems, the learned dynamics should be guaranteed to be well-posed and the stability of the model on unseen trajectories should be ensured over a long duration.
Revised: When we learn a model for the time evolution of dynamical systems, the learned dynamics should be guaranteed to be well-posed and the stability of the model on unseen trajectories should be ensured over a long duration.

Original: This is important because the learned system can be unstable when using a generic neural network (Szegedy et al., 2013; Moosavi-Dezfooli et al., 2017).
Revised: This is important because the learned system can be unstable when using a generic neural network (Szegedy et al., 2013; Moosavi-Dezfooli et al., 2017).

Original: In this case, the resultant output of the learned dynamics may not be continuous with respect to its input.
Revised: In this case, the resultant output of the learned dynamics may not be continuous with respect to its input.

Original: Also, such unstable networks are vulnerable to adversarial attacks, overfitting, and unauthorized exploitation, which may render the network useless in practice.
Revised: Also, such unstable networks are vulnerable to adversarial attacks, overfitting, and unauthorized exploitation, which may render the network useless in practice.

Original: Therefore, stability is a necessary condition in real-world applications.
Revised: Therefore, stability is a necessary condition in real-world applications.

Original: The stability of a model is measured by the sensitivity of the prediction with respect to small perturbations of the inputs.
Revised: The stability of a model is measured by the sensitivity of the prediction with respect to small perturbations of the inputs.

Original: A formal definition is given as follows.
Revised: A formal definition is given as follows.

Original: Definition 3.1.
Revised: Definition 3.1.

Original: (Stability (Hadamard, 1902))
Revised: (Stability (Hadamard, 1902))

Original: A PDE (2) is said to be stable if for any solution u ( t, x ) with initial condition u 0 ( x ) and ϵ > 0 , there exists δ > 0 such that for all new initial function ˜ u 0 ( x ) satisfying ∥ ˜ u 0 − u 

Revised: A PDE (2) is said to be stable if for any solution u ( t, x ) with initial condition u 0 ( x ) and ϵ > 0 , there exists δ > 0 such that for all new initial function ˜ u 0 ( x ) satisfying ∥ ˜ u 0 − u

Original: ∥ < δ , the corresponding solution ˜ u ( t, x ) satisfies ∥
Revised: ∥ < δ , the corresponding solution ˜ u ( t, x ) satisfies ∥

Original: ˜
Revised: ˜

Original: u − u ∥ < ϵ for all t ≥ 0 .
Revised: u − u ∥ < ϵ for all t ≥ 0 .

Original: The stability is a hard constraint imposed upon the model.
Revised: The stability is a hard constraint imposed upon the model.

Original: While some recent studies have begun to address the stability of neural network architectures, it has typically been used as a soft constraint by adding an extra regularization loss term on the training data (Moosavi-Dezfooli et al., 2019), or required computation of the eigenvalues of the Jacobian matrix (Ross & Doshi-Velez, 2018; Hoffman et al., 2019).
Revised: While some recent studies have begun to address the stability of neural network architectures, it has typically been used as a soft constraint by adding an extra regularization loss term on the training data (Moosavi-Dezfooli et al., 2019), or required computation of the eigenvalues of the Jacobian matrix (Ross & Doshi-Velez, 2018; Hoffman et al., 2019).

Original: Because the stability of the Neural ODE corresponds to the stability of its underlying
Revised: Because the stability of the Neural ODE corresponds to the stability of its underlying

Original: ODE, it allows the use of stable discretization techniques of the ODEs (Haber & Ruthotto, 2017; Yan et al., 2020).
Revised: ODE, it allows the use of stable discretization techniques of the ODEs (Haber & Ruthotto, 2017; Yan et al., 2020).

Original: Similarly, the stability of CTFNO is connected to the well-posedness of the learned PDE, which is the PDE whose solution operator is the operator learned by CTFNO.
Revised: Similarly, the stability of CTFNO is connected to the well-posedness of the learned PDE, which is the PDE whose solution operator is the operator learned by CTFNO.

Original: The kernel formulation (3) sheds light on a way to ensure stability.
Revised: The kernel formulation (3) sheds light on a way to ensure stability.

Original: A proof is deferred to appendix B.
Revised: A proof is deferred to appendix B.

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: (Stability of CTFNO)
Revised: (Stability of CTFNO)

Original: If ∥ R ( t, ξ ) ∥ 2 and ∥ W ( t ) ∥ 2 are bounded for every t > 0 , then the corresponding CTFNO with a Lipschitz continuous activation function is stable.
Revised: If ∥ R ( t, ξ ) ∥ 2 and ∥ W ( t ) ∥ 2 are bounded for every t > 0 , then the corresponding CTFNO with a Lipschitz continuous activation function is stable.

Original: A proof is deferred to appendix B. As given in proposition 3.2, if the Fourier kernel and weight have bounded L 2 norm, the global stability of CTFNO is guaranteed.
Revised: A proof is deferred to appendix B. As given in proposition 3.2, if the Fourier kernel and weight have bounded L 2 norm, the global stability of CTFNO is guaranteed.

Original: Because of its expensive computational cost, however, the L 2 norm of the matrix is not useful in practice.
Revised: Because of its expensive computational cost, however, the L 2 norm of the matrix is not useful in practice.

Original: In the following, we suggest a practical method for enforcing stability conditions.
Revised: In the following, we suggest a practical method for enforcing stability conditions.

Original: Gershgorin discs normalization Gershgorin’s circle theorem (Varga, 2010) allows us to make fast deductions on the bound of eigenvalues.
Revised: Gershgorin discs normalization Gershgorin’s circle theorem (Varga, 2010) allows us to make fast deductions on the bound of eigenvalues.

Original: It states that every eigenvalue of a square matrix A lies within the union of discs centered at diagonal elements with a radius of the sum of the magnitudes of the off-diagonal entries from the same row.
Revised: It states that every eigenvalue of a square matrix A lies within the union of discs centered at diagonal elements with a radius of the sum of the magnitudes of the off-diagonal entries from the same row.

Original: Therefore, as we normalize the L 1 norm of each row, we can impose the requisite stability.
Revised: Therefore, as we normalize the L 1 norm of each row, we can impose the requisite stability.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we empirically validate the applicability of the proposed model to study a variety of synthetic, real-world time series and image classification problems.
Revised: In this section, we empirically validate the applicability of the proposed model to study a variety of synthetic, real-world time series and image classification problems.

Original: In all experiments, we build models with comparable network sizes to make a fair comparison.
Revised: In all experiments, we build models with comparable network sizes to make a fair comparison.

Original: Precise implementation details can be found in Appendix C.
Revised: Precise implementation details can be found in Appendix C.

Original: 4. S YNTHETIC DATA
Revised: 4. S YNTHETIC DATA

Original: In this section, we use several illustrative examples to demonstrate the outstanding capacity of the proposed method in learning diverse class of dynamics.
Revised: In this section, we use several illustrative examples to demonstrate the outstanding capacity of the proposed method in learning diverse class of dynamics.

Original: We evaluate the performance of CTFNO in comparison with several related lines of work on a broad range of differential equations:
Revised: We evaluate the performance of CTFNO in comparison with several related lines of work on a broad range of differential equations:

Original: Datasets We compare the performance of CTFNO on a broad range of synthetic data.
Revised: Datasets We compare the performance of CTFNO on a broad range of synthetic data.

Original: Mathematical formulation of these dynamics can be found in Appendix C.1.
Revised: Mathematical formulation of these dynamics can be found in Appendix C.1.

Original: • Stiff ODE (Holt et al., 2022) is a typical example that Neural ODEs fail to learn.
Revised: • Stiff ODE (Holt et al., 2022) is a typical example that Neural ODEs fail to learn.

Original: • Spiral ODE is a two-dimensional system of nonlinear ODEs, commonly arising in biological systems.
Revised: • Spiral ODE is a two-dimensional system of nonlinear ODEs, commonly arising in biological systems.

Original: The dynamics describe spiral shaped trajectories.
Revised: The dynamics describe spiral shaped trajectories.

Original: [Table]
Revised: [Table]

Original: • Sawtooth and Square (Biloš et al., 2021) are considered to evaluate the capability on modeling waveform signals.
Revised: • Sawtooth and Square (Biloš et al., 2021) are considered to evaluate the capability on modeling waveform signals.

Original: These are piecewise differentiable having cusps.
Revised: These are piecewise differentiable having cusps.

Original: • Reaction ODE is commonly used to model chemical reactions and is of the form ∂u/∂t = 6 u (1 − u ) .
Revised: • Reaction ODE is commonly used to model chemical reactions and is of the form ∂u/∂t = 6 u (1 − u ) .

Original: Unlike to aforementioned ODEs, a solution to Reaction ODE is regarded as a function and in implementation it can be regarded as a high-dimensional ODE.
Revised: Unlike to aforementioned ODEs, a solution to Reaction ODE is regarded as a function and in implementation it can be regarded as a high-dimensional ODE.

Original: Heat
Revised: Heat

Original: (Baron Fourier, 1878) and Burgers’ equations (Bateman, 1915) are canonical onedimensional linear and nonlinear PDEs.
Revised: (Baron Fourier, 1878) and Burgers’ equations (Bateman, 1915) are canonical onedimensional linear and nonlinear PDEs.

Original: They take the form
Revised: They take the form

Original: [Equation]
Revised: [Equation]

Original: with the corner cases: heat α = 0 and Burgers’ equation α = 
Revised: with the corner cases: heat α = 0 and Burgers’ equation α = 1 .

Revised: Given an initial function u 0 , we train models to learn ( t, u 0 ( x ))

Revised: (cid:55)→

Original: Baselines We evaluate the performance of our model in comparison with several related lines of works: the standard Neural ODE (NODE)
Revised: Baselines We evaluate the performance of our model in comparison with several related lines of works: the standard Neural ODE (NODE)

Original: (Chen et al., 2018), ANODE (Dupont et al., 2019), and
Revised: (Chen et al., 2018), ANODE (Dupont et al., 2019), and

Original: Neural Flow (Biloš et al., 2021), which directly parametrizes the solution operator of an ODE, are adopted for ODE-based models.
Revised: Neural Flow (Biloš et al., 2021), which directly parametrizes the solution operator of an ODE, are adopted for ODE-based models.

Original: We also consider with Neural Laplace (Holt et al., 2022) which can represent diverse classes of equations by modeling them in the Laplace domain.
Revised: We also consider with Neural Laplace (Holt et al., 2022) which can represent diverse classes of equations by modeling them in the Laplace domain.

Original: Furthermore, we compare with representative PDE-based models; DeepONet (DON) (Lu et al., 2019a), an alternative operator learning method, POD-DeepONet (PDN)
Revised: Furthermore, we compare with representative PDE-based models; DeepONet (DON) (Lu et al., 2019a), an alternative operator learning method, POD-DeepONet (PDN)

Original: (Lu et al., 2022), a functional PCA-based model, and FNO2d (Li et al., 2021), a Fourier neural operator with spatio-temporal inputs.
Revised: (Lu et al., 2022), a functional PCA-based model, and FNO2d (Li et al., 2021), a Fourier neural operator with spatio-temporal inputs.

Original: Results Discussion
Revised: Results Discussion

Original: The overall results of root mean squared error (RMSE) are reported in Table 2.
Revised: The overall results of root mean squared error (RMSE) are reported in Table 2.

Original: The results show that our CTFNO substantially achieves conspicuous success all baseline models by a large margin on all datasets.
Revised: The results show that our CTFNO substantially achieves conspicuous success all baseline models by a large margin on all datasets.

Original: We can see that the performance difference of baselines on different datasets varies a lot.
Revised: We can see that the performance difference of baselines on different datasets varies a lot.

Original: An ODE model that performs well on a certain data fails to learn the dynamics of another data.
Revised: An ODE model that performs well on a certain data fails to learn the dynamics of another data.

Original: But, Neural Laplace, which can express wider dynamics than ODE-based models, performs better than the three ODE-based models (NODE, ANODE, and Neural Flow) throughout all datasets.
Revised: But, Neural Laplace, which can express wider dynamics than ODE-based models, performs better than the three ODE-based models (NODE, ANODE, and Neural Flow) throughout all datasets.

Original: Further, our model, which can describe a broad family of DEs, consistently approximate all synthetic dynamics much better.
Revised: Further, our model, which can describe a broad family of DEs, consistently approximate all synthetic dynamics much better.

Original: These results demonstrate how important the range of expressible dynamics of the model is.
Revised: These results demonstrate how important the range of expressible dynamics of the model is.

Original: Figure 1 provides pictorial results of the learned solutions to Spiral, Square, and Sawtooth data, and further qualitative comparisons are included in Appendix D.5.
Revised: Figure 1 provides pictorial results of the learned solutions to Spiral, Square, and Sawtooth data, and further qualitative comparisons are included in Appendix D.5.

Original: Table 2 also contain the comparison to the original FNO2d.
Revised: Table 2 also contain the comparison to the original FNO2d.

Original: Comparison results of CTFNO with the original FNO demonstrate that the use of the proposed time-dependent structure significantly improves the capacity of the model to handle time.
Revised: Comparison results of CTFNO with the original FNO demonstrate that the use of the proposed time-dependent structure significantly improves the capacity of the model to handle time.

Original: Moreover, our model is superior to existing benchmark PDE models.
Revised: Moreover, our model is superior to existing benchmark PDE models.

Original: The overall results confirm that CTFNO is amenable to learn a wide array of dynamics.
Revised: The overall results confirm that CTFNO is amenable to learn a wide array of dynamics.

Original: 4. R EAL TIME - SERIES APPLICATIONS
Revised: 4. R EAL TIME - SERIES APPLICATIONS

Original: In this section, we investigate the performance of our method on interpolation and prediction tasks using three real-world time-series datasets, including partially observed, multi-variate sequences.
Revised: In this section, we investigate the performance of our method on interpolation and prediction tasks using three real-world time-series datasets, including partially observed, multi-variate sequences.

Original: All models were run three times with different random seeds and we report the averaged value.
Revised: All models were run three times with different random seeds and we report the averaged value.

Original: Results show that the CTFNO performs outperforms a range of baselines.
Revised: Results show that the CTFNO performs outperforms a range of baselines.

Original: Datasets We evaluate our model on MuJoCo hopper, PhyisioNet 2012, and Human Activity datasets, as in Rubanova et al.
Revised: Datasets We evaluate our model on MuJoCo hopper, PhyisioNet 2012, and Human Activity datasets, as in Rubanova et al.

Original: For more details, see Appendix C.1.
Revised: For more details, see Appendix C.1.

Original: [Table]
Revised: [Table]

Original: • MuJoCo (Tassa et al., 2018)
Revised: • MuJoCo (Tassa et al., 2018)

Original: We consider the hopper physics environments from the Deepmind Control Suite.
Revised: We consider the hopper physics environments from the Deepmind Control Suite.

Original: Data records 14-dimensional attributes, including state and action, with 100 timestamps.
Revised: Data records 14-dimensional attributes, including state and action, with 100 timestamps.

Original: To deal with partial observations, we conduct interpolation and prediction tasks, in which we reveal either 10%, 20%, 30%, or 50% of the ground truth.
Revised: To deal with partial observations, we conduct interpolation and prediction tasks, in which we reveal either 10%, 20%, 30%, or 50% of the ground truth.

Original: • PhysioNet 2012 (Silva et al., 2012)
Revised: • PhysioNet 2012 (Silva et al., 2012)

Original: To investigate sparsely observed time series, we evaluate our model on a real-world clinical dataset.
Revised: To investigate sparsely observed time series, we evaluate our model on a real-world clinical dataset.

Original: PhysioNet Challenge 2012 dataset is irregularly sampled and sparsely observed.
Revised: PhysioNet Challenge 2012 dataset is irregularly sampled and sparsely observed.

Original: The goal is to interpolate and predict 41 biomedical features, such as heart rate and glucose, of intensive care unit (ICU) patients.
Revised: The goal is to interpolate and predict 41 biomedical features, such as heart rate and glucose, of intensive care unit (ICU) patients.

Original: • Human Activity Human Activity dataset consists of sporadically observed sensor data collected from five individuals performing several activities (i
Revised: • Human Activity Human Activity dataset consists of sporadically observed sensor data collected from five individuals performing several activities (i

Original: [Table]
Revised: [Table]

Original: e. walking, standing, laying, etc).
Revised: e. walking, standing, laying, etc).

Original: We use the
Revised: We use the

Original: Activity dataset together with all pre-processing steps as they were provided by (Rubanova et al., 2019), resulting in 6554 sequences of 211 time points.
Revised: Activity dataset together with all pre-processing steps as they were provided by (Rubanova et al., 2019), resulting in 6554 sequences of 211 time points.

Original: We train our model to classify the type of human activities from sequential data.
Revised: We train our model to classify the type of human activities from sequential data.

Original: Baselines Five models are chosen for the comparisons: (1) RNN-VAE, a variational autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) model whose encoder and decoder are recurrent neural networks (RNNs).
Revised: Baselines Five models are chosen for the comparisons: (1) RNN-VAE, a variational autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) model whose encoder and decoder are recurrent neural networks (RNNs).

Original: (2) ODE-RNN (Rubanova et al., 2019), a RNN model which uses Neural ODEs to model hidden state dynamics.
Revised: (2) ODE-RNN (Rubanova et al., 2019), a RNN model which uses Neural ODEs to model hidden state dynamics.

Original: (3) Two Latent ODE (LODE) models with RNN (Chen et al., 2018) and ODE-RNN (Rubanova et al., 2019) encoders.
Revised: (3) Two Latent ODE (LODE) models with RNN (Chen et al., 2018) and ODE-RNN (Rubanova et al., 2019) encoders.

Original: (4) Coupling Flow in Neural
Revised: (4) Coupling Flow in Neural

Original: Flow (Biloš et al., 2021), a neural operator which directly models the solution curves of an ODE.
Revised: Flow (Biloš et al., 2021), a neural operator which directly models the solution curves of an ODE.

Original: Results Discussion
Revised: Results Discussion

Original: We follow the encoder-decoder framework of Latent ODEs that leverages a
Revised: We follow the encoder-decoder framework of Latent ODEs that leverages a

Original: VAE architecture to represent incomplete time-series data as a continuous-time model.
Revised: VAE architecture to represent incomplete time-series data as a continuous-time model.

Original: For CTFNO, we employ a simple RNN encoder, and focus on the model representations of inherent dynamics in the latent space.
Revised: For CTFNO, we employ a simple RNN encoder, and focus on the model representations of inherent dynamics in the latent space.

Original: Results on MuJoCo reported in Table 3 show that CTFNO consistently outperforms the baseline models to a large extent on all tasks of both interpolation and prediction across all kinds of observed time points.
Revised: Results on MuJoCo reported in Table 3 show that CTFNO consistently outperforms the baseline models to a large extent on all tasks of both interpolation and prediction across all kinds of observed time points.

Original: This validates that our model can be used as a relevant model for real applications on time-series with missing time steps.
Revised: This validates that our model can be used as a relevant model for real applications on time-series with missing time steps.

Original: Table 4 summarizes interpolative and predictive performance on PhysioNet, and per-time-point classification accuracy on Activity.
Revised: Table 4 summarizes interpolative and predictive performance on PhysioNet, and per-time-point classification accuracy on Activity.

Original: CTFNO is superior to all of the benchmark models on both PhysioNet and Activity, which indicate that our model provides a useful utilization of sparsely observed time-series data and a meaningful representation for classification.
Revised: CTFNO is superior to all of the benchmark models on both PhysioNet and Activity, which indicate that our model provides a useful utilization of sparsely observed time-series data and a meaningful representation for classification.

Original: The overall results demonstrate that our proposed model, which is based on PDEs, consistently improves the performance of ODE-based models in real applications and is a novel approach that can handle a wide range of real-world problems.
Revised: The overall results demonstrate that our proposed model, which is based on PDEs, consistently improves the performance of ODE-based models in real applications and is a novel approach that can handle a wide range of real-world problems.

Original: Model
Revised: Model

Original: PhysioNet
Revised: PhysioNet

Original: Activity
Revised: Activity

Original: Interpolation Prediction Accuracy
Revised: Interpolation Prediction Accuracy

Original: RNN-VAE 5.
Revised: RNN-VAE 5.

Original: ODE-RNN 2.
Revised: ODE-RNN 2.

Original: LODE (RNN enc) 3.
Revised: LODE (RNN enc) 3.

Original: LODE (ODE enc) 2.
Revised: LODE (ODE enc) 2.

Original: Neural Flow 2.
Revised: Neural Flow 2.

Original: CTFNO 1.
Revised: CTFNO 1.

Original: 4. S TUDIES ON S TABILITY
Revised: 4. S TUDIES ON S TABILITY

Original: This section is devoted to validating the effect of the proposed stabilization scheme in Section 3.4.
Revised: This section is devoted to validating the effect of the proposed stabilization scheme in Section 3.4.

Original: We include experimental results on a real-time series data and adversarial attacks on image classification.
Revised: We include experimental results on a real-time series data and adversarial attacks on image classification.

Original: Datasets We the stability on a real-world time-series data and two image datasets.
Revised: Datasets We the stability on a real-world time-series data and two image datasets.

Original: • Plane Vibration (Noël & Schoukens, 2017) is considered to elucidate how stabilization scheme works in practical time-series data.
Revised: • Plane Vibration (Noël & Schoukens, 2017) is considered to elucidate how stabilization scheme works in practical time-series data.

Original: It is a multi-variate data, consisting of five features; force, voltage, and accelerations measured in three spots.
Revised: It is a multi-variate data, consisting of five features; force, voltage, and accelerations measured in three spots.

Original: We comply the experimental setup as provided in HBNODEs (Xia et al., 2021).
Revised: We comply the experimental setup as provided in HBNODEs (Xia et al., 2021).

Original: The task of plane vibration is interested in forecasting the next eight time steps from the previous 64 consecutive time observations.
Revised: The task of plane vibration is interested in forecasting the next eight time steps from the previous 64 consecutive time observations.

Original: • Image datasets We perform experiments on image classification, which is one of the benchmark applications of Neural ODEs, on two benchmark datasets: MNIST (LeCun et al., 1998) and CIFAR10 (Krizhevsky et al., 2009).
Revised: • Image datasets We perform experiments on image classification, which is one of the benchmark applications of Neural ODEs, on two benchmark datasets: MNIST (LeCun et al., 1998) and CIFAR10 (Krizhevsky et al., 2009).

Original: Although this is not a time-dependent problem, we include classification to validate the effect of the proposed stabilization scheme.
Revised: Although this is not a time-dependent problem, we include classification to validate the effect of the proposed stabilization scheme.

Original: Images could be considered as functions a : [0 , 1] 2 → R 3 of light defined on a continuous region of pixel locations, instead of 32 × 32 pixel vectors of RGB values.
Revised: Images could be considered as functions a : [0 , 1] 2 → R 3 of light defined on a continuous region of pixel locations, instead of 32 × 32 pixel vectors of RGB values.

Original: Classification requires us to learn a dynamics that simultaneously drives each input a ( x ) to the corresponding final feature u ( x ) , allocated depending on its label.
Revised: Classification requires us to learn a dynamics that simultaneously drives each input a ( x ) to the corresponding final feature u ( x ) , allocated depending on its label.

Original: Therefore, posing the problem as learning an operator that maps the image a ( x ) to u ( x ) , we can apply FNO to image classification.
Revised: Therefore, posing the problem as learning an operator that maps the image a ( x ) to u ( x ) , we can apply FNO to image classification.

Original: Baselines We compare the performance on plane vibration with the results reported in (Xia et al., 2021), which contains second-order Neural ODE models including ANODE, SONODE (Norcliffe et al., 2020), HBNODE, and GHBNODE (Xia et al., 2021).
Revised: Baselines We compare the performance on plane vibration with the results reported in (Xia et al., 2021), which contains second-order Neural ODE models including ANODE, SONODE (Norcliffe et al., 2020), HBNODE, and GHBNODE (Xia et al., 2021).

Original: For image classification, we take CNN and ANODE as baselines.
Revised: For image classification, we take CNN and ANODE as baselines.

Original: All models are fit to share the similar number of parameters.
Revised: All models are fit to share the similar number of parameters.

Original: Improvement in Model Generalization MSE ( × 10 − 2 ) of Neural ODEs reported in (Xia et al., 2021) is ≥ 2.5, and it is annotated by a green dashed line in Figure 3.
Revised: Improvement in Model Generalization MSE ( × 10 − 2 ) of Neural ODEs reported in (Xia et al., 2021) is ≥ 2.5, and it is annotated by a green dashed line in Figure 3.

Original: Our CTFNO achieves much lower MSE of 1.96.
Revised: Our CTFNO achieves much lower MSE of 1.96.

Original: Figure 3 presents a positive effect of stabilization; a tendency of CTFNO to be less prone to overfitting.
Revised: Figure 3 presents a positive effect of stabilization; a tendency of CTFNO to be less prone to overfitting.

Original: The ability to generalize well outside the training dataset is essential for models to be practically useful.
Revised: The ability to generalize well outside the training dataset is essential for models to be practically useful.

Original: We observe that increasing  M leads to overfitting, and too small
Revised: We observe that increasing the Gershgorin’s discs stabilization parameter M leads to overfitting

Revised: [Table]

Original:  M produces a degenerated system, dropping the performance.
Revised: and too small M produces a degenerated system, dropping the performance.

Original: Besides, the model with congenial stabilization does not suffer from overfitting or degradation of the accuracy, which validates the effect of stabilization.
Revised: Besides, the model with congenial stabilization does not suffer from overfitting or degradation of the accuracy, which validates the effect of stabilization.

Original: Moreover, it is notable that Neural ODEs require tremendous computational burdens due to their use of numerical ODE solvers.
Revised: Moreover, it is notable that Neural ODEs require tremendous computational burdens due to their use of numerical ODE solvers.

Original: By obviating the need for the ODE solver, the computational time of CTFNO is merely about 10% of Neural ODEs.
Revised: By obviating the need for the ODE solver, the computational time of CTFNO is merely about 10% of Neural ODEs.

Original: Similar investigation in image classification is included in Appendix D.2.[Table] [Table] /255 PGD-5/
Revised: Similar investigation in image classification is included in Appendix D.2.

Original: CNN

Original: [Table]

Original: [Table]

Original: ANODE

Original: [Table]

Original: [Table]

Original: [Table]

Original: FNO 64.

Original: FNO

Original: [Table]
Revised: [Table]

Original: WD)

Original: [Table]
Revised: [Table]

Original: FNO (Stab) 65. Robustness on Adversarial Attack By ensuring the stability of the learned PDE in Section 3.4, we may expect that the resulting solution for an input with a small perturbation converges to the same label as the unperturbed one.
Revised: Robustness on Adversarial Attack By ensuring the stability of the learned PDE in Section 3.4, we may expect that the resulting solution for an input with a small perturbation converges to the same label as the unperturbed one.

Original: To investigate this stabilization effect on FNO, we test the trained model in defending against Gaussian noise, fast gradient signed method (FGSM)
Revised: To investigate this stabilization effect on FNO, we test the trained model in defending against Gaussian noise, fast gradient signed method (FGSM)

Original: (Goodfellow et al., 2014), and projected gradient descent (PGD) (Madry et al., 2017) attack.
Revised: (Goodfellow et al., 2014), and projected gradient descent (PGD) (Madry et al., 2017) attack.

Original: Weight decay (WD) is a standard practical technique to regularize the L 2 norm of weights of models.
Revised: Weight decay (WD) is a standard practical technique to regularize the L 2 norm of weights of models.

Original: Because WD has a similar intention to our stabilization scheme, these two are also compared.
Revised: Because WD has a similar intention to our stabilization scheme, these two are also compared.

Original: For fair comparison,
Revised: For fair comparison,

Original: CNN and ANODE are trained with WD.
Revised: CNN and ANODE are trained with WD.

Original: Tables 5 and 6 show that for both datasets, our stabilization significantly enhances the robustness of the model.
Revised: Tables 5 and 6 show that for both datasets, our stabilization significantly enhances the robustness of the model.

Original: Our model achieves better results in both accuracy and robustness than ANODE.
Revised: Our model achieves better results in both accuracy and robustness than ANODE.

Original: Although CNN produces around 10% higher accuracy in CIFAR10, our model outperforms CNN for perturbed images.
Revised: Although CNN produces around 10% higher accuracy in CIFAR10, our model outperforms CNN for perturbed images.

Original: Moreover, the results indicate that FNO trained with WD is prone to mislead by adversarial attacks.
Revised: Moreover, the results indicate that FNO trained with WD is prone to mislead by adversarial attacks.

Original: This demonstrates that it is much more accurate to apply a model-suitable stabilization based on the PDE theory, rather than naively regularizing the
Revised: This demonstrates that it is much more accurate to apply a model-suitable stabilization based on the PDE theory, rather than naively regularizing the

Original: L 2 norm of the weights.
Revised: L 2 norm of the weights.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: ODE based Networks The interpretation of a residual network (He et al., 2016) as a discretization of an ODE (Chen et al., 2018; Haber & Ruthotto, 2017; Lu et al., 2018) provided an interface between deep learning and ODEs.
Revised: ODE based Networks The interpretation of a residual network (He et al., 2016) as a discretization of an ODE (Chen et al., 2018; Haber & Ruthotto, 2017; Lu et al., 2018) provided an interface between deep learning and ODEs.

Original: Subsequently, extensive work has been conducted on parametrizing the continuous dynamics of hidden states using an ODE specified by neural networks (Greydanus et al., 2019;
Revised: Subsequently, extensive work has been conducted on parametrizing the continuous dynamics of hidden states using an ODE specified by neural networks (Greydanus et al., 2019;

Original: Lu et al., 2019b; Liu et al., 2021).
Revised: Lu et al., 2019b; Liu et al., 2021).

Original: These approaches have found wide applicability across diverse applications including vision tasks (Scao, 2020; Park & Kim, 2021; Pinckaers & Litjens, 2019; Xu et al., 2021), mean-field games (Ruthotto et al., 2020), speech synthesis (Kim et al., 2020a), and density estimation (Chen et al., 2019; Yildiz et al., 2019).
Revised: These approaches have found wide applicability across diverse applications including vision tasks (Scao, 2020; Park & Kim, 2021; Pinckaers & Litjens, 2019; Xu et al., 2021), mean-field games (Ruthotto et al., 2020), speech synthesis (Kim et al., 2020a), and density estimation (Chen et al., 2019; Yildiz et al., 2019).

Original: Owing to the continuous representation of neural networks, Neural ODEs are particularly attractive for irregularly-sampled time series data (Rubanova et al., 2019; De Brouwer et al., 2019;
Revised: Owing to the continuous representation of neural networks, Neural ODEs are particularly attractive for irregularly-sampled time series data (Rubanova et al., 2019; De Brouwer et al., 2019;

Original: Chang et al., 2019; Kidger et al., 2020; Chen et al., 2020).
Revised: Chang et al., 2019; Kidger et al., 2020; Chen et al., 2020).

Original: A recent work(Biloš et al., 2021) circumvents the usage of an expensive numerical integration by directly parametrize the solution trajectory of an ODE.
Revised: A recent work(Biloš et al., 2021) circumvents the usage of an expensive numerical integration by directly parametrize the solution trajectory of an ODE.

Original: Furthermore, stable (Kolter & Manek, 2019; Zhang et al., 2019; Kang et al., 2021; Tuor et al., 2020) and reversible (Chang et al., 2018;
Revised: Furthermore, stable (Kolter & Manek, 2019; Zhang et al., 2019; Kang et al., 2021; Tuor et al., 2020) and reversible (Chang et al., 2018;

Original: Grathwohl et al., 2018) architectures has been developed based on the intrinsic properties of ODEs.
Revised: Grathwohl et al., 2018) architectures has been developed based on the intrinsic properties of ODEs.

Original: PDE based Networks Some studies have integrated PDEs and neural networks (Eliasof et al., 2021; Ruthotto & Haber, 2020; Ben-Yair et al., 2021; Sun et al., 2020).
Revised: PDE based Networks Some studies have integrated PDEs and neural networks (Eliasof et al., 2021; Ruthotto & Haber, 2020; Ben-Yair et al., 2021; Sun et al., 2020).

Original: These works have established based on a link between convolution filters in CNNs and differential operators (Long et al., 2018), or on a finite difference discretization (Strikwerda, 2004) of derivatives.
Revised: These works have established based on a link between convolution filters in CNNs and differential operators (Long et al., 2018), or on a finite difference discretization (Strikwerda, 2004) of derivatives.

Original: Recently, a new approach (Kim et al., 2020b) that uses a physics-informed neural network (PINN) (Raissi et al., 2019) has been proposed.
Revised: Recently, a new approach (Kim et al., 2020b) that uses a physics-informed neural network (PINN) (Raissi et al., 2019) has been proposed.

Original: All of these works assign author-defined specific PDEs to neural networks.
Revised: All of these works assign author-defined specific PDEs to neural networks.

Original: Because the nature of each problem to which the approach has been applied differ considerably, the choice of dynamics has been shown to be quite important.
Revised: Because the nature of each problem to which the approach has been applied differ considerably, the choice of dynamics has been shown to be quite important.

Original: Therefore, such architectures designed with fixed
Revised: Therefore, such architectures designed with fixed

Original: PDEs cannot be widely applied.
Revised: PDEs cannot be widely applied.

Original: Despite their PDE-based motivation, those approaches were only applied to image classification tasks and did not explore the application on continuous inputs and time-series data.
Revised: Despite their PDE-based motivation, those approaches were only applied to image classification tasks and did not explore the application on continuous inputs and time-series data.

Original:  C ONCLUSION 
Revised:  C ONCLUSION & L IMITATIONS

Original: In this paper, we presented a novel approach for modeling time-series in terms of PDEs.
Revised: In this paper, we presented a novel approach for modeling time-series in terms of PDEs.

Original: As time is intrinsically continuous, we proposed a neural operator CTFNO that learns the underlying PDE by equipping FNO with the ability to represent time in a continuous manner.
Revised: As time is intrinsically continuous, we proposed a neural operator CTFNO that learns the underlying PDE by equipping FNO with the ability to represent time in a continuous manner.

Original: We also provided theoretical guarantees for the universal approximation and the stability of CTFNO.
Revised: We also provided theoretical guarantees for the universal approximation and the stability of CTFNO.

Original: Our comprehensive experiments demonstrated that CTFNO outperforms existing differential equation based models on synthetic and real-world datasets, and the proposed stabilization method effectively improves robustness and prevents overfitting.
Revised: Our comprehensive experiments demonstrated that CTFNO outperforms existing differential equation based models on synthetic and real-world datasets, and the proposed stabilization method effectively improves robustness and prevents overfitting.

Original: We note that FNO is hard to embody discontinuous features because Fourier transform only captures global information.
Revised: We note that FNO is hard to embody discontinuous features because Fourier transform only captures global information.

Original: We expect that extending our approach to a model that can extract local features, such as (Gupta et al., 2021), provides interesting avenues for future work.
Revised: We expect that extending our approach to a model that can extract local features, such as (Gupta et al., 2021), provides interesting avenues for future work.

Revised: Also, the stabilization parameter M is a user-defined hyperparameter.

Revised: The optimal

Revised: M for a given data is unknown and this is one of our limitation.

Original: A U NIVERSAL A PPROXIMATION A. M ATHEMATICAL N OTATION
Revised: A U NIVERSAL A PPROXIMATION A. M ATHEMATICAL N OTATION

Original: We introduce the symbols and mathematical notations that are frequently used in this paper.
Revised: We introduce the symbols and mathematical notations that are frequently used in this paper.

Original: [Table]
Revised: [Table]

Original: T
Revised: T

Original: [Table]
Revised: [Table]

Original: L the number
Revised: L the number

Original: [Table]
Revised: [Table]

Original: layers Ω ⊂ R d spatial domain x point in the the spatial domain ξ frequency variable in the Fourier domain
Revised: layers Ω ⊂ R d spatial domain x point in the the spatial domain ξ frequency variable in the Fourier domain

Original: A ( D ; R d a ) input function space
Revised: A ( D ; R d a ) input function space

Original: U ( D ; R d u ) output function space
Revised: U ( D ; R d u ) output function space

Original: F X , F − 1 X
Revised: F X , F − 1 X

Original: Fourier transform and inverse Fourier transform with respect to X
Revised: Fourier transform and inverse Fourier transform with respect to X

Original: F N , F − 1 N discrete Fourier transform and inverse
Revised: F N , F − 1 N discrete Fourier transform and inverse

Original: K N
Revised: K N

Original: Fourier wavenumbers
Revised: Fourier wavenumbers

Original: [Equation]
Revised: [Equation]

Original: (cid:9)
Revised: (cid:9)

Original: P lifting operator
Revised: P lifting operator

Original: L ℓ neural operator layer
Revised: L ℓ neural operator layer

Original: Q projection operator
Revised: Q projection operator

Original: Lspace of square-integrable functions
Revised: Lspace of square-integrable functions

Original: L 2 N
Revised: L 2 N

Original: L 2 N ⊂ L 2 trigonometric polynomials of degree ≤ N
Revised: L 2 N ⊂ L 2 trigonometric polynomials of degree ≤ N

Original: H s
Revised: H s

Original: Sobolev space of smoothness s , with norm ∥ · ∥ H s
Revised: Sobolev space of smoothness s , with norm ∥ · ∥ H s

Original: P N
Revised: P N

Original: L 2 -orthogonal Fourier projection P N : L 2 → L 2 N
Revised: L 2 -orthogonal Fourier projection P N : L 2 → L 2 N

Original: C ([0 , T ] × X ) space of continuous functions u : [0 , T ] →
Revised: C ([0 , T ] × X ) space of continuous functions u : [0 , T ] →

Original: X satisfying
Revised: X satisfying

Original: [Equation]
Revised: [Equation]

Original: ∞
Revised: ∞

Original: D n space of n × n diagonal matrices
Revised: D n space of n × n diagonal matrices

Original: I n n × n identity matrix
Revised: I n n × n identity matrix

Original: A. P ROOF OF UNIVERSAL APPROXIMATION
Revised: A. P ROOF OF UNIVERSAL APPROXIMATION

Original: A
Revised: A

Original: neural
Revised: neural

Original: operator
Revised: operator

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: a ) (6) is defined as the following form
Revised: a ) (6) is defined as the following form

Original: [Equation]
Revised: [Equation]

Original: with a lifting operator
Revised: with a lifting operator

Original: [Equation]
Revised: [Equation]

Original: for all a ∈ A (cid:0) Ω; R d a (cid:1) , x ∈ Ω , and a projection operator
Revised: for all a ∈ A (cid:0) Ω; R d a (cid:1) , x ∈ Ω , and a projection operator

Original: [Equation]
Revised: [Equation]

Original: Moreover, as introduced in Section 3.2 the time-dependent neural operator layer is defined by
Revised: Moreover, as introduced in Section 3.2 the time-dependent neural operator layer is defined by

Original: [Equation]
Revised: [Equation]

Original: Before we establish a proof of the universality of CTFNO, let us start by recalling the following
Revised: Before we establish a proof of the universality of CTFNO, let us start by recalling the following

Original: Lemma provided in (Kovachki et al., 2021).
Revised: Lemma provided in (Kovachki et al., 2021).

Original: Lemma A.1 ((Kovachki et al., 2021)) .
Revised: Lemma A.1 ((Kovachki et al., 2021)) .

Original: For FNO introduced in Section 2, the following holds.
Revised: For FNO introduced in Section 2, the following holds.

Original: (a) Let G : H s (cid:0) T d ; R d a (cid:1)
Revised: (a) Let G : H s (cid:0) T d ; R d a (cid:1)

Original: → H s ′ (cid:0) T d ; R d u (cid:1) be continuous operator with s, s ′ ≥ 0 and K
Revised: → H s ′ (cid:0) T d ; R d u (cid:1) be continuous operator with s, s ′ ≥ 0 and K

Original: ⊂ H s be a compact subset.
Revised: ⊂ H s be a compact subset.

Original: Then for any ϵ > 0 there exists N ∈ N , such that for all a ∈ K
Revised: Then for any ϵ > 0 there exists N ∈ N , such that for all a ∈ K

Original: [Equation]
Revised: [Equation]

Original: where G N : H s (cid:0) T d (cid:1) → L 2 (cid:0) T d (cid:1) is defined by G N ( a ) :
Revised: where G N : H s (cid:0) T d (cid:1) → L 2 (cid:0) T d (cid:1) is defined by G N ( a ) :

Original: = P N G ( P N a ) .
Revised: = P N G ( P N a ) .

Original: (b) Let B > 0 and ϵ > 0 be given.
Revised: (b) Let B > 0 and ϵ > 0 be given.

Original: Then there exists a FNO N FT such that for all v ∈ L 2 with ∥ v ∥ L 2 ≤ B
Revised: Then there exists a FNO N FT such that for all v ∈ L 2 with ∥ v ∥ L 2 ≤ B

Original: [Equation]
Revised: [Equation]

Original: (c) Let B > 0 and ϵ > 0 be given.
Revised: (c) Let B > 0 and ϵ > 0 be given.

Original: Then there eixsts a FNO N IFT such that for all { ˆ v ( ξ ) : ξ ∈ K N } with ∥ ˆ v ∥ L ∞ ( K N ) ≤ B
Revised: Then there eixsts a FNO N IFT such that for all { ˆ v ( ξ ) : ξ ∈ K N } with ∥ ˆ v ∥ L ∞ ( K N ) ≤ B

Original: [Equation]
Revised: [Equation]

Original: (d) Let B > 0 and ϵ > 0 be given.
Revised: (d) Let B > 0 and ϵ > 0 be given.

Original: Then there exists a FNO (cid:98) N such that for any { ˆ v ( ξ ) } ∈ C K N with ∥ ˆ v ∥ L ∞ ( K N ) ≤ B such that
Revised: Then there exists a FNO (cid:98) N such that for any { ˆ v ( ξ ) } ∈ C K N with ∥ ˆ v ∥ L ∞ ( K N ) ≤ B such that

Original: [Equation]
Revised: [Equation]

Original: In what follows, we prove our main theorem showing that the proposed CTFNO is a universal approximator of continuous operators.
Revised: In what follows, we prove our main theorem showing that the proposed CTFNO is a universal approximator of continuous operators.

Original: Theorem A.2 (Universal approximation of CTFNO) .
Revised: Theorem A.2 (Universal approximation of CTFNO) .

Original: Let s, s ′ ≥ 0 .
Revised: Let s, s ′ ≥ 0 .

Original: Suppose G : H s (cid:0) T d ; R d a (cid:1) →
Revised: Suppose G : H s (cid:0) T d ; R d a (cid:1) →

Original: C (cid:16) [0 , T ] × H s ′ (cid:0) T d ; R d u (cid:1)(cid:17) is a continuous operator and K
Revised: C (cid:16) [0 , T ] × H s ′ (cid:0) T d ; R d u (cid:1)(cid:17) is a continuous operator and K

Original: ⊂ H s (cid:0) T d ; R d a (cid:1) is a compact subset.
Revised: ⊂ H s (cid:0) T d ; R d a (cid:1) is a compact subset.

Original: Then for any ϵ > 0 , there exists a CTFNO N : H s (cid:0) T d ; R d a (cid:1) → C (cid:16) [0 , T ] × H s ′ (cid:0) T d ; R d u (cid:1)(cid:17) , of the form (7) , continuous as an operator H s (cid:0) T d ; R d a (cid:1) → C (cid:16) [0 , T ] × H s ′ (cid:0) T d ; R d u (cid:1)(cid:17) , such that
Revised: Then for any ϵ > 0 , there exists a CTFNO N : H s (cid:0) T d ; R d a (cid:1) → C (cid:16) [0 , T ] × H s ′ (cid:0) T d ; R d u (cid:1)(cid:17) , of the form (7) , continuous as an operator H s (cid:0) T d ; R d a (cid:1) → C (cid:16) [0 , T ] × H s ′ (cid:0) T d ; R d u (cid:1)(cid:17) , such that

Original: [Equation]
Revised: [Equation]

Original: ,T ] × H s ′ ( T d
Revised: ,T ] × H s ′ ( T d

Original: [Equation]
Revised: [Equation]

Original: R du ) )
Revised: R du ) )

Original: <
Revised: <

Original: ϵ.
Revised: ϵ.

Original: The proof is essentially split into two parts.
Revised: The proof is essentially split into two parts.

Original: The first part states that the Fourier projection can approximate continuous operators.
Revised: The first part states that the Fourier projection can approximate continuous operators.

Original: To simplify notation, we use shorthand H s
Revised: To simplify notation, we use shorthand H s

Original: = H s (cid:0) T d ; R d a (cid:1) ,
Revised: = H s (cid:0) T d ; R d a (cid:1) ,

Original: H s ′ =
Revised: H s ′ =

Original: H s ′ (cid:0) T d ; R d u (cid:1) and a t ( x ) := a ( t, x ) in the remainder of the paper.
Revised: H s ′ (cid:0) T d ; R d u (cid:1) and a t ( x ) := a ( t, x ) in the remainder of the paper.

Original: Lemma A.3.
Revised: Lemma A.3.

Original: Let G : C ([0 , T ] × H s )
Revised: Let G : C ([0 , T ] × H s )

Original: → C (cid:16) [0 , T ] × H s ′ (cid:17) be a continuous operator and K ⊂
Revised: → C (cid:16) [0 , T ] × H s ′ (cid:17) be a continuous operator and K ⊂

Original: C ([0 , T ] × H s ) a compact subset.
Revised: C ([0 , T ] × H s ) a compact subset.

Original: Then for any ϵ > 0 there exists N ∈ N , such that for all a ∈ K and t ∈ [0 , T ]
Revised: Then for any ϵ > 0 there exists N ∈ N , such that for all a ∈ K and t ∈ [0 , T ]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since G is uniformly continuous on a compact domain G ( K ) , for any ϵ > 0 there exists δ >such that
Revised: Since G is uniformly continuous on a compact domain G ( K ) , for any ϵ > 0 there exists δ >such that

Original: [Equation]
Revised: [Equation]

Original: provided | t 1 − t 2 | < δ .
Revised: provided | t 1 − t 2 | < δ .

Original: Let 0 =
Revised: Let 0 =

Original: t 0 < t 1 < . . .
Revised: t 0 < t 1 < . . .

Original: <
Revised: <

Original: t M = T be a partition of [0 , T ] with t
Revised: t M = T be a partition of [0 , T ] with t

Original: i +1 − t i <
Revised: i +1 − t i <

Original: δ and G i : H s → H s ′ be defined by a (cid:55)→ G ( a ) t i for each i .
Revised: δ and G i : H s → H s ′ be defined by a (cid:55)→ G ( a ) t i for each i .

Original: Then, (a) in Lemma A.1 furnishes that there exists N i ∈ N such that
Revised: Then, (a) in Lemma A.1 furnishes that there exists N i ∈ N such that

Original: [Equation]
Revised: [Equation]

Original: Combining all, for N > max { N i : i = 0 , . .
Revised: Combining all, for N > max { N i : i = 0 , . .

Original: , M } , we have
Revised: , M } , we have

Original: [Equation]
Revised: [Equation]

Original: for any a ∈ K and t ∈ [0 , T ] .
Revised: for any a ∈ K and t ∈ [0 , T ] .

Original: This proves the lemma.
Revised: This proves the lemma.

Original: The second part is to prove that any time-dependent continuous function can be approximated by a three-layered feed forward network combined with the proposed time-module.
Revised: The second part is to prove that any time-dependent continuous function can be approximated by a three-layered feed forward network combined with the proposed time-module.

Original: Lemma A.4.
Revised: Lemma A.4.

Original: Let f : [0 , T ] × R d x → R d y be a continuous function and K ⊆ R d x a compact subset.
Revised: Let f : [0 , T ] × R d x → R d y be a continuous function and K ⊆ R d x a compact subset.

Original: Then for any ϵ > 0 , there exists a three-layered feed forward network with time
Revised: Then for any ϵ > 0 , there exists a three-layered feed forward network with time

Original: [Equation]
Revised: [Equation]

Original: satisfying
Revised: satisfying

Original: [Equation]
Revised: [Equation]

Original: where σ is an activation function, ψ w,ℓ : [0 , T ] → D d v and ψ b,ℓ : [0 , T ] → R d v are continuous, Q is a projection matrix, and W ℓ , b ℓ are weight matrix and bias vector for each ℓ , respectively.
Revised: where σ is an activation function, ψ w,ℓ : [0 , T ] → D d v and ψ b,ℓ : [0 , T ] → R d v are continuous, Q is a projection matrix, and W ℓ , b ℓ are weight matrix and bias vector for each ℓ , respectively.

Original: Proof.
Revised: Proof.

Original: Without loss of generality, we may assume d y = 1 .
Revised: Without loss of generality, we may assume d y = 1 .

Original: Construct W 1 ∈ R
Revised: Construct W 1 ∈ R

Original: d x +1)
Revised: d x +1)

Original: ×
Revised: ×

Original: d x and b 1 ∈ R
Revised: d x and b 1 ∈ R

Original: d
Revised: d

Original: x +1 as
Revised: x +1 as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , with ψ w, 1 ( t ) = 

Revised: , with ψ w, 1 ( t ) =

Original: d x +1 , ψ b, 1 ( t ) =
Revised: d x +1 , ψ b, 1 ( t ) =

Original: t 1 d x +1 .
Revised: t 1 d x +1 .

Original: Then, L 1 ( t, x ) = (cid:0) t, ( x, t ) T (cid:1) ∈
Revised: Then, L 1 ( t, x ) = (cid:0) t, ( x, t ) T (cid:1) ∈

Original: [0 , T ] × R d x +1 .
Revised: [0 , T ] × R d x +1 .

Original: By the universal approximation theorem of Cybenko (1989), for any ϵ > 0 there exists W 2 ∈ R d v × ( d x +1) , b 2 ∈ R d v , and Q ∈ R 1 × d v such that
Revised: By the universal approximation theorem of Cybenko (1989), for any ϵ > 0 there exists W 2 ∈ R d v × ( d x +1) , b 2 ∈ R d v , and Q ∈ R 1 × d v such that

Original: [Equation]
Revised: [Equation]

Original: By setting ψ w, 2 = I d v , ψ b, 2 = 1 d v , we have the desired N .
Revised: By setting ψ w, 2 = I d v , ψ b, 2 = 1 d v , we have the desired N .

Original: With these lemmas out of the way, we are ready to provide a proof of Theorem 3.1.
Revised: With these lemmas out of the way, we are ready to provide a proof of Theorem 3.1.

Original: For given continuous operator G : C ([0 , T ] × H s )
Revised: For given continuous operator G : C ([0 , T ] × H s )

Original: → C (cid:0) [0 , T ] × L 2 (cid:1) , compact subset K ⊂
Revised: → C (cid:0) [0 , T ] × L 2 (cid:1) , compact subset K ⊂

Original: C ([0 , T ] × H s ) and small ϵ > 0 , Lemma A.3 gives an integer N ∈ N such that
Revised: C ([0 , T ] × H s ) and small ϵ > 0 , Lemma A.3 gives an integer N ∈ N such that

Original: [Equation]
Revised: [Equation]

Original: for each time t ∈ [0 , T ] .
Revised: for each time t ∈ [0 , T ] .

Original: Since K is compact, there exists B > 0 satisfying ∥ v t ∥
Revised: Since K is compact, there exists B > 0 satisfying ∥ v t ∥

Original: L 2 ≤ B for all v ∈ K and t ∈ [0 , T ] .
Revised: L 2 ≤ B for all v ∈ K and t ∈ [0 , T ] .

Original: For given small δ ∈ (0 , 1) , let N FT be a CTFNO such that for all function w with ∥ w ∥ L 2 ≤ B
Revised: For given small δ ∈ (0 , 1) , let N FT be a CTFNO such that for all function w with ∥ w ∥ L 2 ≤ B

Original: [Equation]
Revised: [Equation]

Original: In fact, δ is chosen according to the Lipschitz constant of (cid:98) N , which will be constructed.
Revised: In fact, δ is chosen according to the Lipschitz constant of (cid:98) N , which will be constructed.

Original: As claimed by ( c ) in Lemma A.1, there exists a CTFNO N IFT such that for all { ˆ w ( ξ ) : ξ ∈ K N } with | ˆ
Revised: As claimed by ( c ) in Lemma A.1, there exists a CTFNO N IFT such that for all { ˆ w ( ξ ) : ξ ∈ K N } with | ˆ

Original: [Equation]
Revised: [Equation]

Original: sup
Revised: sup

Original: [Equation]
Revised: [Equation]

Original: The continuity of F − 1 N and N IFT leads to the existence of B ′′ > 0 that satisfies
Revised: The continuity of F − 1 N and N IFT leads to the existence of B ′′ > 0 that satisfies

Original: [Equation]
Revised: [Equation]

Original: for all { ˆ w ( ξ ) : ξ ∈ K N } with | ˆ w ( ξ ) | ≤ B ′′ .
Revised: for all { ˆ w ( ξ ) : ξ ∈ K N } with | ˆ w ( ξ ) | ≤ B ′′ .

Original: By (d) in Lemma A.1 and Lemma A.4, we can construct a CTFNO (cid:98) N such that for any { ˆ w ( ξ ) } ∈ C K N with ∥ ˆ w ∥ L ∞ ≤ B ′′ ,
Revised: By (d) in Lemma A.1 and Lemma A.4, we can construct a CTFNO (cid:98) N such that for any { ˆ w ( ξ ) } ∈ C K N with ∥ ˆ w ∥ L ∞ ≤ B ′′ ,

Original: [Equation]
Revised: [Equation]

Original: Note that the construction of (cid:98) N is independent on δ and N FT .
Revised: Note that the construction of (cid:98) N is independent on δ and N FT .

Original: Now we set δ > 0 so small that
Revised: Now we set δ > 0 so small that

Original: [Equation]
Revised: [Equation]

Original: for all a ∈ K , which implies that
Revised: for all a ∈ K , which implies that

Original: [Equation]
Revised: [Equation]

Original: Finally, for each t ∈ [0 , T ] we can deduce the following inequality:
Revised: Finally, for each t ∈ [0 , T ] we can deduce the following inequality:

Original: [Equation]
Revised: [Equation]

Original: F − 1 N ◦ (cid:98) G N ◦ F N ◦ P N ( a t ) − N IFT ◦ (cid:98) N ◦ N FT ( a t ) (cid:13)(cid:13)(cid:13) L≤ ϵ
Revised: F − 1 N ◦ (cid:98) G N ◦ F N ◦ P N ( a t ) − N IFT ◦ (cid:98) N ◦ N FT ( a t ) (cid:13)(cid:13)(cid:13) L≤ ϵ

Original: + sup a ∈ K
Revised: + sup a ∈ K

Original: (cid:13)(cid:13)(cid:13) F − 1 N ◦ (cid:98) G N ◦ F N ◦ P N ( a t ) − F − 1 N ◦ (cid:98) N ◦ N FT ( a t ) (cid:13)(cid:13)(cid:13) L+ sup a ∈ K (cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13) F − 1 N ◦ (cid:98) G N ◦ F N ◦ P N ( a t ) − F − 1 N ◦ (cid:98) N ◦ N FT ( a t ) (cid:13)(cid:13)(cid:13) L+ sup a ∈ K (cid:13)(cid:13)(cid:13)

Original: F − 1 N ◦ (cid:98) N ◦ N FT ( a t ) − N IFT ◦ (cid:98) N ◦ N FT ( a t ) (cid:13)(cid:13)(cid:13) L≤ ϵ
Revised: F − 1 N ◦ (cid:98) N ◦ N FT ( a t ) − N IFT ◦ (cid:98) N ◦ N FT ( a t ) (cid:13)(cid:13)(cid:13) L≤ ϵ

Original: + sup a ∈ K
Revised: + sup a ∈ K

Original: (cid:13)(cid:13)(cid:13) (cid:98) G N ◦ F N ◦ P N ( a t ) − (cid:98) N ◦ N FT ( a t ) (cid:13)(cid:13)(cid:13) ∞ + sup { ˆ w ( ξ ) }∈ C K N ∥
Revised: (cid:13)(cid:13)(cid:13) (cid:98) G N ◦ F N ◦ P N ( a t ) − (cid:98) N ◦ N FT ( a t ) (cid:13)(cid:13)(cid:13) ∞ + sup { ˆ w ( ξ ) }∈ C K N ∥

Original: ˆ w ( ξ ) ∥ ∞ ≤ B ′′ (cid:13)(cid:13)
Revised: ˆ w ( ξ ) ∥ ∞ ≤ B ′′ (cid:13)(cid:13)

Original: F − 1 N ( { ˆ w ( ξ ) } ) − N IFT ( { ˆ w ( ξ ) } )
Revised: F − 1 N ( { ˆ w ( ξ ) } ) − N IFT ( { ˆ w ( ξ ) } )

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: L< 2 ϵ + sup a ∈ K
Revised: L< 2 ϵ + sup a ∈ K

Original: (cid:13)(cid:13)(cid:13) (cid:98) G N ◦ F N ◦ P N ( a t ) − (cid:98) N ◦ F N ◦ P N ( a t ) (cid:13)(cid:13)(cid:13) ∞ + sup a ∈ K (cid:13)(cid:13)(cid:13) (cid:98) N ◦ F N ◦ P N ( a t ) − (cid:98) N ◦ N FT ( a t ) (cid:13)(cid:13)(cid:13) L≤
Revised: (cid:13)(cid:13)(cid:13) (cid:98) G N ◦ F N ◦ P N ( a t ) − (cid:98) N ◦ F N ◦ P N ( a t ) (cid:13)(cid:13)(cid:13) ∞ + sup a ∈ K (cid:13)(cid:13)(cid:13) (cid:98) N ◦ F N ◦ P N ( a t ) − (cid:98) N ◦ N FT ( a t ) (cid:13)(cid:13)(cid:13) L≤

Original: [Equation]
Revised: [Equation]

Original: Since t ∈ [0 , T ] is arbitrary
Revised: Since t ∈ [0 , T ] is arbitrary

Original: [Equation]
Revised: [Equation]

Original: the above inequalities bring us to the desired result
Revised: the above inequalities bring us to the desired result

Original: [Equation]
Revised: [Equation]

Original: B S TABILITY
Revised: B S TABILITY

Original: Proof of stability Suppose σ is a Lipschitz continuous activation function with Lipschitz constant
Revised: Proof of stability Suppose σ is a Lipschitz continuous activation function with Lipschitz constant

Original: C , and both sup ξ ∥ R ( t, ξ ) ∥ 22 and ∥ W ( t ) ∥ 22 are bounded by M for every t .
Revised: C , and both sup ξ ∥ R ( t, ξ ) ∥ 22 and ∥ W ( t ) ∥ 22 are bounded by M for every t .

Original: Let ˜ v be a perturbed initial condition with ∥ ˜ v − v ∥ 2 < ϵ. Then by Plancherel theorem and Fourier
Revised: Let ˜ v be a perturbed initial condition with ∥ ˜ v − v ∥ 2 < ϵ. Then by Plancherel theorem and Fourier

Original: convolution theorem, we have
Revised: convolution theorem, we have

Original: [Equation]
Revised: [Equation]

Original: C E XPERIMENTAL D ETAILS
Revised: C E XPERIMENTAL D ETAILS

Original: In this section
Revised: In this section

Original: [Equation]
Revised: [Equation]

Original: we present our experimental settings in detail.
Revised: we present our experimental settings in detail.

Original: All experiments were conducted on a single NVIDIA RTX 3090 GPU.
Revised: All experiments were conducted on a single NVIDIA RTX 3090 GPU.

Original: C. D ATASETS
Revised: C. D ATASETS

Original: Spiral Data generation of Spiral, Stiff, Sawtooth, and Square follows the implementation of Holt et al.
Revised: Spiral Data generation of Spiral, Stiff, Sawtooth, and Square follows the implementation of Holt et al.

Original: (2022) unless stated.
Revised: (2022) unless stated.

Original: We generate the Spiral dataset by following governing equation:
Revised: We generate the Spiral dataset by following governing equation:

Original: [Equation]
Revised: [Equation]

Original: where A = (cid:18) − 1 / −− 1 /(cid:19) .
Revised: where A = (cid:18) − 1 / −− 1 /(cid:19) .

Original: The other settings such as the number of training and test samples, initial value u (0) , and the number of timesteps follow the implementation of Holt et al.
Revised: The other settings such as the number of training and test samples, initial value u (0) , and the number of timesteps follow the implementation of Holt et al.

Original: Stiff Van der Pol Oscillator
Revised: Stiff Van der Pol Oscillator

Original: We generate data by following the instructions of (Van der Pol & Van
Revised: We generate data by following the instructions of (Van der Pol & Van

Original: Der Mark, 1927), which exhibits regions of high stiffness.
Revised: Der Mark, 1927), which exhibits regions of high stiffness.

Original: The governing equation is
Revised: The governing equation is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 12) where µ = 1000 .
Revised: 12) where µ = 1000 .

Original: We sample initial conditions from x (0) ∈ [0 . 1 , 2] , y (0) = 0 .
Revised: We sample initial conditions from x (0) ∈ [0 . 1 , 2] , y (0) = 0 .

Original: Sawtooth & Square We also explore the benchmarks on a periodic discontinuous function u ( t ) .
Revised: Sawtooth & Square We also explore the benchmarks on a periodic discontinuous function u ( t ) .

Original: We sample two datasets, namely sawtooth and square, and these are sampled by the equation u ( t )
Revised: We sample two datasets, namely sawtooth and square, and these are sampled by the equation u ( t )

Original: =
Revised: =

Original: t 2 π − ⌊ t 2 π ⌋ , and u ( t ) = 2 (cid:0) 1 − ⌊ 2 (cid:0) t 2 π − ⌊ t 2 π ⌋ (cid:1) ⌋ (cid:1) , respectively.
Revised: t 2 π − ⌊ t 2 π ⌋ , and u ( t ) = 2 (cid:0) 1 − ⌊ 2 (cid:0) t 2 π − ⌊ t 2 π ⌋ (cid:1) ⌋ (cid:1) , respectively.

Original: We sample initial values ( t 0 , u ( t 0 )) by sampling t 0 in uniformly at random on the interval [0 , 2 π ] .
Revised: We sample initial values ( t 0 , u ( t 0 )) by sampling t 0 in uniformly at random on the interval [0 , 2 π ] .

Original: Each trajectories are generated from intervals of [ t 0 , t 0 + 20] .
Revised: Each trajectories are generated from intervals of [ t 0 , t 0 + 20] .

Original: Reaction equation The semi-linear ODE presented in Section 4.1 has an analytic solution
Revised: Reaction equation The semi-linear ODE presented in Section 4.1 has an analytic solution

Original: [Equation]
Revised: [Equation]

Original: (13) for an initial condition f ( x ) .
Revised: (13) for an initial condition f ( x ) .

Original: The reaction coefficient ρ is chosen to be 6 and the domain is unit interval.
Revised: The reaction coefficient ρ is chosen to be 6 and the domain is unit interval.

Original: Initial conditions randomly generated as 12 ( z 1 sin (2 πk 1 x ) + z 2 sin (2 πk 2 x ))
Revised: Initial conditions randomly generated as 12 ( z 1 sin (2 πk 1 x ) + z 2 sin (2 πk 2 x ))

Original: + z 3 e − x + 2 , where z i ∼ N (0 , 1) and
Revised: + z 3 e − x + 2 , where z i ∼ N (0 , 1) and

Original: k
Revised: k

Original: i is a uniformly sampled integer.
Revised: i is a uniformly sampled integer.

Original: Analytic solutions are attained up to t = 1 .
Revised: Analytic solutions are attained up to t = 1 .

Original: The solution data is generated by solving (13).
Revised: The solution data is generated by solving (13).

Original: The spatial grid is discretized with resolution of 100 along time step size of ∆
Revised: The spatial grid is discretized with resolution of 100 along time step size of ∆

Original: Heat equation The one-dimensional heat equation used in Section 4.1 takes the form
Revised: Heat equation The one-dimensional heat equation used in Section 4.1 takes the form

Original: [Equation]
Revised: [Equation]

Original: with periodic boundary conditions.
Revised: with periodic boundary conditions.

Original: It describes how a quantity such as heat diffuses through a given region over time.
Revised: It describes how a quantity such as heat diffuses through a given region over time.

Original: The initial function u 0 ( x ) is generated from the Gaussian random field
Revised: The initial function u 0 ( x ) is generated from the Gaussian random field

Original: N (cid:16) 0 , 20 2 (cid:0) −△ + 3 . 5 2 (cid:1) − 2 . 5 (cid:17) , where △ refers to the Laplacian.
Revised: N (cid:16) 0 , 20 2 (cid:0) −△ + 3 . 5 2 (cid:1) − 2 . 5 (cid:17) , where △ refers to the Laplacian.

Original: The diffusivity constant ν is set to be 0.001 and the solution data is generated by exactly solving (14) in Fourier space on a uniform spatial grid with resolution 1024 along time step size △ t = 0 .
Revised: The diffusivity constant ν is set to be 0.001 and the solution data is generated by exactly solving (14) in Fourier space on a uniform spatial grid with resolution 1024 along time step size △ t = 0 .

Original: Burgers’ equation The one-dimensional Burgers’ equation with diffusive regularization is a canonical non-linear PDE, taking the form
Revised: Burgers’ equation The one-dimensional Burgers’ equation with diffusive regularization is a canonical non-linear PDE, taking the form

Original: [Equation]
Revised: [Equation]

Original: with periodic boundary condition.
Revised: with periodic boundary condition.

Original: It is a dissipative nonlinear system with shock formation and has various applications including the flow of viscous fluid dynamics.
Revised: It is a dissipative nonlinear system with shock formation and has various applications including the flow of viscous fluid dynamics.

Original: The viscosity is set to ν = 0 .
Revised: The viscosity is set to ν = 0 .

Original: Starting from initial functions sampled from N (cid:16) 0 , 7 2 ( −△ + 49) − 2 .
Revised: Starting from initial functions sampled from N (cid:16) 0 , 7 2 ( −△ + 49) − 2 .

Original: 5 (cid:17) , numerical ground truth is generated separately where the linear diffusion component is exactly solved and the remaining nonlinear part is solved in Fourier space using forward Euler method as in (Li et al., 2021).
Revised: 5 (cid:17) , numerical ground truth is generated separately where the linear diffusion component is exactly solved and the remaining nonlinear part is solved in Fourier space using forward Euler method as in (Li et al., 2021).

Original: The spatial domain is discretized with resolution 1024.
Revised: The spatial domain is discretized with resolution 1024.

Original: The training set consists of 400 periodic trajectories and each trajectory has regularly-sampled time points with △ t = 0 .
Revised: The training set consists of 400 periodic trajectories and each trajectory has regularly-sampled time points with △ t = 0 .

Original: and we test the trained models for 100 trajectories.
Revised: and we test the trained models for 100 trajectories.

Original: MuJoCo We use a trajectory of physical simulation for the hopper with three joints and four body parts.
Revised: MuJoCo We use a trajectory of physical simulation for the hopper with three joints and four body parts.

Original: Each time series is 14-dimensional, consisting of a five-dimensional position, six-dimensional velocity, and three-dimensional action.
Revised: Each time series is 14-dimensional, consisting of a five-dimensional position, six-dimensional velocity, and three-dimensional action.

Original: Rubanova et al.
Revised: Rubanova et al.

Original: (2019) generated 10 , 000 sequences oftimesteps and used 80% of the data for training and the rest 20% for evaluation.
Revised: (2019) generated 10 , 000 sequences oftimesteps and used 80% of the data for training and the rest 20% for evaluation.

Original: For the interpolation task, they randomly sampled 100 consecutive observation timesteps.
Revised: For the interpolation task, they randomly sampled 100 consecutive observation timesteps.

Original: For the prediction task, they use all 200 observation timesteps and divide them into two parts: the first 1/3 as an input for the model and the latter 2/3 as an output to be forecasted.
Revised: For the prediction task, they use all 200 observation timesteps and divide them into two parts: the first 1/3 as an input for the model and the latter 2/3 as an output to be forecasted.

Original: For all the tasks, they randomly sampled a portion of the input timesteps at a specified rate and masked the rest of them.
Revised: For all the tasks, they randomly sampled a portion of the input timesteps at a specified rate and masked the rest of them.

Original: Note that the values used for the subsampling ratio are 10%, 20%, 30%, and 50%.
Revised: Note that the values used for the subsampling ratio are 10%, 20%, 30%, and 50%.

Original: PhysioNet This is a publicly available dataset of 8000 time series describing the stay of patients within an ICU over 48 hours.
Revised: PhysioNet This is a publicly available dataset of 8000 time series describing the stay of patients within an ICU over 48 hours.

Original: For each patient, 41 biomedical features are irregularly observed and converted to one minutely resolution.
Revised: For each patient, 41 biomedical features are irregularly observed and converted to one minutely resolution.

Original: In (Rubanova et al., 2019), they used 80% of the data for training, and the rest 20% for evaluation, as in MuJoCo.
Revised: In (Rubanova et al., 2019), they used 80% of the data for training, and the rest 20% for evaluation, as in MuJoCo.

Original: For the interpolation task, they did not subsample the data because the measurements were already sparse.
Revised: For the interpolation task, they did not subsample the data because the measurements were already sparse.

Original: For the prediction task, they halved the data so that the first 1/2 timesteps were used for inputs, and the latter 1/2 timesteps were used for outputs to be forecasted.
Revised: For the prediction task, they halved the data so that the first 1/2 timesteps were used for inputs, and the latter 1/2 timesteps were used for outputs to be forecasted.

Original: Note that Rubanova et al.
Revised: Note that Rubanova et al.

Original: (2019) also performed per-sequence classification experiments on PhysioNet.
Revised: (2019) also performed per-sequence classification experiments on PhysioNet.

Original: Since Latent ODEs with RNN or ODE-RNN encoders compress the whole input data to a single latent vector z 0 and an MLP classifier receives only z 0 as its input, such a per-sequence classification does not depend on all the other latent vectors z 1 , . .
Revised: Since Latent ODEs with RNN or ODE-RNN encoders compress the whole input data to a single latent vector z 0 and an MLP classifier receives only z 0 as its input, such a per-sequence classification does not depend on all the other latent vectors z 1 , . .

Original: . , z T .
Revised: . , z T .

Original: This is inconsistent with the goal of CTFNO, which focuses on learning representations of the inherent dynamics for all timesteps.
Revised: This is inconsistent with the goal of CTFNO, which focuses on learning representations of the inherent dynamics for all timesteps.

Original: Hence we excluded per-sequence classification experiments in this paper.
Revised: Hence we excluded per-sequence classification experiments in this paper.

Original: Activity This dataset consists of time series from five individuals performing serveral activities (i.e. walking, standing, laying, etc).
Revised: Activity This dataset consists of time series from five individuals performing serveral activities (i.e. walking, standing, laying, etc).

Original: Each time series includes 12 features indicating tags attached to their belt, chest and ankles.
Revised: Each time series includes 12 features indicating tags attached to their belt, chest and ankles.

Original: We used the same pre-processing as (Rubanova et al., 2019), resulting in 6554 sequences of 211 time points.
Revised: We used the same pre-processing as (Rubanova et al., 2019), resulting in 6554 sequences of 211 time points.

Original: The task is to classify each time point into one of seven types of activities and we performed per-time-point classification experiments as in (Rubanova et al., 2019).
Revised: The task is to classify each time point into one of seven types of activities and we performed per-time-point classification experiments as in (Rubanova et al., 2019).

Original: Plane Vibration
Revised: Plane Vibration

Original: The dataset is consisted of time 0 to 73627, with five attributes recorded per timestamp.
Revised: The dataset is consisted of time 0 to 73627, with five attributes recorded per timestamp.

Original: We randomly take out 10% of data to make the time series irregularly sampled.
Revised: We randomly take out 10% of data to make the time series irregularly sampled.

Original: Following
Revised: Following

Original: 𝐴
Revised: 𝐴

Original: [Figure]
Revised: [Figure]

Original: 𝜑(𝑡) 𝑡 𝑅
Revised: 𝜑(𝑡) 𝑡 𝑅

Original: MLP 𝑆ℎ𝑎𝑟𝑖𝑛𝑔 𝑛𝑒𝑡 𝜑 (1) 𝑅 (1) 𝑅 (2) 𝑅 (3) 𝑅 (4) 𝜑 (2) 𝜑 (3)
Revised: MLP 𝑆ℎ𝑎𝑟𝑖𝑛𝑔 𝑛𝑒𝑡 𝜑 (1) 𝑅 (1) 𝑅 (2) 𝑅 (3) 𝑅 (4) 𝜑 (2) 𝜑 (3)

Original: [Figure]
Revised: [Figure]

Original: the implementation of (Xia et al., 2021), we use the first 50% of data as our train set, the next 25% as a validation set, and the rest as a test set.
Revised: the implementation of (Xia et al., 2021), we use the first 50% of data as our train set, the next 25% as a validation set, and the rest as a test set.

Original: We divide each set into partitions of consecutive 64 timestamps of the irregularly-sampled time series, and our goal is to forecast 8 consecutive timestamps starting from the last timestamp of the segment.
Revised: We divide each set into partitions of consecutive 64 timestamps of the irregularly-sampled time series, and our goal is to forecast 8 consecutive timestamps starting from the last timestamp of the segment.

Original: C. N ETWORK S TRUCTURE
Revised: C. N ETWORK S TRUCTURE

Original: Time Embedding In order to incorporate the time of observations into the model, we adopt a positional embedding of transformer (Vaswani et al., 2017).
Revised: Time Embedding In order to incorporate the time of observations into the model, we adopt a positional embedding of transformer (Vaswani et al., 2017).

Original: A time encoding function converts an one-dimensional time into a multi-dimensional input by passing the time t through trigonometric functions of varying frequencies:
Revised: A time encoding function converts an one-dimensional time into a multi-dimensional input by passing the time t through trigonometric functions of varying frequencies:

Original: [Equation]
Revised: [Equation]

Original: where i runs over a range of integers { 0 , .
Revised: where i runs over a range of integers { 0 , .

Original: , m − 1 } with encoding dimension m .
Revised: , m − 1 } with encoding dimension m .

Original: It allows the model to attend relative time effortlessly and capture long dependencies.
Revised: It allows the model to attend relative time effortlessly and capture long dependencies.

Original: Multi-head Fourier kernel Time modulation multiples a single value φ ℓ ( t, ξ ) ∈ R to a d v × d v
Revised: Multi-head Fourier kernel Time modulation multiples a single value φ ℓ ( t, ξ ) ∈ R to a d v × d v

Original: Fourier kernel R ℓ ( ξ ) , that is,
Revised: Fourier kernel R ℓ ( ξ ) , that is,

Original: [Equation]
Revised: [Equation]

Original: However, often there are multiple different aspects channel elements attend to, and scalar multiplication may not be a good option for it.
Revised: However, often there are multiple different aspects channel elements attend to, and scalar multiplication may not be a good option for it.

Original: To attempt to lift this restriction, we employ parallel heads.
Revised: To attempt to lift this restriction, we employ parallel heads.

Original: Specifically, we divide the Fourier kernel R l into h kernel blocks R ( i ) l ( ξ ) ∈ R
Revised: Specifically, we divide the Fourier kernel R l into h kernel blocks R ( i ) l ( ξ ) ∈ R

Original: d k × d v for i = 1 . . .
Revised: d k × d v for i = 1 . . .

Original: , h , where h is the number of heads and d k = d v /h .
Revised: , h , where h is the number of heads and d k = d v /h .

Original: Afterward, we concatenate the heads and combine them with a final kernel matrix.
Revised: Afterward, we concatenate the heads and combine them with a final kernel matrix.

Original: (See Figure 4.)
Revised: (See Figure 4.)

Original: This operation can be expressed as:
Revised: This operation can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: The multi-head kernel allows the model to share different representations at different channels.
Revised: The multi-head kernel allows the model to share different representations at different channels.

Original: Model architecture for 2D data While the Fourier kernel R plays a key role in FNO models for learning the solution operator of PDEs, it requires quite many parameters ( d 2 v for each frequency).
Revised: Model architecture for 2D data While the Fourier kernel R plays a key role in FNO models for learning the solution operator of PDEs, it requires quite many parameters ( d 2 v for each frequency).

Original: This parameter redundancy worsens for two-dimensional data; images in Section 4.3.
Revised: This parameter redundancy worsens for two-dimensional data; images in Section 4.3.

Original: To overcome parameter inefficiency, we modify the Fourier kernel as follows.
Revised: To overcome parameter inefficiency, we modify the Fourier kernel as follows.

Original: One modification is the use of blockdiagonal structure of R , as proposed in (Guibas et al., 2021).
Revised: One modification is the use of blockdiagonal structure of R , as proposed in (Guibas et al., 2021).

Original: This reduces the number of parameters, however, the diagonal structure does not fully mix the information between channels, which may lessen the expressive power of the model.
Revised: This reduces the number of parameters, however, the diagonal structure does not fully mix the information between channels, which may lessen the expressive power of the model.

Original: To compensate for this, we additionally introduce a local 1 × 1 convolution layer V in Fourier space.
Revised: To compensate for this, we additionally introduce a local 1 × 1 convolution layer V in Fourier space.

Original: The overall architecture can be written as
Revised: The overall architecture can be written as

Original: [Equation]
Revised: [Equation]

Original: We empirically show that this improves the training ability of FNO with significantly less parameters.
Revised: We empirically show that this improves the training ability of FNO with significantly less parameters.

Original: [Table]
Revised: [Table]

Original: Lifting Layers for Real Application Fourier transform may be hard to embody discontinuous functions because Fourier filters are global sinusoidal functions.
Revised: Lifting Layers for Real Application Fourier transform may be hard to embody discontinuous functions because Fourier filters are global sinusoidal functions.

Original: As a consequence, FNO has difficulty in learning sharp features (Lu et al., 2022).
Revised: As a consequence, FNO has difficulty in learning sharp features (Lu et al., 2022).

Original: In many applications, data is discontinuous, such as images.
Revised: In many applications, data is discontinuous, such as images.

Original: Hence, the performance of FNO models is plagued by the discontinuous nature of such input data.
Revised: Hence, the performance of FNO models is plagued by the discontinuous nature of such input data.

Original: Contrary to regular PDE problems where the pointwise lifting is adequate, we hereby propose an alternative lifting layer to provide a smoother form of the data.
Revised: Contrary to regular PDE problems where the pointwise lifting is adequate, we hereby propose an alternative lifting layer to provide a smoother form of the data.

Original: A low pass filter is the standard way of smoothing out the data; it tends to retain the low frequency modes while truncating out the high frequency information.
Revised: A low pass filter is the standard way of smoothing out the data; it tends to retain the low frequency modes while truncating out the high frequency information.

Original: To cutoff high spatial frequency, we additionally introduce a low pass filter after the original lifting layer.
Revised: To cutoff high spatial frequency, we additionally introduce a low pass filter after the original lifting layer.

Original: For parameter efficiency, the low pass filter is implemented by an 1 × 1 convolution layer in Fourier space, as we introduced in the preceding paragraph.
Revised: For parameter efficiency, the low pass filter is implemented by an 1 × 1 convolution layer in Fourier space, as we introduced in the preceding paragraph.

Original: In this way, input data is smoothed by decreasing the disparity between neighboring values without loss of information.
Revised: In this way, input data is smoothed by decreasing the disparity between neighboring values without loss of information.

Original: C. H YPERPARAMETERS
Revised: C. H YPERPARAMETERS

Original: We choose the hyperparameters that delivered the best performance for the baseline models.
Revised: We choose the hyperparameters that delivered the best performance for the baseline models.

Original: Note that each model has different optimal optimizer settings, so we carefully choose the learning rate and its decay rate that show the best performance.
Revised: Note that each model has different optimal optimizer settings, so we carefully choose the learning rate and its decay rate that show the best performance.

Original: C.3. S YNTHETIC
Revised: C.3. S YNTHETIC

Original: Low dimensional data Low dimensional data includes Spiral, Stiff, Sawtooth, and Square.
Revised: Low dimensional data Low dimensional data includes Spiral, Stiff, Sawtooth, and Square.

Original: All the experiments on these datasets are trained for 1000 epochs with a batch size of 120.
Revised: All the experiments on these datasets are trained for 1000 epochs with a batch size of 120.

Original: We do not normalize data since all data are properly bounded.
Revised: We do not normalize data since all data are properly bounded.

Original: Implementation of NODE, ANODE, Latent ODE,
Revised: Implementation of NODE, ANODE, Latent ODE,

Original: Neural Flow, and Neural Laplace follow (Holt et al., 2022), other than the data normalization.
Revised: Neural Flow, and Neural Laplace follow (Holt et al., 2022), other than the data normalization.

Original: For
Revised: For

Original: CTFNO, a single data point is linearly transformed into a 16-dimensional vector.
Revised: CTFNO, a single data point is linearly transformed into a 16-dimensional vector.

Original: Then, we regard the temporal dimension of 100 as a channel size and convert it to 16 by a lifting layer.
Revised: Then, we regard the temporal dimension of 100 as a channel size and convert it to 16 by a lifting layer.

Original: We use the learning rate of 1e-3 with no decay.
Revised: We use the learning rate of 1e-3 with no decay.

Original: We do not use the stabilization constant and use Gaussian error
Revised: We do not use the stabilization constant and use Gaussian error

Original: Linear Units (GeLU) (Hendrycks & Gimpel, 2016) activation in all FNO-based experiments.
Revised: Linear Units (GeLU) (Hendrycks & Gimpel, 2016) activation in all FNO-based experiments.

Original: In the time-embedding layer, we employ Sigmoid Linear Units (SiLU) (Elfwing et al., 2018) activation function.
Revised: In the time-embedding layer, we employ Sigmoid Linear Units (SiLU) (Elfwing et al., 2018) activation function.

Original: Note that all experiments for CTFNO use the same activation function.
Revised: Note that all experiments for CTFNO use the same activation function.

Original: High-dimensional data In Reaction ODE, Heat, and Burgers’ experiments, which are high dimensional datasets
Revised: High-dimensional data In Reaction ODE, Heat, and Burgers’ experiments, which are high dimensional datasets

Original: , we set comparison models to have a large number of parameters, at least 1M.
Revised: , we set comparison models to have a large number of parameters, at least 1M.

Original: In NODE and ANODE experiments, we set an ODE function f
Revised: In NODE and ANODE experiments, we set an ODE function f

Original: ( t, u ( t )) to a five layer Multi-Layer
Revised: ( t, u ( t )) to a five layer Multi-Layer

Original: Perceptron (MLP), of 1024 units.
Revised: Perceptron (MLP), of 1024 units.

Original: For ANODE, we set the augmented dimension of zeros to 100.
Revised: For ANODE, we set the augmented dimension of zeros to 100.

Original: We employ the learning rate of 1e-5, and the number of epochs of 2000.
Revised: We employ the learning rate of 1e-5, and the number of epochs of 2000.

Original: Other hyperparameters follow the
Revised: Other hyperparameters follow the

Original: NODE implementation of Holt et al.
Revised: NODE implementation of Holt et al.

Original: Moreover, in the implementation of Neural Laplace, we set the latent dimension and the number of hidden units to 64.
Revised: Moreover, in the implementation of Neural Laplace, we set the latent dimension and the number of hidden units to 64.

Original: Other hyperparameters are set the same as low-dimensional data experiments.
Revised: Other hyperparameters are set the same as low-dimensional data experiments.

Original: In the implementation of Neural Flow, we use a 32-layered coupling flow.
Revised: In the implementation of Neural Flow, we use a 32-layered coupling flow.

Original: Each coupling flow has two hidden layers with 128 hidden dimensions.
Revised: Each coupling flow has two hidden layers with 128 hidden dimensions.

Original: Timesteps are embedded into 32 sinusoidal features.
Revised: Timesteps are embedded into 32 sinusoidal features.

Original: We set the learning rate of 1e-3 with the decay rate of 0.8 for every 200 epochs.
Revised: We set the learning rate of 1e-3 with the decay rate of 0.8 for every 200 epochs.

Original: Other hyperparameters for Neural Flow follow the synthetic experiments in (Biloš et al., 2021).
Revised: Other hyperparameters for Neural Flow follow the synthetic experiments in (Biloš et al., 2021).

Original: In DON and PDN experiments, we use the number of channels of 256 with six residual layers for each branch and trunk net.
Revised: In DON and PDN experiments, we use the number of channels of 256 with six residual layers for each branch and trunk net.

Original: We use SiLU activation, the training epochs of 20000, the batch size of 50, and the learning rate of 1e-3.
Revised: We use SiLU activation, the training epochs of 20000, the batch size of 50, and the learning rate of 1e-3.

Original: The hyperparameters of FNO-based models, including
Revised: The hyperparameters of FNO-based models, including

Original: CTFNO, are summarized in Table 8.
Revised: CTFNO, are summarized in Table 8.

Original: We train FNO-based models with a batch size of 20, the training epochs of 2000, and the learning rate of 1e-3 with a decay rate of 0.8 for every 100 epochs.
Revised: We train FNO-based models with a batch size of 20, the training epochs of 2000, and the learning rate of 1e-3 with a decay rate of 0.8 for every 100 epochs.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C
Revised: C

Original: [Table] [Table]
Revised: [Table] [Table]

Original: R EAL TIME - SERIES
Revised: R EAL TIME - SERIES

Original: Our implementation builds on open-source codes 1234 (MIT License).
Revised: Our implementation builds on open-source codes 1234 (MIT License).

Original: For all real time-series datasets, we refer to the results of RNN-VAE, ODE-RNN and Latent ODEs reported in (Rubanova et al., 2019).
Revised: For all real time-series datasets, we refer to the results of RNN-VAE, ODE-RNN and Latent ODEs reported in (Rubanova et al., 2019).

Original: For CTFNO, we commonly use the number of heads of one and two padding dimensions.
Revised: For CTFNO, we commonly use the number of heads of one and two padding dimensions.

Original: Note that all layers are equipped with GeLU activation, except for only time-embedding layers with SiLU activation.
Revised: Note that all layers are equipped with GeLU activation, except for only time-embedding layers with SiLU activation.

Original: Other model hyperparameters for CTFNO are shown in Table 9.
Revised: Other model hyperparameters for CTFNO are shown in Table 9.

Original: Moreover, we use the batch size of 50, and the Adamax optimizer (Kingma & Ba, 2014) with the learning rate 1e-2.
Revised: Moreover, we use the batch size of 50, and the Adamax optimizer (Kingma & Ba, 2014) with the learning rate 1e-2.

Original: MuJoCo
Revised: MuJoCo

Original: Overall experimental settings follow (Rubanova et al., 2019), unless stated.
Revised: Overall experimental settings follow (Rubanova et al., 2019), unless stated.

Original: We use the latent dimension of 20, the hidden state dimension of 100 for the RNN encoder, and the training epochs of 300.
Revised: We use the latent dimension of 20, the hidden state dimension of 100 for the RNN encoder, and the training epochs of 300.

Original: Since Biloš et al.
Revised: Since Biloš et al.

Original: (2021) did not contain the results of interpolation and prediction tasks with different subsampling ratios, we carefully trained Neural Flow, following the official implementation 5 .
Revised: (2021) did not contain the results of interpolation and prediction tasks with different subsampling ratios, we carefully trained Neural Flow, following the official implementation 5 .

Original: PhysioNet Overall experimental settings follow (Rubanova et al., 2019), unless stated.
Revised: PhysioNet Overall experimental settings follow (Rubanova et al., 2019), unless stated.

Original: We use the latent dimension of 20 and the hidden state dimension of 40 for the RNN encoder.
Revised: We use the latent dimension of 20 and the hidden state dimension of 40 for the RNN encoder.

Original: The training epochs are 200 and 50 for the interpolation task and the prediction task, respectively.
Revised: The training epochs are 200 and 50 for the interpolation task and the prediction task, respectively.

Original: As in (Rubanova et al., 2019), we employ the importance weighted likelihood loss (Burda et al., 2015) without leveraging classification loss.
Revised: As in (Rubanova et al., 2019), we employ the importance weighted likelihood loss (Burda et al., 2015) without leveraging classification loss.

Original: Activity Overall experimental settings follow (Rubanova et al., 2019), unless stated.
Revised: Activity Overall experimental settings follow (Rubanova et al., 2019), unless stated.

Original: We use the latent dimension of 20, the hidden state dimension of 100 for the RNN encoder, and the training epochs of 200.
Revised: We use the latent dimension of 20, the hidden state dimension of 100 for the RNN encoder, and the training epochs of 200.

Original: For classification, the cross entropy loss was used, as in (Rubanova et al., 2019).
Revised: For classification, the cross entropy loss was used, as in (Rubanova et al., 2019).

Original: experiments
Revised: experiments

Original: [Table]
Revised: [Table]

Original: C.3. S TUDIES ON STABILITY
Revised: C.3. S TUDIES ON STABILITY

Original: Plane Vibration
Revised: Plane Vibration

Original: In this experiment, we follow the settings of Xia et al.
Revised: In this experiment, we follow the settings of Xia et al.

Original: For CTFNO, five-dimensional attributes are concatenated with input time and then embedded into 20-dimensional space by a linear transformation.
Revised: For CTFNO, five-dimensional attributes are concatenated with input time and then embedded into 20-dimensional space by a linear transformation.

Original: We regard temporal dimension 64 as a channel size and convert it to eight by a lifting layer.
Revised: We regard temporal dimension 64 as a channel size and convert it to eight by a lifting layer.

Original: We train the model for 500 epochs and use the learning rate of 1e-2 with decay rate 0.5 per 100 epochs.
Revised: We train the model for 500 epochs and use the learning rate of 1e-2 with decay rate 0.5 per 100 epochs.

Original: Image classification All of the experiments use gradient clipping of 10 and cross-entropy loss.
Revised: Image classification All of the experiments use gradient clipping of 10 and cross-entropy loss.

Original: We use a learning rate of 1e-3 and weight decay of 1e-4 for ANODE and CNN.
Revised: We use a learning rate of 1e-3 and weight decay of 1e-4 for ANODE and CNN.

Original: Other implementation details of ANODE follow Dupont et al.
Revised: Other implementation details of ANODE follow Dupont et al.

Original: We use five convolutional layers in CNN experiments.
Revised: We use five convolutional layers in CNN experiments.

Original: For MNIST, we apply a kernel size of seven for the first two convolutional layers and 32 number of channels.
Revised: For MNIST, we apply a kernel size of seven for the first two convolutional layers and 32 number of channels.

Original: For CIFAR10, we use a kernel size of seven for the first convolutional layers andnumber of channels.
Revised: For CIFAR10, we use a kernel size of seven for the first convolutional layers andnumber of channels.

Original: Other convolutional layers have a kernel size of three.
Revised: Other convolutional layers have a kernel size of three.

Original: We train CNN forepochs.
Revised: We train CNN forepochs.

Original: Note that accuracy of both ANODE and CNN tend to decrease after the designated number of epochs.
Revised: Note that accuracy of both ANODE and CNN tend to decrease after the designated number of epochs.

Original: In FNO experiments, FNO with weight decay (WD) and FNO with normalization (Stab), we train models for 100 epochs with the learning rate of 5e-3.
Revised: In FNO experiments, FNO with weight decay (WD) and FNO with normalization (Stab), we train models for 100 epochs with the learning rate of 5e-3.

Original: We used three Fourier layers with six Fourier modes.
Revised: We used three Fourier layers with six Fourier modes.

Original: For MNIST, we employ the number of channel of 32, and three-dimensional zero padding.
Revised: For MNIST, we employ the number of channel of 32, and three-dimensional zero padding.

Original: For CIFAR10, we employ the number of channel of 64, and four-dimensional zero padding.
Revised: For CIFAR10, we employ the number of channel of 64, and four-dimensional zero padding.

Original: For the experiments with decay, we choose the decay rates 1e-6 and 1e-4 for MNIST and CIFAR10, respectively.
Revised: For the experiments with decay, we choose the decay rates 1e-6 and 1e-4 for MNIST and CIFAR10, respectively.

Original: For the experiments with normalization, we employ the spectral norm of 1.2 for the normalization bounds.
Revised: For the experiments with normalization, we employ the spectral norm of 1.2 for the normalization bounds.

Original: Note that GeLU activation function was used for both CNN and FNO experiments.
Revised: Note that GeLU activation function was used for both CNN and FNO experiments.

Original: To estimate the robustness of the trained model, we consider three commonly-used perturbation schemes, namely random Gaussian perturbations, FGSM, and PGD attacks.
Revised: To estimate the robustness of the trained model, we consider three commonly-used perturbation schemes, namely random Gaussian perturbations, FGSM, and PGD attacks.

Original: For random Gaussian attack, we use zero mean Gaussian noise with standard deviation of 50/255, 100/255 for MNIST and 15/255, 20/255 for CIFAR10.
Revised: For random Gaussian attack, we use zero mean Gaussian noise with standard deviation of 50/255, 100/255 for MNIST and 15/255, 20/255 for CIFAR10.

Original: For adversarial attacks, we use 30/255, 50/255 attack for MNIST and 2/255, 5/255 for CIFAR10.
Revised: For adversarial attacks, we use 30/255, 50/255 attack for MNIST and 2/255, 5/255 for CIFAR10.

Original: The number of steps of PGD attack is 10.
Revised: The number of steps of PGD attack is 10.

Original: D F URTHER R ESULTS D. C OMPARISON TO VARIOUS TIME - INJECTED FNO- BASED MODELS
Revised: D F URTHER R ESULTS D. C OMPARISON TO VARIOUS TIME - INJECTED FNO- BASED MODELS

Original: In Section 3.2, we construct various baselines injecting time t into the network for comparative analysis of CTFNO.
Revised: In Section 3.2, we construct various baselines injecting time t into the network for comparative analysis of CTFNO.

Original: This section includes implementation details.
Revised: This section includes implementation details.

Original: Since CTFNO puts t into weights of the Fourier layers, the other way to inject t is to join it in the input function u 0 ( x ) or intermediate features v 0 ( x ) , v 1 ( x ) , . .
Revised: Since CTFNO puts t into weights of the Fourier layers, the other way to inject t is to join it in the input function u 0 ( x ) or intermediate features v 0 ( x ) , v 1 ( x ) , . .

Original: . , v L ( x ) .
Revised: . , v L ( x ) .

Original: To cover variants of such cases, we constructed three other baselines adding t into the input or feature space as discussed in Section 3.2.
Revised: To cover variants of such cases, we constructed three other baselines adding t into the input or feature space as discussed in Section 3.2.

Original: The architecture of these three scenarios is pictorized in Figure 5.
Revised: The architecture of these three scenarios is pictorized in Figure 5.

Original: Comparison results of these three baselines together with FNO2d (proposed in the original FNO paper) and CTFNO in ODE/PDE problems are reported in Table 11.
Revised: Comparison results of these three baselines together with FNO2d (proposed in the original FNO paper) and CTFNO in ODE/PDE problems are reported in Table 11.

Original: The results show that our model significantly outperforms other models.
Revised: The results show that our model significantly outperforms other models.

Original: The results further show that three scenarios, which are structures that can continuously handle arbitrary time t , are similar to or worse than FNO2d that can only evaluate the solution value at times on a specific grid.
Revised: The results further show that three scenarios, which are structures that can continuously handle arbitrary time t , are similar to or worse than FNO2d that can only evaluate the solution value at times on a specific grid.

Original: Even though all three baselines and
Revised: Even though all three baselines and

Original: CTFNO model continuous time operators, it was found through the experiments which only the structure of CTFNO is effective.
Revised: CTFNO model continuous time operators, it was found through the experiments which only the structure of CTFNO is effective.

Original: The difference from the baselines is that CTFNO inserts temporal information into weights.
Revised: The difference from the baselines is that CTFNO inserts temporal information into weights.

Original: The rationale of the way of imposing temporal information on CTFNO is based on Green’s formula of time-dependent PDEs.
Revised: The rationale of the way of imposing temporal information on CTFNO is based on Green’s formula of time-dependent PDEs.

Original: The results validate that it is much more effective to construct a model structure based on the theoretical Green’s formula to approximate
Revised: The results validate that it is much more effective to construct a model structure based on the theoretical Green’s formula to approximate

Original: the
Revised: the

Original: 𝑃 𝑢 0 (𝑥) 𝑠ℎ𝑎𝑟𝑖𝑛𝑔 𝑛𝑒𝑡 𝜑(𝑡) 𝑢(𝑡,𝑥) 𝑄 𝑡 𝑣 0 (𝑡,𝑥) 𝑣 1 (𝑡,𝑥) ℒ𝑣 2 (𝑡,𝑥) ℒ𝑣 3 (𝑡,𝑥) ℒ𝐴 1 𝐴𝐴𝐴𝑃 𝑢 0 (𝑥) 𝑠ℎ𝑎𝑟𝑖𝑛𝑔 𝑛𝑒𝑡 𝜑(𝑡) 𝑢(𝑡,𝑥) 𝑄 𝑡 𝑣 0 (𝑡,𝑥) 𝑣 1 (𝑡,𝑥) ℒ𝑣 2 (𝑡,𝑥) ℒ𝑣 3 (𝑡,𝑥) ℒ𝐴𝑃 𝑢 0 (t,𝑥) 𝑢(𝑡,𝑥)
Revised: 𝑃 𝑢 0 (𝑥) 𝑠ℎ𝑎𝑟𝑖𝑛𝑔 𝑛𝑒𝑡 𝜑(𝑡) 𝑢(𝑡,𝑥) 𝑄 𝑡 𝑣 0 (𝑡,𝑥) 𝑣 1 (𝑡,𝑥) ℒ𝑣 2 (𝑡,𝑥) ℒ𝑣 3 (𝑡,𝑥) ℒ𝐴 1 𝐴𝐴𝐴𝑃 𝑢 0 (𝑥) 𝑠ℎ𝑎𝑟𝑖𝑛𝑔 𝑛𝑒𝑡 𝜑(𝑡) 𝑢(𝑡,𝑥) 𝑄 𝑡 𝑣 0 (𝑡,𝑥) 𝑣 1 (𝑡,𝑥) ℒ𝑣 2 (𝑡,𝑥) ℒ𝑣 3 (𝑡,𝑥) ℒ𝐴𝑃 𝑢 0 (t,𝑥) 𝑢(𝑡,𝑥)

Original: 𝑄 𝑡 𝑣 0 (𝑡,𝑥) 𝑣 1 (𝑡,𝑥) ℒ𝑣 2 (𝑡,𝑥) ℒ𝑣 3 (𝑡,𝑥) ℒ
Revised: 𝑄 𝑡 𝑣 0 (𝑡,𝑥) 𝑣 1 (𝑡,𝑥) ℒ𝑣 2 (𝑡,𝑥) ℒ𝑣 3 (𝑡,𝑥) ℒ

Original: [Figure]
Revised: [Figure]

Original: Scenario
Revised: Scenario

Original: [Figure]
Revised: [Figure]

Original: solution operator of PDE.
Revised: solution operator of PDE.

Original: In summary, we validate through additional experiments that the structure of CTFNO designed based on the solution operator of the PDE (what we want to learn) is very effective in actually learning the PDE.
Revised: In summary, we validate through additional experiments that the structure of CTFNO designed based on the solution operator of the PDE (what we want to learn) is very effective in actually learning the PDE.

Original: [Table]
Revised: [Table]

Original: Heat
Revised: Heat

Original: Burgers
Revised: Burgers

Original: [Table]
Revised: [Table]

Original: D. S TABILITY
Revised: D. S TABILITY

Original: As we discussed in Section 3.4, the upper bound M on the ∥ R ( t, · ) ∥ 2 and ∥ W ( t, · ) ∥ 2 emanates from the necessity of stabilizing the proposed model.
Revised: As we discussed in Section 3.4, the upper bound M on the ∥ R ( t, · ) ∥ 2 and ∥ W ( t, · ) ∥ 2 emanates from the necessity of stabilizing the proposed model.

Original: In Section 4.3, we conducted experiments to evaluate the robustness of our proposed stabilization scheme, and compare it with the vanilla FNO and weight decay.
Revised: In Section 4.3, we conducted experiments to evaluate the robustness of our proposed stabilization scheme, and compare it with the vanilla FNO and weight decay.

Original: From Tables 5 and 6, FNO trained with stabilization outperforms vanilla FNO on all types of perturbations.
Revised: From Tables 5 and 6, FNO trained with stabilization outperforms vanilla FNO on all types of perturbations.

Original: M is a user-defined hyperparameter.
Revised: M is a user-defined hyperparameter.

Original: Here, we analyze the performance of FNO to illustrate how increasing stabilization parameter M leads to a more robust model.
Revised: Here, we analyze the performance of FNO to illustrate how increasing stabilization parameter M leads to a more robust model.

Original: Experiments are carried out on
Revised: Experiments are carried out on

Original: CIFAR

Revised: CIFAR

Original: and we perturb images by PGD-5/255 attack.
Revised: and we perturb images by PGD-5/255 attack.

Original: The left panel of Figure 6 shows the accuracy and robustness of the learned models for several values of M .
Revised: The left panel of Figure 6 shows the accuracy and robustness of the learned models for several values of M .

Original: We can see that imposing stability does come with trade-offs between test accuracy and robustness on the attack.
Revised: We can see that imposing stability does come with trade-offs between test accuracy and robustness on the attack.

Original: The accuracy is higher at low M and it becomes decreasing as M increases, which makes the model more robust on various attacks.
Revised: The accuracy is higher at low M and it becomes decreasing as M increases, which makes the model more robust on various attacks.

Original: Moreover, the models with weak stabilization show brittle training procedures.
Revised: Moreover, the models with weak stabilization show brittle training procedures.

Original: Also, the model degrades the overall performance as stabilization effects become dominant.
Revised: Also, the model degrades the overall performance as stabilization effects become dominant.

Original: Results certify that small corruptions or extra noise in the input are not likely to change the output of the network with proper stabilization.
Revised: Results certify that small corruptions or extra noise in the input are not likely to change the output of the network with proper stabilization.

Original: Moreover, the stability of the model is related to how well the model generalizes on the data on which it has not trained, which in turn is related to overfitting.
Revised: Moreover, the stability of the model is related to how well the model generalizes on the data on which it has not trained, which in turn is related to overfitting.

Original: This can be confirmed in Figure 6 (right).
Revised: This can be confirmed in Figure 6 (right).

Original: It presents the evolution of the loss function of models with and without stabilization.
Revised: It presents the evolution of the loss function of models with and without stabilization.

Original: As one can see, after some iterations, the test accuracy of the model without stabilization has started to decrease.
Revised: As one can see, after some iterations, the test accuracy of the model without stabilization has started to decrease.

Original: This means that the longer we train the model, the more specialized the weights will become to the training data.
Revised: This means that the longer we train the model, the more specialized the weights will become to the training data.

Original: This is evidence of overfitting.
Revised: This is evidence of overfitting.

Original: On the other hand, the model with stabilization keeps increasing its test accuracy.
Revised: On the other hand, the model with stabilization keeps increasing its test accuracy.

Original: This confirms that the stabilization prevents overfitting and helps the model to work better on unseen data.
Revised: This confirms that the stabilization prevents overfitting and helps the model to work better on unseen data.

Original: The relation between the stabilization and overfitting was also discussed in Section 4.3 on the Plane
Revised: The relation between the stabilization and overfitting was also discussed in Section 4.3 on the Plane

Original: Vibration dataset.
Revised: Vibration dataset.

Original: By increasing M , the training MSE loss gradually decreases, while the test MSE eventually goes up (see Figure 3).
Revised: By increasing M , the training MSE loss gradually decreases, while the test MSE eventually goes up (see Figure 3).

Original: Figure 7 plots the distribution of the L 2 norm of weights of trained CTFNO with M = 1 . 2 and 10 on the plane vibration data.
Revised: Figure 7 plots the distribution of the L 2 norm of weights of trained CTFNO with M = 1 . 2 and 10 on the plane vibration data.

Original: The weights of the model trained with M = 10 have larger L 2 norms.
Revised: The weights of the model trained with M = 10 have larger L 2 norms.

Original: As we analyzed in Theorem 3.2, the weights that have a large spectrum make the network unstable.
Revised: As we analyzed in Theorem 3.2, the weights that have a large spectrum make the network unstable.

Original: More precisely, the spectrum of the weights grows in size to handle the specifics of the training data.
Revised: More precisely, the spectrum of the weights grows in size to handle the specifics of the training data.

Original: As the weights become specialized to the training data, overfitting occurs.
Revised: As the weights become specialized to the training data, overfitting occurs.

Original: On the other hand, the trained model with M = 1 . 2 learned weights with a spectrum smaller than 1.
Revised: On the other hand, the trained model with M = 1 . 2 learned weights with a spectrum smaller than 1.

Original: It forces the network to have small changes in output for small changes in the inputs, which gives more ability to generalize better.
Revised: It forces the network to have small changes in output for small changes in the inputs, which gives more ability to generalize better.

Original: D. G ENERALIZATION TO RESOLUTION
Revised: D. G ENERALIZATION TO RESOLUTION

Original: [Table]
Revised: [Table]

Original: MNIST
Revised: MNIST

Original: [Table]
Revised: [Table]

Original: CIFAR
Revised: CIFAR

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Classical neural networks that map between finitedimensional Euclidean spaces are grid-dependent, and thus cannot generalize over image resolution.
Revised: Classical neural networks that map between finitedimensional Euclidean spaces are grid-dependent, and thus cannot generalize over image resolution.

Original: By viewing images as functions, FNO is not tied to a specific discretization and can be applied to arbitrary discretization of a ( x ) .
Revised: By viewing images as functions, FNO is not tied to a specific discretization and can be applied to arbitrary discretization of a ( x ) .

Original: To validate the resolution invariance of FNO, we classify high-resolution images using FNO trained on the original low-resolution images.
Revised: To validate the resolution invariance of FNO, we classify high-resolution images using FNO trained on the original low-resolution images.

Original: Tablevalidates that FNO is able to learn from coarse images and generalize to higher resolutions.
Revised: Tablevalidates that FNO is able to learn from coarse images and generalize to higher resolutions.

Original: D. S PEED IMPROVEMENTS
Revised: D. S PEED IMPROVEMENTS

Original: [Table]
Revised: [Table]

Original: We carefully measured execution time on the MuJoCo experiment (§4.2) and the results are reported in Table 13.
Revised: We carefully measured execution time on the MuJoCo experiment (§4.2) and the results are reported in Table 13.

Original: The main reason for this difference is that numerical integration used in neural ODEs is computationally expensive, and Neural Flow and CTFNO are neural operators that directly represent the solution trajectory of ODE and PDE, respectively.
Revised: The main reason for this difference is that numerical integration used in neural ODEs is computationally expensive, and Neural Flow and CTFNO are neural operators that directly represent the solution trajectory of ODE and PDE, respectively.

Original: In other words, the neural operator’s replacement of the role of ODE solvers has contributed significantly to overcoming the increased complexity of PDEs compared to ODEs and reducing time costs.
Revised: In other words, the neural operator’s replacement of the role of ODE solvers has contributed significantly to overcoming the increased complexity of PDEs compared to ODEs and reducing time costs.

Original: D. A DDITIONAL S YNTHETIC R ESULTS
Revised: D. A DDITIONAL S YNTHETIC R ESULTS

Original: We include additional heatmaps for reaction, heat, and Burgers’ equations (§4.1).
Revised: We include additional heatmaps for reaction, heat, and Burgers’ equations (§4.1).

Original: See Figure 8, 9, and 10.
Revised: See Figure 8, 9, and 10.

Original: E B ROADER I MPACT
Revised: E B ROADER I MPACT

Original: We introduced a new framework for modeling temporal dynamics of observed data.
Revised: We introduced a new framework for modeling temporal dynamics of observed data.

Original: It has a wide range of potential applications, some of which we investigated in this paper.
Revised: It has a wide range of potential applications, some of which we investigated in this paper.

Original: We explored a healthcare dataset and here we hope to bring affirmative impact in medical applications.
Revised: We explored a healthcare dataset and here we hope to bring affirmative impact in medical applications.

Original: We also examined the modeling the vibration of an airplane and we expect it to contribute to the advancement of airplane designs and civil engineering.
Revised: We also examined the modeling the vibration of an airplane and we expect it to contribute to the advancement of airplane designs and civil engineering.

Original: Furthermore, as many problems have arisen in sciences and engineering tied with complex PDE systems, we expect that our work has potential applicability in the enormous area such as climate forecasting, epidemics, molecular simulations, micro-mechanics, and modeling turbulent flows.
Revised: Furthermore, as many problems have arisen in sciences and engineering tied with complex PDE systems, we expect that our work has potential applicability in the enormous area such as climate forecasting, epidemics, molecular simulations, micro-mechanics, and modeling turbulent flows.

Original: PDEs can also be applied in the development of military equipment.
Revised: PDEs can also be applied in the development of military equipment.

Original: As with all numerical methods, however, it is not a work of developing a technique to go to warfare, and we hope and encourage users of our model to concenter on the positive impact of this work.
Revised: As with all numerical methods, however, it is not a work of developing a technique to go to warfare, and we hope and encourage users of our model to concenter on the positive impact of this work.