Original: PDE-N ET : L EARNING PDE S FROM D ATA
Revised: PDE-N ET : L EARNING PDE S FROM D ATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Partial differential equations (PDEs) play a prominent role in many disciplines such as applied mathematics, physics, chemistry, material science, computer science, etc.
Revised: Partial differential equations (PDEs) play a prominent role in many disciplines such as applied mathematics, physics, chemistry, material science, computer science, etc.

Original: PDEs are commonly derived based on physical laws or empirical observations.
Revised: PDEs are commonly derived based on physical laws or empirical observations.

Original: However, the governing equations for many complex systems in modern applications are still not fully known.
Revised: However, the governing equations for many complex systems in modern applications are still not fully known.

Original: With the rapid development of sensors, computational power, and data storage in the past decade, huge quantities of data can be easily collected and efﬁciently stored.
Revised: With the rapid development of sensors, computational power, and data storage in the past decade, huge quantities of data can be easily collected and efﬁciently stored.

Original: Such vast quantity of data offers new opportunities for data-driven discovery of hidden physical laws.
Revised: Such vast quantity of data offers new opportunities for data-driven discovery of hidden physical laws.

Original: Inspired by the latest development of neural network designs in deep learning, we propose a new feed-forward deep network, called PDE-Net, to fulﬁll two objectives at the same time: to accurately predict dynamics of complex systems and to uncover the underlying hidden PDE models.
Revised: Inspired by the latest development of neural network designs in deep learning, we propose a new feed-forward deep network, called PDE-Net, to fulﬁll two objectives at the same time: to accurately predict dynamics of complex systems and to uncover the underlying hidden PDE models.

Original: The basic idea of the proposed PDE-Net is to learn differential operators by learning convolution kernels (ﬁlters), and apply neural networks or other machine learning methods to approximate the unknown nonlinear responses.
Revised: The basic idea of the proposed PDE-Net is to learn differential operators by learning convolution kernels (ﬁlters), and apply neural networks or other machine learning methods to approximate the unknown nonlinear responses.

Original: Comparing with existing approaches, which either assume the form of the nonlinear response is known or ﬁx certain ﬁnite difference approximations of differential operators, our approach has the most ﬂexibility by learning both differential operators and the nonlinear responses.
Revised: Comparing with existing approaches, which either assume the form of the nonlinear response is known or ﬁx certain ﬁnite difference approximations of differential operators, our approach has the most ﬂexibility by learning both differential operators and the nonlinear responses.

Original: A special feature of the proposed PDE-Net is that all ﬁlters are properly constrained, which enables us to easily identify the governing PDE models while still maintaining the expressive and predictive power of the network.
Revised: A special feature of the proposed PDE-Net is that all ﬁlters are properly constrained, which enables us to easily identify the governing PDE models while still maintaining the expressive and predictive power of the network.

Original: These constrains are carefully designed by fully exploiting the relation between the orders of differential operators and the orders of sum rules of ﬁlters (an important concept originated from wavelet theory).
Revised: These constrains are carefully designed by fully exploiting the relation between the orders of differential operators and the orders of sum rules of ﬁlters (an important concept originated from wavelet theory).

Original: We also discuss relations of the PDE-Net with some existing networks in computer vision such as Network-In-Network (NIN) and Residual Neural Network (ResNet).
Revised: We also discuss relations of the PDE-Net with some existing networks in computer vision such as Network-In-Network (NIN) and Residual Neural Network (ResNet).

Original: Numerical experiments show that the PDE-Net has the potential to uncover the hidden PDE of the observed dynamics, and predict the dynamical behavior for a relatively long time, even in a noisy environment.
Revised: Numerical experiments show that the PDE-Net has the potential to uncover the hidden PDE of the observed dynamics, and predict the dynamical behavior for a relatively long time, even in a noisy environment.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Differential equations, especially partial differential equations(PDEs), play a prominent role in many disciplines to describe the governing physical laws underlying a given system of interest.
Revised: Differential equations, especially partial differential equations(PDEs), play a prominent role in many disciplines to describe the governing physical laws underlying a given system of interest.

Original: Traditionally, PDEs are derived based on simple physical principles such as conservation laws, minimum energy principles, or based on empirical observations.
Revised: Traditionally, PDEs are derived based on simple physical principles such as conservation laws, minimum energy principles, or based on empirical observations.

Original: Important examples include the NavierStokes equations in ﬂuid dynamics, the Maxwell’s equations for electromagnetic propagation, and the Schr¨odinger’s equations in quantum mechanics.
Revised: Important examples include the NavierStokes equations in ﬂuid dynamics, the Maxwell’s equations for electromagnetic propagation, and the Schr¨odinger’s equations in quantum mechanics.

Original: However, many complex systems in modern applications (such as many problems in climate science, neuroscience, ﬁnance, etc.) still have eluded mechanisms, and the governing equations of these systems are only partially known.
Revised: However, many complex systems in modern applications (such as many problems in climate science, neuroscience, ﬁnance, etc.) still have eluded mechanisms, and the governing equations of these systems are only partially known.

Original: With the rapid development of sensors, computational power, and data storage in the last decade, huge quantities of data can be easily collected and efﬁciently stored .
Revised: With the rapid development of sensors, computational power, and data storage in the last decade, huge quantities of data can be easily collected and efﬁciently stored .

Original: Such vast quantity of data offers new opportunities for data-driven discovery of potentially new physical laws.
Revised: Such vast quantity of data offers new opportunities for data-driven discovery of potentially new physical laws.

Original: Then, one may ask the following interesting and intriguing question: can we learn a PDE model (if there exists one) from a given data set and perform accurate and efﬁcient predictions using the learned model?
Revised: Then, one may ask the following interesting and intriguing question: can we learn a PDE model (if there exists one) from a given data set and perform accurate and efﬁcient predictions using the learned model?

Original: One of earlier attempts on data-driven discovery of hidden physical laws is by Bongard & Lipson (2007) and Schmidt & Lipson (2009).
Revised: One of earlier attempts on data-driven discovery of hidden physical laws is by Bongard & Lipson (2007) and Schmidt & Lipson (2009).

Original: Their main idea is to compare numerical differentiations of the experimental data with analytic derivatives of candidate functions, and apply the symbolic regression and the evolutionary algorithm to determining the nonlinear dynamical system.
Revised: Their main idea is to compare numerical differentiations of the experimental data with analytic derivatives of candidate functions, and apply the symbolic regression and the evolutionary algorithm to determining the nonlinear dynamical system.

Original: Recently,
Revised: Recently,

Original: Brunton et al.
Revised: Brunton et al.

Original: (2016), Schaeffer (2017), Rudy et al.
Revised: (2016), Schaeffer (2017), Rudy et al.

Original: and Wu & Zhang (2017) propose an alternative approach using sparse regression.
Revised: and Wu & Zhang (2017) propose an alternative approach using sparse regression.

Original: They construct a dictionary of simple functions and partial derivatives that are likely to appear in the unknown governing equations.
Revised: They construct a dictionary of simple functions and partial derivatives that are likely to appear in the unknown governing equations.

Original: Then, they take advantage of sparsity promoting techniques to select candidates that most accurately represent the data.
Revised: Then, they take advantage of sparsity promoting techniques to select candidates that most accurately represent the data.

Original: When the form of the nonlinear response of a PDE is known, except for some scalar parameters, Raissi & Karniadakis (2017) presented a framework to learn these unknown parameters by introducing regularity between two consecutive time step using Gaussian process. 
Revised: When the form of the nonlinear response of a PDE is known, except for some scalar parameters, Raissi & Karniadakis (2017) presented a framework to learn these unknown parameters by introducing regularity between two consecutive time step using Gaussian process. More recently, Raissi et al.

Revised: (2017) introduced a new class of universal function approximators called the physics informed neural networks which is capable of discovering nonlinear PDEs parameterized by scalars.

Original: These recent work greatly advanced the progress of the problem.
Revised: These recent work greatly advanced the progress of the problem.

Original: However, symbolic regression is expensive and does not scale very well to large systems.
Revised: However, symbolic regression is expensive and does not scale very well to large systems.

Original: The sparse regression method requires to ﬁx certain numerical approximations of the spatial differentiations in the dictionary beforehand, which limits the expressive and predictive power of the dictionary.
Revised: The sparse regression method requires to ﬁx certain numerical approximations of the spatial differentiations in the dictionary beforehand, which limits the expressive and predictive power of the dictionary.

Original: Although the framework presented by
Revised: Although the framework presented by

Original: Raissi & Karniadakis  (2017) is able to learn hidden physical laws using less data than the approach of sparse regression, the explicit form of the PDEs are assumed to be known except for a few scalar learnable parameters.
Revised: Raissi & Karniadakis (2017); Raissi et al. (2017) is able to learn hidden physical laws using less data than the approach of sparse regression, the explicit form of the PDEs are assumed to be known except for a few scalar learnable parameters.

Original: Therefore, extracting governing equations from data in a less restrictive setting remains a great challenge.
Revised: Therefore, extracting governing equations from data in a less restrictive setting remains a great challenge.

Original: The main objective of this paper is to accurately predict the dynamics of complex systems and to uncover the underlying hidden PDE models (should they exist) at the same time, with minimal prior knowledge on the systems.
Revised: The main objective of this paper is to accurately predict the dynamics of complex systems and to uncover the underlying hidden PDE models (should they exist) at the same time, with minimal prior knowledge on the systems.

Original: Our inspiration comes from the latest development of deep learning techniques in computer vision.
Revised: Our inspiration comes from the latest development of deep learning techniques in computer vision.

Original: An interesting fact is that some popular networks in computer vision, such as ResNet(He et al., 2016a;b), have close relationship with PDEs (E, 2017; Haber & Ruthotto, 2017; Sonoda & Murata, 2017; Lu et al., 2017).
Revised: An interesting fact is that some popular networks in computer vision, such as ResNet(He et al., 2016a;b), have close relationship with PDEs (Chen et al., 2015; E, 2017; Haber & Ruthotto, 2017; Sonoda & Murata, 2017; Lu et al., 2017).

Original: Furthermore, the deeper is the network, the more expressive power the network possesses, which may enable us to learn more complex dynamics arose from ﬁelds other than computer vision.
Revised: Furthermore, the deeper is the network, the more expressive power the network possesses, which may enable us to learn more complex dynamics arose from ﬁelds other than computer vision.

Original: However, existing deep networks designed in deep learning mostly emphasis on expressive power and prediction accuracy.
Revised: However, existing deep networks designed in deep learning mostly emphasis on expressive power and prediction accuracy.

Original: These networks are not transparent enough to be able to reveal the underlying PDE models, although they may perfectly ﬁt the observed data and perform accurate predictions.
Revised: These networks are not transparent enough to be able to reveal the underlying PDE models, although they may perfectly ﬁt the observed data and perform accurate predictions.

Original: Therefore, we need to carefully design the network by combining knowledge from deep learning and applied mathematics so that we can learn the governing PDEs of the dynamics and make accurate predictions at the same time.
Revised: Therefore, we need to carefully design the network by combining knowledge from deep learning and applied mathematics so that we can learn the governing PDEs of the dynamics and make accurate predictions at the same time.

Revised: Note that our work is closely related to Chen et al. (2015) where the authors designed their network based on discretization of quasilinear parabolic equations.

Revised: However, it is not clear if the dynamics of image denoising has to be governed by PDEs, nor did the authors attempt to recover the PDE (should there exists one).

Original: In this paper, we design a deep feed-forward network, named PDE-Net, based on the following generic nonlinear evolution PDE
Revised: In this paper, we design a deep feed-forward network, named PDE-Net, based on the following generic nonlinear evolution PDE

Original: [Equation]
Revised: [Equation]

Original: The objective of the PDE-Net is to learn the form of the nonlinear response F and to perform accurate predictions.
Revised: The objective of the PDE-Net is to learn the form of the nonlinear response F and to perform accurate predictions.

Original: Unlike the existing work, the proposed network only requires minor knowledge on the form of the nonlinear response function F , and requires no knowledge on the involved differential operators (except for their maximum possible order) and their associated discrete approximations.
Revised: Unlike the existing work, the proposed network only requires minor knowledge on the form of the nonlinear response function F , and requires no knowledge on the involved differential operators (except for their maximum possible order) and their associated discrete approximations.

Original: The nonlinear response function F can be learned using neural networks or other machine learning methods, while discrete approximations of the differential operators are learned using convolution kernels (i.e. ﬁlters) jointly with the learning of the response function F .
Revised: The nonlinear response function F can be learned using neural networks or other machine learning methods, while discrete approximations of the differential operators are learned using convolution kernels (i.e. ﬁlters) jointly with the learning of the response function F .

Original: If we have a prior knowledge on the form of the response function F , we can easily adjust the network architecture by taking advantage of the additional information.
Revised: If we have a prior knowledge on the form of the response function F , we can easily adjust the network architecture by taking advantage of the additional information.

Original: This may simplify the training and improve the results.
Revised: This may simplify the training and improve the results.

Original: We will also discuss relations of the PDE-Net to some existing networks in computer vision such as Network-In-Network (NIN) and ResNet.
Revised: We will also discuss relations of the PDE-Net to some existing networks in computer vision such as Network-In-Network (NIN) and ResNet.

Original: Details are given in Section 2.
Revised: Details are given in Section 2.

Original: In Section 3 and Section 4, we conduct numerical experiments on a linear PDE (convection-diffusion equation) and a nonlinear PDE (convection-diffusion equation with a nonlinear source).
Revised: In Section 3 and Section 4, we conduct numerical experiments on a linear PDE (convection-diffusion equation) and a nonlinear PDE (convection-diffusion equation with a nonlinear source).

Original: We generate data set for each PDE using high precision numerical methods and add Gaussian noise to mimic real situations.
Revised: We generate data set for each PDE using high precision numerical methods and add Gaussian noise to mimic real situations.

Original: Our numerical results show that the PDE-Net can uncover the hidden equations of the observed dynamics, and can predict the dynamical behavior for a relatively long time, even in a noisy environment.
Revised: Our numerical results show that the PDE-Net can uncover the hidden equations of the observed dynamics, and can predict the dynamical behavior for a relatively long time, even in a noisy environment.

Original: A particular novelty of our approach is that we impose appropriate constraints on the learnable ﬁlters in order to easily identify the governing PDE models while still maintaining the expressive and predictive power of the network.
Revised: A particular novelty of our approach is that we impose appropriate constraints on the learnable ﬁlters in order to easily identify the governing PDE models while still maintaining the expressive and pre- dictive power of the network.

Original: This makes our approach different from existing deep convolutional networks which mostly emphasis on the prediction accuracy of the networks, as well as all the existing approaches of learning PDEs from data which assume either the form of the response function is known or have ﬁxed approximations of the differential operators.
Revised: This makes our approach different from existing deep convolutional networks which mostly emphasis on the prediction accuracy of the networks, as well as all the existing approaches of learning PDEs from data which assume either the form of the response function is known or have ﬁxed approximations of the differential operators.

Original: In other words, our proposed approach not only has vast ﬂexibility in ﬁtting observed dynamics and is able to accurately predict its future behavior, but is also able to reveal the hidden equations driving the observed dynamics.
Revised: In other words, our proposed approach not only has vast ﬂexibility in ﬁtting observed dynamics and is able to accurately predict its future behavior, but is also able to reveal the hidden equations driving the observed dynamics.

Original: The constraints on the ﬁlters are motivated by the earlier work of Cai et al.
Revised: The constraints on the ﬁlters are motivated by the earlier work of Cai et al.

Original: ; Dong et al.
Revised: ; Dong et al.

Original: (2017) where general relations between wavelet frame transforms and differential operators were established.
Revised: (2017) where general relations between wavelet frame transforms and differential operators were established.

Original: In particular, it was observed in Dong et al.
Revised: In particular, it was observed in Dong et al.

Original: (2017) that we can relate ﬁlters and ﬁnite difference approximation of differential operators by examining the orders of sum rules of the ﬁlters (an important concept in wavelet theory and closely related to vanishing moments of wavelet functions).
Revised: (2017) that we can relate ﬁlters and ﬁnite difference approximation of differential operators by examining the orders of sum rules of the ﬁlters (an important concept in wavelet theory and closely related to vanishing moments of wavelet functions).

Original: These constraints on the ﬁlters may also be useful in network designs for machine learning tasks in computer vision.
Revised: These constraints on the ﬁlters may also be useful in network designs for machine learning tasks in computer vision.

Original:  PDE-N ET : A F LEXIBLE D EEP A RCHTECTURE TO L EARN PDE S FROM D ATA
Revised:  PDE-N ET : A F LEXIBLE D EEP A RCHTECTURE TO L EARN PDE S FROM D ATA

Original: Given a series of measurements of some physical quantities { u ( t, · ) :
Revised: Given a series of measurements of some physical quantities { u ( t, · ) :

Original: t = t 0 , t 1 , · · · } on the spatial domain Ω ⊂ R 2 , with u ( t, · ) : Ω (cid:55)→ R , we want to discover the governing PDEs of the data.
Revised: t = t 0 , t 1 , · · · } on the spatial domain Ω ⊂ R 2 , with u ( t, · ) : Ω (cid:55)→ R , we want to discover the governing PDEs of the data.

Original: We assume that the observed data are associated with a PDE that takes the following general form:
Revised: We assume that the observed data are associated with a PDE that takes the following general form:

Original: [Equation]
Revised: [Equation]

Original: Our objective is to design a feed-forward network, named the PDE-Net, that approximates the PDE (1) in the way that: 1) we can predict the dynamical behavior of the equation for as long time as possible; 2) we are able to reveal the form of the response function F and the differential operators involved.
Revised: Our objective is to design a feed-forward network, named the PDE-Net, that approximates the PDE (1) in the way that: 1) we can predict the dynamical behavior of the equation for as long time as possible; 2) we are able to reveal the form of the response function F and the differential operators involved.

Original: There are two main components of the PDE-Net that are combined together in the same network: one is automatic determination on the differential operators involved in the PDE and their discrete approximations; the other is to approximate the nonlinear response function F .
Revised: There are two main components of the PDE-Net that are combined together in the same network: one is automatic determination on the differential operators involved in the PDE and their discrete approximations; the other is to approximate the nonlinear response function F .

Original: In this section, we start with discussions on the relation between convolutions and differentiations in discrete setting.
Revised: In this section, we start with discussions on the relation between convolutions and differentiations in discrete setting.

Original:  C ONVOLUTIONS AND D IFFERENTIATIONS
Revised: 2. C ONVOLUTIONS AND D IFFERENTIATIONS

Original: A comprehensive analysis on the relations between convolutions and differentiations within variational and PDE framework were laid out by Cai et al.
Revised: A comprehensive analysis on the relations between convolutions and differentiations within variational and PDE framework were laid out by Cai et al.

Original: (2012) and Dong et al.
Revised: (2012) and Dong et al.

Original: (2017), where the authors established general connections between PDE based approach and wavelet frame based approach for image restoration problems.
Revised: (2017), where the authors established general connections between PDE based approach and wavelet frame based approach for image restoration problems.

Original: We demonstrate one of the key observations of their work using a simple example.
Revised: We demonstrate one of the key observations of their work using a simple example.

Original: Consider the 2-dimensional Haar wavelet frame ﬁlter bank contains one low-pass ﬁlter h 

Revised: Consider the 2-dimensional Haar wavelet frame ﬁlter bank contains one low-pass ﬁlter h

Original: and three high pass ﬁlters h 10 , h 01 and h 11 :
Revised: and three high pass ﬁlters h 10 , h 01 and h 11 :

Original: [Equation]
Revised: [Equation]

Original: 00 =

Original: [Equation]

Original: The associated Haar wavelet frame transform on an image u is deﬁned by
Revised: The associated Haar wavelet frame transform on an image u is deﬁned by

Original: [Equation]
Revised: [Equation]

Revised: where (cid:126) is the circular convolution.

Original: It is easy to verify using Taylor’s expansion that the high frequency coefﬁcients of the Haar wavelet frame transform on u are discrete approximations of differential operators:
Revised: It is easy to verify using Taylor’s expansion that the high frequency coefﬁcients of the Haar wavelet frame transform on u are discrete approximations of differential operators:

Original: [Equation]
Revised: [Equation]

Original: Here, δ x and δ y represent the horizontal and vertical spatial grid size respectively.
Revised: Here, δ x and δ y represent the horizontal and vertical spatial grid size respectively.

Original: For simplicity of notation, we use regular character to denote both discrete and continuum functions, since there should be no confusion within the context.
Revised: For simplicity of notation, we use regular character to denote both discrete and continuum functions, since there should be no confusion within the context.

Original: A profound relationship between convolutions and differentiations was presented in Dong et al.
Revised: A profound relationship between convolutions and differentiations was presented in Dong et al.

Original: (2017), where the authors discussed the connection between the order of sum rules of ﬁlters and the orders of differential operators.
Revised: (2017), where the authors discussed the connection between the order of sum rules of ﬁlters and the orders of differential operators.

Original: Note that the order of sum rules is closely related to the order of vanishing moments in wavelet theory (Daubechies, 1992; Mallat, 1999).
Revised: Note that the order of sum rules is closely related to the order of vanishing moments in wavelet theory (Daubechies, 1992; Mallat, 1999).

Original: We ﬁrst recall the deﬁnition of the order of sum rules.
Revised: We ﬁrst recall the deﬁnition of the order of sum rules.

Original: Deﬁnition 2.1 (Order of Sum Rules) .
Revised: Deﬁnition 2.1 (Order of Sum Rules) .

Original: For a ﬁlter q , we say q to have sum rules of order α
Revised: For a ﬁlter q , we say q to have sum rules of order α

Original: =
Revised: =

Original: ( α 1 , α 2 ) , where α ∈ Z 2+ , provided that
Revised: ( α 1 , α 2 ) , where α ∈ Z 2+ , provided that

Original: [Equation]
Revised: [Equation]

Original: for all β ∈ Z 2+ with | β | < | α | and for all β ∈ Z 2+ with | β | = | α | but β (cid:54) =
Revised: for all β ∈ Z 2+ with | β | < | α | and for all β ∈ Z 2+ with | β | = | α | but β (cid:54) =

Original: α .
Revised: α .

Original: Specially, we say q to have total sum rules of order K with K ∈ Z + , if (2) holds for all β ∈ Z 2+ with | β | < K .

Original: If (2) holds for all β ∈ Z 2+ with | β | < K except for β (cid:54) =
Revised: If (2) holds for all β ∈ Z 2+ with | β | < K except for β (cid:54) =

Original: β 0 with certain β 0 ∈
Revised: β 0 with certain β 0 ∈

Original: Z 2+ and | β 0 | =
Revised: Z 2+ and | β 0 | =

Original: J < K , then we say q to have total sum rules of order K \{ J + 1 } .
Revised: J < K , then we say q to have total sum rules of order K \{ J + 1 } .

Original: The following proposition from Dong et al.
Revised: The following proposition from Dong et al.

Original: (2017) links the orders of sum rules with orders of differential operator.
Revised: (2017) links the orders of sum rules with orders of differential operator.

Original: Propositin 2.1.
Revised: Propositin 2.1.

Original: Let q be a ﬁlter with sum rules of order α ∈ Z 2+ .
Revised: Let q be a ﬁlter with sum rules of order α ∈ Z 2+ .

Original: Then for a smooth function F ( x ) on R 2 , we have
Revised: Then for a smooth function F ( x ) on R 2 , we have

Original: [Equation]
Revised: [Equation]

Original: where C α is the constant deﬁned by
Revised: where C α is the constant deﬁned by

Original: [-C α = 
-]
Revised: [-C α = 
-]

Original: α !

Original: [Equation]
Revised: [Equation]

Original: If, in addition, q has total sum rules of order K \{| α | + 1 } for some K > | α | , then
Revised: If, in addition, q has total sum rules of order K \{| α | + 1 } for some K > | α | , then

Original: [Equation]
Revised: [Equation]

Original: According to Proposition 2.1, an α th order differential operator can be approximated by the convolution of a ﬁlter with α order of sum rules.
Revised: According to Proposition 2.1, an α th order differential operator can be approximated by the convolution of a ﬁlter with α order of sum rules.

Original: Furthermore, according to (4), one can obtain a high order approximation of a given differential operator if the corresponding ﬁlter has an order of total sum rules with K > | α | + k, k
Revised: Furthermore, according to (4), one can obtain a high order approximation of a given differential operator if the corresponding ﬁlter has an order of total sum rules with K > | α | + k, k

Original: (cid:62) 1 .
Revised: (cid:62) 1 .

Revised: For example, the ﬁlter h 10 in the Haar wavelet frame ﬁlter bank has a sum rules of order (1 , 0) , and a total sum rules of order 2 \{ 2 } .

Revised: Thus, up to a constant and a proper scaling, h 10 corresponds to a discretization of ∂∂x with ﬁrst order approximation.

Revised: The ﬁler h 11 has a sum rules of order (1 , 1) , and a total sum rules of order 3 \{ 3 } .

Revised: Thus, up to a constant and a proper scaling, h 11 corresponds to a discretization of ∂ 2 ∂x∂y with ﬁrst order approximation.

Revised: Finally, consider ﬁlter

Original:  Now, we introduce the concept of moment matrix for a given ﬁlter that will be used to constrain ﬁlters in the PDE-Net.
Revised: [Equation] It has a sum rules of order (1 , 0) , and a total sum rules of order 3 \{ 2 } . Thus, up to a constant and a proper scaling, q corresponds to a discretization of ∂∂x with second order approximation. Now, we introduce the concept of moment matrix for a given ﬁlter that will be used to constrain ﬁlters in the PDE-Net.

Original: For an N × N ﬁlter q , deﬁne the moment matrix of q asM
Revised: For an N × N ﬁlter q , deﬁne the moment matrix of q as

Original: [Equation]
Revised: [Equation]

Original: q
Revised: i,j

Original: [Equation]
Revised: [Equation]

Original: m i,j ) N × N , where m i,j =
Revised: N × N , where m i,j =

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: for i, j = 1 , 2 , . . .
Revised: for i, j = 1 , 2 , . . .

Original: , N .
Revised: , N .

Original: We shall call the ( i, j ) -element of M ( q ) the ( i − 1 , j − 1) -moment of q for simplicity.
Revised: We shall call the ( i, j ) -element of M ( q ) the ( i − 1 , j − 1) -moment of q for simplicity.

Original: Combining (5) and Proposition 2.1, one can easily see that ﬁlter q can be designed to approximate any differential operator at any given approximation order by imposing constraints on
Revised: Combining (5) and Proposition 2.1, one can easily see that ﬁlter q can be designed to approximate any differential operator at any given approximation order by imposing constraints on

Original: M ( q ) .
Revised: M ( q ) .

Original: For example, if we want to approximate ∂u∂x (up to a constant) by convolution q (cid:126) u where q is a 3 × 3 ﬁlter, we can consider the following constrains on M ( q ) :
Revised: For example, if we want to approximate ∂u∂x (up to a constant) by convolution q (cid:126) u where q is a 3 × 3 ﬁlter, we can consider the following constrains on M ( q ) :

Original: [Equation]
Revised: [Equation]

Original: Here, (cid:63) means no constraint on the corresponding entry.
Revised: Here, (cid:63) means no constraint on the corresponding entry.

Original: The constraints described by the moment matrix on the left of (6) guarantee the approximation accuracy is at least ﬁrst order, and the ones on the right guarantee an approximation of at least second order.
Revised: The constraints described by the moment matrix on the left of (6) guarantee the approximation accuracy is at least ﬁrst order, and the ones on the right guarantee an approximation of at least second order.

Original: In particular, when all entries of M ( q ) are constrained, e.g.
Revised: In particular, when all entries of M ( q ) are constrained, e.g.

Original: [Equation]
Revised: [Equation]

Original:  the corresponding ﬁlter can be uniquely determined, in which case we call it a “frozen” ﬁlter.
Revised: 0(cid:33) , the corresponding ﬁlter can be uniquely determined, in which case we call it a “frozen” ﬁlter.

Original: In the
Revised: In the

Original: PDE-Net which shall be introduced in the next subsection, all ﬁlters are learned subjected to partial constraints on their associated moment matrices.
Revised: PDE-Net which shall be introduced in the next subsection, all ﬁlters are learned subjected to partial constraints on their associated moment matrices.

Original: It is worth noticing that the approximation property of a ﬁlter is limited by its size.
Revised: It is worth noticing that the approximation property of a ﬁlter is limited by its size.

Original: Generally speaking, large ﬁlters can approximate higher order differential operators or lower order differential operators with higher approximation orders.
Revised: Generally speaking, large ﬁlters can approximate higher order differential operators or lower order differential operators with higher approximation orders.

Original: Taking 1-dimensional case as an example, 3-element ﬁlters cannot approximate the ﬁfth order differential operator, whereas 7-element ﬁlters can.
Revised: Taking 1-dimensional case as an example, 3-element ﬁlters cannot approximate the ﬁfth order differential operator, whereas 7-element ﬁlters can.

Original: In other words, the larger are the ﬁlters, the stronger is the representation capability of ﬁlters.
Revised: In other words, the larger are the ﬁlters, the stronger is the representation capability of ﬁlters.

Original: However, larger ﬁlters lead to more memory overhead and higher computation cost.
Revised: However, larger ﬁlters lead to more memory overhead and higher computation cost.

Original: It is a wisdom to balance the trade-off in practice.
Revised: It is a wisdom to balance the trade-off in practice.

Original: 2. A RCHITECTURE OF PDE-N ET
Revised: 2. A RCHITECTURE OF PDE-N ET

Original: Given the evolution PDE (1), we consider forward Euler as the temporal discretization.
Revised: Given the evolution PDE (1), we consider forward Euler as the temporal discretization.

Original: One may consider more sophisticated temporal discretization which leads to different network architectures.
Revised: One may consider more sophisticated temporal discretization which leads to different network architectures.

Original: For simplicity, we focus on forward Euler in this paper.
Revised: For simplicity, we focus on forward Euler in this paper.

Original: O NE δt -B LOCK :
Revised: O NE δt -B LOCK :

Original: Let ˜ u
Revised: Let ˜ u

Original: ( t i +1 , · ) be the predicted value of u at time t
Revised: ( t i +1 , · ) be the predicted value of u at time t

Original: i +1 based on the value of u at t i .
Revised: i +1 based on the value of u at t i .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Here, the operators D 0 and D ij are convolution operators with the underlying ﬁlters denoted by qand q ij  .
Revised: Here, the operators D 0 and D ij are convolution operators with the underlying ﬁlters denoted by qand q ij , i.e. D 0 u = q 0 (cid:126) u and D ij u = q ij (cid:126) u .

Original: The operators D 10 , D 01 , D 11 , etc.
Revised: The operators D 10 , D 01 , D 11 , etc.

Original: approximate differential operators, i.e. D ij u ≈ ∂
Revised: approximate differential operators, i.e. D ij u ≈ ∂

Original: i + j u ∂ i x∂
Revised: i + j u ∂ i x∂

Original: j y .
Revised: j y .

Original: The operators D 0 and D 00 are average operators.
Revised: The operators D 0 and D 00 are average operators.

Original: The purpose of introducing these average operators in stead of using the identity is to improve stability of the network and enables it to capture more complex dynamics.
Revised: The purpose of introducing these average operators in stead of using the identity is to improve stability of the network and enables it to capture more complex dynamics.

Original: Other than the assumption that the observed dynamics is governed by a PDE of the form (1), we assume that the highest order of the PDE is less than some positive integer.
Revised: Other than the assumption that the observed dynamics is governed by a PDE of the form (1), we assume that the highest order of the PDE is less than some positive integer.

Original: Then, the task of approximating F is equivalent to a multivariate regression problem, which can be approximated by a point-wise neural network (with shared weights across the computation domain Ω ) or other classical machine learning methods.
Revised: Then, the task of approximating F is equivalent to a multivariate regression problem, which can be approximated by a point-wise neural network (with shared weights across the computation domain Ω ) or other classical machine learning methods.

Original: Combining the approximation of differential operators and the nonlinear function F , we achieve an approximation framework of (7) which will be referred to as a δt -block (see Figure 1).
Revised: Combining the approximation of differential operators and the nonlinear function F , we achieve an approximation framework of (7) which will be referred to as a δt -block (see Figure 1).

Original: Note that if we have a prior knowledge on the form of the response function F , we can easily adjust the network architecture by taking advantage of the additional information.
Revised: Note that if we have a prior knowledge on the form of the response function F , we can easily adjust the network architecture by taking advantage of the additional information.

Original: This may simplify the training and improve the results.
Revised: This may simplify the training and improve the results.

Original: [Equation]
Revised: [Equation]

Original: ULTIPLE δt -B LOCKS ):

Original: One δt -block only guarantees the accuracy of one-step dynamics, which does not take error accumulation into consideration.
Revised: One δt -block only guarantees the accuracy of one-step dynamics, which does not take error accumulation into consideration.

Original:  To improve the stability of the network and enable long-term prediction, we stack multiple δt -blocks into a deep network, and call this network the PDE-Net (see Figure 2).
Revised: This may cause severe instability in prediction. To improve the stability of the network and enable long-term prediction, we stack multiple δt -blocks into a deep network, and call this network the PDE-Net (see Figure 2).

Original: The importance of stacking multiple δt -blocks will be demonstrated in Section 3.
Revised: The importance of stacking multiple δt -blocks will be demonstrated in Section 3.

Original: The PDE-Net can be easily described as: (1) stacking one δt -block multiple times; (2) sharing parameters in all δt -blocks.
Revised: The PDE-Net can be easily described as: (1) stacking one δt -block multiple times; (2) sharing parameters in all δt -blocks.

Original: Given an input data u ( t i , · ) , training a PDE-Net with n δt -blocks needs to minimize the accumulated error
Revised: Given an input data u ( t i , · ) , training a PDE-Net with n δt -blocks needs to minimize the accumulated error

Original: ||
Revised: ||

Original: u
Revised: u

Original: ( t i + n , · ) − ˜ u
Revised: ( t i + n , · ) − ˜ u

Original: ( t i + n , · ) || 22 , where ˜ u ( t i + n , · ) is the output from the PDE-Net (i.e. n δt -blocks) with input u ( t i , · ) .
Revised: ( t i + n , · ) || 22 , where ˜ u ( t i + n , · ) is the output from the PDE-Net (i.e. n δt -blocks) with input u ( t i , · ) .

Original: Thus, the PDE-Net with bigger n owns a longer time stability.
Revised: Thus, the PDE-Net with bigger n owns a longer time stability.

Original: Note that sharing parameters is a common practice in deep learning, which decreases the number of parameters and leads to signiﬁcant memory reduction (Goodfellow et al., 2016).
Revised: Note that sharing parameters is a common practice in deep learning, which decreases the number of parameters and leads to signiﬁcant memory reduction (Goodfellow et al., 2016).

Original: L OSS F UNCTION AND C ONSTRAINTS :
Revised: L OSS F UNCTION AND C ONSTRAINTS :

Original: Consider the data set { u j ( t i , · ) : i, j = 0 , 1 , . . .
Revised: Consider the data set { u j ( t i , · ) : i, j = 0 , 1 , . . .

Original: } , where j indicates the j -th solution path with a certain initial condition of the unknown dynamics.
Revised: } , where j indicates the j -th solution path with a certain initial condition of the unknown dynamics.

Original: We would like to train the PDE-Net with n δt -blocks.
Revised: We would like to train the PDE-Net with n δt -blocks.

Original: For a given n ≥ 1 , every pair of the data { u j ( t i , · ) , u j ( t i + n , · ) } , for each i and j , is a training sample, where u j ( t i , · ) is the input and u j ( t i + n , · ) is the label that we need to match with the output from the PDE-Net.
Revised: For a given n ≥ 1 , every pair of the data { u j ( t i , · ) , u j ( t i + n , · ) } , for each i and j , is a training sample, where u j ( t i , · ) is the input and u j ( t i + n , · ) is the label that we need to match with the output from the PDE-Net.

Original: We select the following simple (cid:96) 2 loss function for training:
Revised: We select the following simple (cid:96) 2 loss function for training:

Original: [Equation]
Revised: [Equation]

Original: i,j l ij , where l ij = ||
Revised: l ij , where l ij = ||

Original: u j ( t i + n , · ) − ˜ u
Revised: u j ( t i + n , · ) − ˜ u

Original: j ( t i + n , · ) || 22 , where ˜ u j ( t i + n , · ) is the output of the PDE-Net with u j ( t i , · ) as the input.
Revised: j ( t i + n , · ) || 22 , where ˜ u j ( t i + n , · ) is the output of the PDE-Net with u j ( t i , · ) as the input.

Original: All the ﬁlters involved in the PDE-Net are properly constrained using their associated moment matrices.
Revised: All the ﬁlters involved in the PDE-Net are properly constrained using their associated moment matrices.

Original: Let q 0 and q ij be the underlying ﬁlters of D 0 and D ij .
Revised: Let q 0 and q ij be the underlying ﬁlters of D 0 and D ij .

Original: We impose the following constrains
Revised: We impose the following constrains

Original: [Equation]
Revised: [Equation]

Original: and for i + j >
Revised: and for i + j >

Original: [Equation]
Revised: [Equation]

Original: For example, for 3 × 3 ﬁlters, we have
Revised: For example, for 3 × 3 ﬁlters, we have

Original: [Equation]
Revised: [Equation]

Original: (cid:32) 1 (cid:63) (cid:63) (cid:63) (cid:63) (cid:63) (cid:63) (cid:63) (cid:63) (cid:33) and
Revised: (cid:63) (cid:63) (cid:63) (cid:63) (cid:63) (cid:63) (cid:63) (cid:63) (cid:33) and

Original: M
Revised: M

Original: [Equation]

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: (cid:63)

Original: [Equation]
Revised: [Equation]

Original:  (cid:63) (cid:33)
Revised: 0 (cid:63) (cid:63) (cid:33)

Original: [Equation]
Revised: [Equation]

Original: To demonstrate the necessity of learnable ﬁlters, we will compare the PDE-Net having the aforementioned constrains on the ﬁlters with the PDE-Net having frozen ﬁlters.
Revised: To demonstrate the necessity of learnable ﬁlters, we will compare the PDE-Net having the aforementioned constrains on the ﬁlters with the PDE-Net having frozen ﬁlters.

Original: To differentiate the two cases, we shall call the PDE-Net with frozen ﬁlters “the Frozen-PDE-Net”.
Revised: To differentiate the two cases, we shall call the PDE-Net with frozen ﬁlters “the Frozen-PDE-Net”.

Original: To further increase the expressive power and ﬂexibility of the PDE-Net, we may associate multiple ﬁlters to approximate a given differential operator.
Revised: To further increase the expressive power and ﬂexibility of the PDE-Net, we may associate multiple ﬁlters to approximate a given differential operator.

Original: However, in order not to mess up the identiﬁability of the underlying PDE model, we may select only one of the ﬁlters to provide correct approximation to the given differential operator in the way as described above.
Revised: However, in order not to mess up the identiﬁability of the underlying PDE model, we may select only one of the ﬁlters to provide correct approximation to the given differential operator in the way as described above.

Original: The rest of the ﬁlters are constrained in the way that they only contribute to modify the local truncation errors.
Revised: The rest of the ﬁlters are constrained in the way that they only contribute to modify the local truncation errors.

Original: For example, consider two 3 × 3 ﬁlters { q 0 , q 1 } and constrain their moment matrices as follows
Revised: For example, consider two 3 × 3 ﬁlters { q 0 , q 1 } and constrain their moment matrices as follows

Original: [Equation]
Revised: [Equation]

Original: (cid:63) (cid:63) (cid:63) (cid:63) (cid:33) .
Revised: Then,

Original: Then, q 0 (cid:126) u + q 1 (cid:126) u is potentially a better approximation to u x (up to a constant) than q 0 (cid:126) u .
Revised: [Equation] 1 (cid:126) u is potentially a better approximation to u x (up to a constant) than q 0 (cid:126) u .

Original: However, for simplicity, we only use one ﬁlter to approximate a given differential operator in this paper.
Revised: However, for simplicity, we only use one ﬁlter to approximate a given differential operator in this paper.

Original: N OVELTY OF THE PDE-N ET :
Revised: N OVELTY OF THE PDE-N ET :

Original: Different from ﬁxing numerical approximations of differentiations in advance in sparse regression methods (Schaeffer, 2017; Rudy et al., 2017), using learnable ﬁlters makes the PDE-Net more ﬂexible, and enables more robust approximation of unknown dynamics and longer time prediction (see numerical experiments in Section 3 and Section 4).
Revised: Different from ﬁxing numerical approximations of differentiations in advance in sparse regression methods (Schaeffer, 2017; Rudy et al., 2017), using learnable ﬁlters makes the PDE-Net more ﬂexible, and enables more robust approximation of unknown dynamics and longer time prediction (see numerical experiments in Section 3 and Section 4).

Original: Furthermore, the speciﬁc form of the response function F is also approximated from the data, rather than assumed to be known in advance (such as (Raissi & Karniadakis, 2017)).
Revised: Furthermore, the speciﬁc form of the response function F is also approximated from the data, rather than assumed to be known in advance (such as (Raissi & Karniadakis, 2017)).

Original: On the other hand, by inﬂicting constrains on moment matrices, we can identify which differential operators are included in the underlying PDE which helps with identifying the nonlinear response function F .
Revised: On the other hand, by inﬂicting constrains on moment matrices, we can identify which differential operators are included in the underlying PDE which helps with identifying the nonlinear response function F .

Original: This grants transparency to the PDE-Net and the potential to reveal hidden physical laws.
Revised: This grants transparency to the PDE-Net and the potential to reveal hidden physical laws.

Original: Therefore, the proposed PDE-Net is distinct from the existing learning based method to discover PDEs from data, as well as networks designed in deep learning for computer vision tasks.
Revised: Therefore, the proposed PDE-Net is distinct from the existing learning based method to discover PDEs from data, as well as networks designed in deep learning for computer vision tasks.

Original: 2. I NITIALIZATION AND TRAINING
Revised: 2. I NITIALIZATION AND TRAINING

Original: In the PDE-Net, parameters can be divided into three groups:
Revised: In the PDE-Net, parameters can be divided into three groups:

Original: • ﬁlters to approximate differential operators; • the parameters of the point-wise neural network to approximate F ; • hyper-parameters, such as the number of ﬁlters, the size of ﬁlters, the number of layers, etc.
Revised: • ﬁlters to approximate differential operators ; • the parameters of the point-wise neural network to approximate F ; • hyper-parameters, such as the number of ﬁlters, the size of ﬁlters, the number of layers, etc.

Original: The parameters of the point-wise neural network are shared across the computation domain Ω , and are initialized by random sampling from a Gaussian distribution.
Revised: The parameters of the point-wise neural network are shared across the computation domain Ω , and are initialized by random sampling from a Gaussian distribution.

Original: For the ﬁlters, we initialize them by freezing them to their corresponding differential operators.
Revised: For the ﬁlters, we initialize them by freezing them to their corresponding differential operators.

Original: For example, if a ﬁlter is to approximate ∂∂x , we freeze it by constraining its (1 , 0) -moment to 1 and other moments to 0.
Revised: For example, if a ﬁlter is to approximate ∂∂x , we freeze it by constraining its (1 , 0) -moment to 1 and other moments to 0.

Original: During the training process, we release the ﬁlters by switching to the constrains described in Section 2.2.
Revised: During the training process, we release the ﬁlters by switching to the constrains described in Section 2.2.

Original: Instead of training an n -layer PDE-Net directly, we adopt layer-wise training, which improves the training speed.
Revised: Instead of training an n -layer PDE-Net directly, we adopt layer-wise training, which improves the training speed.

Original: To be more precise, we start with training the PDE-Net on the ﬁrst δt -block, and then use the results of the ﬁrst δt -block as the initialization and restart training the PDE-Net on the ﬁrst two δt -blocks.
Revised: To be more precise, we start with training the PDE-Net on the ﬁrst δt -block, and then use the results of the ﬁrst δt -block as the initialization and restart training the PDE-Net on the ﬁrst two δt -blocks.

Original: Repeat until we complete all n blocks.
Revised: Repeat until we complete all n blocks.

Original: Note that all the parameters in each of the δt -block are shared across layers.
Revised: Note that all the parameters in each of the δt -block are shared across layers.

Original: In addition, we add a warm-up step before the training of the ﬁrst δt -block.
Revised: In addition, we add a warm-up step before the training of the ﬁrst δt -block.

Original: The warm-up step is to obtain a good initial guess of the parameters of the point-wise neural network that approximates F by using frozen ﬁlters.
Revised: The warm-up step is to obtain a good initial guess of the parameters of the point-wise neural network that approximates F by using frozen ﬁlters.

Original: 2. R ELATIONS TO SOME EXISTING NETWORKS
Revised: 2. R ELATIONS TO SOME EXISTING NETWORKS

Original: In recent years, a variety of deep neural networks have been introduced with great success in computer vision.
Revised: In recent years, a variety of deep neural networks have been introduced with great success in computer vision.

Original: The structure of the proposed PDE-Net is similar to some existing networks such as the
Revised: The structure of the proposed PDE-Net is similar to some existing networks such as the

Original: Network-In-Network (NIN) (Lin et al., 2013) and the deep Residual Neural Network (ResNet) (He et al., 2016a;b).
Revised: Network-In-Network (NIN) (Lin et al., 2013) and the deep Residual Neural Network (ResNet) (He et al., 2016a;b).

Original: The NIN is an improvement over the traditional convolutional neural networks.
Revised: The NIN is an improvement over the traditional convolutional neural networks.

Original: One of the special designs of NIN is the use of multilayer perceptron convolution (mlpconv) layers instead of the ordinary convolution layers.
Revised: One of the special designs of NIN is the use of multilayer perceptron convolution (mlpconv) layers instead of the ordinary convolution layers.

Original: An mlpconv layer contains the convolutions and small point-wise neural networks.
Revised: An mlpconv layer contains the convolutions and small point-wise neural networks.

Original: Such design can improve the ability of the network to extract nonlinear features from shallow layers.
Revised: Such design can improve the ability of the network to extract nonlinear features from shallow layers.

Original: The inner structure of one δt -block of the PDE-Net is similar to the mlpconv layer, and the multiple δt -blocks structure is similar to the NIN structure, except for the pooling and ReLU operations.
Revised: The inner structure of one δt -block of the PDE-Net is similar to the mlpconv layer, and the multiple δt -blocks structure is similar to the NIN structure, except for the pooling and ReLU operations.

Original: On the other hand, each δt -block of the PDE-Net has two paths (see Figure 1 and Figure 2): one is for the averaged quantity of u and the other is for the increment F .
Revised: On the other hand, each δt -block of the PDE-Net has two paths (see Figure 1 and Figure 2): one is for the averaged quantity of u and the other is for the increment F .

Original: This structure coincides with the “residual block” introduced in ResNet.
Revised: This structure coincides with the “residual block” introduced in ResNet.

Original: In fact, there has been a substantial study on the relation between ResNet and dynamical systems recently (E, 2017; Haber & Ruthotto, 2017; Sonoda & Murata, 2017).
Revised: In fact, there has been a substantial study on the relation between ResNet and dynamical systems recently (E, 2017; Haber & Ruthotto, 2017; Sonoda & Murata, 2017).

Original:  N UMERICAL S TUDIES : C ONVECTION -D IFFUSION E QUATIONS
Revised:  N UMERICAL S TUDIES : C ONVECTION -D IFFUSION E QUATIONS

Original: Convection-diffusion equations are classical PDEs that are used to describe physical phenomena where particles, energy, or other physical quantities are transferred inside a physical system due to two processes: diffusion and convection (Chandrasekhar, 1943).
Revised: Convection-diffusion equations are classical PDEs that are used to describe physical phenomena where particles, energy, or other physical quantities are transferred inside a physical system due to two processes: diffusion and convection (Chandrasekhar, 1943).

Original: Convection-diffusion equations are widely applied in many scientiﬁc areas and industrial ﬁelds, such as pollutants dispersion in rivers or atmosphere, solute transferring in a porous medium, and oil reservoir simulation.
Revised: Convection-diffusion equations are widely applied in many scientiﬁc areas and industrial ﬁelds, such as pollutants dispersion in rivers or atmosphere, solute transferring in a porous medium, and oil reservoir simulation.

Original: In practical situations, usually the physical and chemical properties on different locations cannot be the same (called anisotropy in physics), thus it is more reasonable that convection coefﬁcients and diffusion coefﬁcients are variables instead of constants.
Revised: In practical situations, usually the physical and chemical properties on different locations cannot be the same (called anisotropy in physics), thus it is more reasonable that convection coefﬁcients and diffusion coefﬁcients are variables instead of constants.

Original: 3. S IMULATED D ATA , T RAINING AND T ESTING
Revised: 3. S IMULATED D ATA , T RAINING AND T ESTING

Original: We consider a 2-dimensional linear variable-coefﬁcient convection-diffusion equation on Ω = [0 , 2 π ] × [0 , 2 π ] ,
Revised: We consider a 2-dimensional linear variable-coefﬁcient convection-diffusion equation on Ω = [0 , 2 π ] × [0 , 2 π ] ,

Original: [Equation]
Revised: [Equation]

Original: du

Original: [Equation]
Revised: [Equation]

Original: =
Revised: x

Original: [Equation]
Revised: [Equation]

Original: y
Revised: 8) where

Original: [Equation]
Revised: [Equation]

Original: with ( t, x, y ) ∈ [0 , 0 . 2] × Ω , (8) where a ( x, y ) = 0 . 5(cos( y ) + x (2 π − x ) sin( x )) + 0 . 6 , b ( x, y ) =
Revised: 2 and d

Original: 2(cos( y ) + sin( x )) + 0 . 8 , c = 0 . 2 and d = 0 . 3 .
Revised: [Equation]

Original: The computation domain Ω is discretized using a 50 × 50 regular mesh.
Revised: The computation domain Ω is discretized using a 50 × 50 regular mesh.

Original: Data is generated by solving problem (8) using a high precision numerical scheme with pseudo-spectral method for spatial discretization and 4th order Runge-Kutta for temporal discretization (with time step size δt = 0 . 01 ).
Revised: Data is generated by solving problem (8) using a high precision numerical scheme with pseudo-spectral method for spatial discretization and 4th order Runge-Kutta for temporal discretization (with time step size δt = 0 . 01 ).

Original: We assume periodic boundary condition and the initial value u 0 ( x, y ) is generated from 
Revised: We assume periodic boundary condition and the initial value u 0 ( x, y ) is generated from [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where N = 9 , λ k,l , γ k,l ∼ N (0 , 150 ) , and k and l are chosen randomly.
Revised: ) where N = 9 , λ k,l , γ k,l ∼ N (0 , 150 ) , and k and l are chosen randomly.

Original: In order to mimic real world scenarios, we add noise to the generated data.
Revised: In order to mimic real world scenarios, we add noise to the generated data.

Original: For each sample sequence u ( x, y, t ) , t ∈ [0 , 0 . 2] , the noise is added as
Revised: For each sample sequence u ( x, y, t ) , t ∈ [0 , 0 . 2] , the noise is added as

Original: [Equation]
Revised: [Equation]

Original:  where M = max x,y,t { u ( x, y, t ) } , W ∼ N (0 , 1) and N (0 , 1) represents the standard normal distribution.
Revised: (10) where M = max x,y,t { u ( x, y, t ) } , W ∼ N (0 , 1) and N (0 , 1) represents the standard normal distribution.

Original: Suppose we know a priori that the underlying PDE is linear with order no more than 4.
Revised: Suppose we know a priori that the underlying PDE is linear with order no more than 4.

Original: Then, the response function F takes the following form
Revised: Then, the response function F takes the following form

Original: [Equation]
Revised: [Equation]

Original: Each δt -block of the PDE-Net can be written as
Revised: Each δt -block of the PDE-Net can be written as

Original: [Equation]
Revised: [Equation]

Original: where { D 0 , D ij : i + j ≤ 4 } are convolution operators and { c ij : i + j ≤ 4 } are 2D arrays which approximate functions f ij ( x, y ) on Ω .
Revised: where { D 0 , D ij : i + j ≤ 4 } are convolution operators and { c ij : i + j ≤ 4 } are 2D arrays which approximate functions f ij ( x, y ) on Ω .

Original: The approximation is achieved using piecewise quadratic polynomial interpolation with smooth transitions at the boundaries of each piece.
Revised: The approximation is achieved using piecewise quadratic polynomial interpolation with smooth transitions at the boundaries of each piece.

Original: The ﬁlters associated to the convolution operators { D 0 , D ij : i + j ≤ 4 } and the coefﬁcients of the piecewise quadratic polynomials are the trainable parameters of the network.
Revised: The ﬁlters associated to the convolution operators { D 0 , D ij : i + j ≤ 4 } and the coefﬁcients of the piecewise quadratic polynomials are the trainable parameters of the network.

Original: During training and testing, the data is generated on-the-ﬂy, i.e. we only generate the data needed following the aforementioned procedure when training and testing the PDE-Net.
Revised: During training and testing, the data is generated on-the-ﬂy, i.e. we only generate the data needed following the aforementioned procedure when training and testing the PDE-Net.

Original: In our experiments, the size of the ﬁlters that will be used is 5 × 5 or 7 × 7 .
Revised: In our experiments, the size of the ﬁlters that will be used is 5 × 5 or 7 × 7 .

Original: The total number of trainable parameters for each δt -block is approximately 17 k. During training, we use LBFGS, instead of SGD, to optimize the parameters.
Revised: The total number of trainable parameters for each δt -block is approximately 17 k. During training, we use LBFGS, instead of SGD, to optimize the parameters.

Original: We use 28 data samples per batch to train each layer (i.e. δt -block) and we only construct the PDE-Net up to 20 layers, which requires totally 560 data samples per batch.
Revised: We use 28 data samples per batch to train each layer (i.e. δt -block) and we only construct the PDE-Net up to 20 layers, which requires totally 560 data samples per batch.

Original: Note that the PDE-Net is designed with the assumption that it approximates nonlinear evolution PDEs, which is a relatively stronger assumption than the networks in deep learning.
Revised: Note that the PDE-Net is designed with the assumption that it approximates nonlinear evolution PDEs, which is a relatively stronger assumption than the networks in deep learning.

Original: Therefore, we require less training data and LBFGS performs better than SGD (which is widely adopted in deep learning).
Revised: Therefore, we require less training data and LBFGS performs better than SGD (which is widely adopted in deep learning).

Original: Furthermore, as will be shown by our numerical results, the learned PDE-Net generalizes very well.
Revised: Furthermore, as will be shown by our numerical results, the learned PDE-Net generalizes very well.

Original: The PDE-Net can accurately predict the dynamics even when the initial data u 0 does not come from the same distribution as in the training process.
Revised: The PDE-Net can accurately predict the dynamics even when the initial data u 0 does not come from the same distribution as in the training process.

Original: 3. R ESULTS AND D ISCUSSIONS
Revised: 3. R ESULTS AND D ISCUSSIONS

Original: This section presents numerical results of training the PDE-Net using the data set described in the previous subsection.
Revised: This section presents numerical results of training the PDE-Net using the data set described in the previous subsection.

Original: We will speciﬁcally observe how the learned PDE-Net performs in terms of prediction of dynamical behavior and identiﬁcation of the underlying PDE model.
Revised: We will speciﬁcally observe how the learned PDE-Net performs in terms of prediction of dynamical behavior and identiﬁcation of the underlying PDE model.

Original: Furthermore, we will investigate the effects of some of the hyper-parameters (e.g. size of the ﬁlters, number of δt -blocks) on the learned PDE-Net.
Revised: Furthermore, we will investigate the effects of some of the hyper-parameters (e.g. size of the ﬁlters, number of δt -blocks) on the learned PDE-Net.

Original: P REDICTING LONG - TIME DYNAMICS
Revised: P REDICTING LONG - TIME DYNAMICS

Original: We demonstrate the ability of the trained PDE-Net in prediction, which in the language of machine learning is the ability to generalize.
Revised: We demonstrate the ability of the trained PDE-Net in prediction, which in the language of machine learning is the ability to generalize.

Original: After the PDE-Net with n δt -blocks ( 1 ≤ n ≤ 20 ) is trained, we randomly generate 560 initial guesses based on (9) and (10), feed them to the PDE-Net, and measure the relative error between the predicted dynamics (i.e. the output of the PDE-Net) and the actual dynamics (obtained by solving (8) using high precision numerical scheme). 
Revised: After the PDE-Net with n δt -blocks ( 1 ≤ n ≤ 20 ) is trained, we randomly generate 560 initial guesses based on (9) and (10), feed them to the PDE-Net, and measure the normalized error between the predicted dynamics (i.e. the output of the PDE-Net) and the actual dynamics (obtained by solving (8) using high precision numerical scheme). The normalized error between the true data u and the predicted data ˜ u is deﬁned as [Equation] where ¯ u is the spatial average of u .

Original: The error plots are shown in Figure 3.
Revised: The error plots are shown in Figure 3.

Original: Results of longer prediction for the PDE-Net with 7 × 7 learnable ﬁlters are shown in Figure 4.
Revised: Results of longer prediction for the PDE-Net with 7 × 7 learnable ﬁlters are shown in Figure 4.

Original: Some of the images of the predicted dynamics are presented in Figure 5.
Revised: Some of the images of the predicted dynamics are presented in Figure 5.

Original: From these results, we can see that:
Revised: From these results, we can see that:

Original: • Even trained with noisy data, the PDE-Net is able to perform long-term prediction (see Figure 5); • Having multiple δt -blocks helps with the stability of the PDE-Net and ensures long-term prediction (see Figure 3); • The PDE-Net performs signiﬁcantly better than Frozen-PDE-Net, especially for 7 × 7 ﬁlters (see Figure 3); • The PDE-Net with 7 × 7 ﬁlters signiﬁcantly outperforms the PDE-Net with 5 × 5 ﬁlters in terms of the length of reliable predictions (see Figure 3 and 4).
Revised: • Even trained with noisy data, the PDE-Net is able to perform long-term prediction (see Figure 5); • Having multiple δt -blocks helps with the stability of the PDE-Net and ensures long-term prediction (see Figure 3); • The PDE-Net performs signiﬁcantly better than Frozen-PDE-Net, especially for 7 × 7 ﬁlters (see Figure 3); • The PDE-Net with 7 × 7 ﬁlters signiﬁcantly outperforms the PDE-Net with 5 × 5 ﬁlters in terms of the length of reliable predictions (see Figure 3 and 4).

Original: To reach an O (1) error, the length of prediction for the PDE-Net with 7 × 7 ﬁlters is about 10 times of that for the PDE-Net with 5 × 5 ﬁlters.
Revised: To reach an O (1) error, the length of prediction for the PDE-Net with 7 × 7 ﬁlters is about 10 times of that for the PDE-Net with 5 × 5 ﬁlters.

Original: To further demonstrate how well the learned PDE-Net generalizes, we generate initial values following (9) with highest frequency equal to 12, followed by adding noise (10). Note that the maximum allowable frequency in the training set is 9. The results of long-time prediction and the estimated dynamics are shown in Figure 6. Although oscillations are observed in the prediction, the estimated dynamic still captures the main pattern of the true dynamic.
Revised: [Figure]

Original: D ISCOVERING THE HIDDEN EQUATION
Revised: D ISCOVERING THE HIDDEN EQUATION

Original: For the linear problem, identifying the PDE amounts to ﬁnding the coefﬁcients { c ij : i + j ≤ 4 } that approximate { f ij : i + j ≤ 4 } .
Revised: For the linear problem, identifying the PDE amounts to ﬁnding the coefﬁcients { c ij : i + j ≤ 4 } that approximate { f ij : i + j ≤ 4 } .

Original: The coefﬁcients { c ij : i + j ≤ 2 } of the trained PDE-Net are shown in Figure 7.
Revised: The coefﬁcients { c ij : i + j ≤ 2 } of the trained PDE-Net are shown in Figure 6.

Original: Note that { f 11 } ∪ { f ij : 2 < i + j ≤ 4 } are absent from the PDE (8), and the corresponding coefﬁcients learned by the PDE-Net are indeed close to zero.
Revised: Note that { f 11 } ∪ { f ij : 2 < i + j ≤ 4 } are absent from the PDE (8), and the corresponding coefﬁcients learned by the PDE-Net are indeed close to zero.

Original: In order to have a more concise demonstration of the results, we only show the image of { c ij : i + j ≤ 2 } in Figure 7.
Revised: In order to have a more concise demonstration of the results, we only show the image of { c ij : i + j ≤ 2 } in Figure 6.

Original: Comparing the ﬁrst three rows of Figure 7, the coefﬁcients { c ij } learned by the PDE-Net are close to the true coefﬁcients { f ij } except for some oscillations due to the presence of noise in the training data.
Revised: Comparing the ﬁrst three rows of Figure 6, the coefﬁcients { c ij } learned by the PDE-Net are close to the true coefﬁcients { f ij } except for some oscillations due to the presence of noise in the training data.

Original: Furthermore, the last row of Figure 7 indicates that having multiple δt -blocks helps with estimation of the coefﬁcients.
Revised: Furthermore, the last row of Figure 6 indicates that having multiple δt -blocks helps with estimation of the coefﬁcients.

Original: However, having larger ﬁlters does not seem to improve the learning of the coefﬁcients, though it helps tremendously in prolonging predictions of the PDE-Net.
Revised: However, having larger ﬁlters does not seem to improve the learning of the coefﬁcients, though it helps tremendously in prolonging predictions of the PDE-Net.

Original: [Figure]
Revised: To further demonstrate how well the learned PDE-Net generalizes, we generate initial values following (9) with highest frequency equal to 12, followed by adding noise (10). Note that the maximum allowable frequency in the training set is 9. The results of long-time prediction and the estimated dynamics are shown in Figure 7.

Original:  In summary, the numerical experiments show that the PDE-Net is  able to conduct accurate prediction and identify the underlying PDE model at the same time, even in a noisy environment.
Revised: Although oscillations are observed in the prediction, the estimated dynamic still captures the main pattern of the true dynamic. The PDE (8) is of second order. In our previous experiments, we assumed that the PDE does not exceed the 4th order. If we know that the PDE is of second order, we will be able to have a more accurate estimation of the variable coefﬁcients of the convection and diffusion terms.

Revised: However, the prediction errors are slightly higher since we have fewer trainable parameters.

Revised: Nonetheless, since we are using a more accurate prior knowledge on the unknown PDE, the variance of the prediction errors are smaller than before.

Revised: These results are summarized in Figure 8 (green curves) and Figure 9.

Revised: To further demonstrate the importance of the moment constraints on the ﬁlters in the PDE-Net, we trained the network without any moment constraints and skipped any steps that utilize the knowledge of the relation between the ﬁlters and differential operators (i.e. we skipped warm-up and the initialization using ﬁnite difference ﬁlters).

Revised: For simplicity, we call the PDE-Net train in this way as the Freed-PDE-Net.

Original: Multiple δt -blocks, i.e. deeper structure of the PDE-Net, makes the PDE-Net more stable and enables longer time prediction.
Revised: The prediction errors of the Freed-PDE-Net are shown as the red curves in Figure 8. Since without moment constraints, we do not know the correspondence of the ﬁlters with differential operators. Therefore, we cannot identify the correspondence of the learned variable coefﬁcients either. We plot all the 15 variable coefﬁcients (assuming the underlying PDE is of order ≤ 4 ) in Figure 10.

Original: Comparisons between the PDE-Net and the Frozen-PDE-Net demonstrate the importance of using learnable and yet partially constrained ﬁlters, which is new to  the literature.
Revised: As one can see that the Freed-PDE-Net is better in prediction than the PDE-Net since it has more trainable parameters than the PDE-Net. However, we are unable to identify the PDE from the Free-PDE-Net.

Revised: Q UICK S UMMARY :

Revised: In summary, the numerical experiments show that the PDE-Net is able to conduct accurate prediction and identify the underlying PDE model at the same time, even in a noisy environment.

Revised: Multiple δt -blocks, i e. deeper structure of the PDE-Net, makes the PDE-Net more stable and enables longer time prediction.

Original: Furthermore, using larger ﬁlters helps with stability and can prolong reliable predictions.
Revised: Furthermore, using larger ﬁlters helps with stability and can prolong reliable predictions.

Revised: Comparisons of the PDE-Net with the Frozen-PDE-Net and Freed-PDE-Net demonstrate the importance of using learnable and yet partially constrained ﬁlters, which is new to the literature.

Original:  N UMERICAL S TUDIES : D IFFUSION E QUATIONS WITH N ONLINEAR S OURCE
Revised:  N UMERICAL S TUDIES : D IFFUSION E QUATIONS WITH N ONLINEAR S OURCE

Original: When modeling physical processes like particle transportation or energy transfer, in addition to convection and diffusion, we have to consider source/sink terms.
Revised: When modeling physical processes like particle transportation or energy transfer, in addition to convection and diffusion, we have to consider source/sink terms.

Original: In some problems, the source/sink plays an important role.
Revised: In some problems, the source/sink plays an important role.

Original: For example, when convection-diffusion equations are used to describe the distribution and ﬂow of pollutants in water or atmosphere, identifying the intensity of pollution source is equivalent to ﬁnding the source term, which is important for environmental pollution control problems.[Figure]
Revised: For example, when convection-diffusion equations are used to describe the distribution and ﬂow of pollutants in water or atmosphere, identifying the intensity of pollution source is equivalent to ﬁnding the source term, which is important for environmental pollution control problems.

Original:  S IMULATED D ATA , T RAINING AND T ESTING
Revised: 4. S IMULATED D ATA , T RAINING AND T ESTING

Original: We consider a 2-dimensional linear diffusion equation with a nonlinear source on Ω = [0 , 2 π ] × [0 , 2 π ] , (cid:26) ∂u∂t = c
Revised: We consider a 2-dimensional linear diffusion equation with a nonlinear source on Ω = [0 , 2 π ] × [0 , 2 π ] , (cid:26) [Equation]

Original: ∆ u + f s ( u ) u | t = =
Revised: = [Equation] ) u | [Equation]

Original:  u 0 ( x, y ) , with ( t, x, y ) ∈ [0 , 0 . 2] × Ω , (11) where c = 0 .
Revised: = = u 0 ( x, y ) , with ( t, x, y ) ∈ [0 , 0 . 2] × Ω , (11) where c = 0 .

Original: 3 and f s ( u ) = 15 sin( u ) .
Revised: 3 and f s ( u ) = 15 sin( u ) .

Original: The computation domain Ω is discretized using a 50 ×regular mesh.
Revised: The computation domain Ω is discretized using a 50 ×regular mesh.

Original: Data is generated by solving problem (11) using forward Euler for temporal discretization (with time step size δt = 0 . 0009 ) and central differencing for spatial discretization on 100 × 100 mesh, and then restricted to the 50 × 50 mesh.
Revised: Data is generated by solving problem (11) using forward Euler for temporal discretization (with time step size δt = 0 . 0009 ) and central differencing for spatial discretization on 100 × 100 mesh, and then restricted to the 50 × 50 mesh.

Original: We assume zero boundary condition and the initial value u 0 ( x, y ) is generated by u 0 ( x, y )
Revised: We assume zero boundary condition and the initial value u 0 ( x, y ) is generated by u 0 ( x, y )

Original: =
Revised: =

Original: u (cid:48) 

Revised: u (cid:48)

Original: ( x, y ) x (2 π − x ) y
Revised: ( x, y ) x (2 π − x ) y

Original: (2 π − y ) (2 π ), where u (cid:48) 0 is obtained from (9) with maximum allowable frequency N = 6 .
Revised: (2 π − y ) ( [Equation] ), where u (cid:48) 0 is obtained from (9) with maximum allowable frequency N = 6 .

Original: Same as the numerical setting in Section 3,
Revised: Same as the numerical setting in Section 3,

Original: Gaussian noise is added to each sample sequence u
Revised: Gaussian noise is added to each sample sequence u

Original: ( x, y, t ) , t ∈ [0 , 0 . 2] as described by (10).
Revised: ( x, y, t ) , t ∈ [0 , 0 . 2] as described by (10).

Original: Suppose we know a priori that the underlying PDE is a convection-diffusion equation of order no more than 2 with a nonlinear source depending on the variable u .
Revised: Suppose we know a priori that the underlying PDE is a convection-diffusion equation of order no more than 2 with a nonlinear source depending on the variable u .

Original: Then, the response function F takes the following form
Revised: Then, the response function F takes the following form

Original: [Equation]
Revised: [Equation]

Original: Each δt -block of the PDE-Net can be written as
Revised: Each δt -block of the PDE-Net can be written as

Original: [Equation]
Revised: [Equation]

Original:  where { D 0 , D ij : 1 ≤ i + j ≤ 2 } are convolution operators and { c ij : 1 ≤ i + j ≤ 2 } are 2D arrays which approximate functions f ij ( x, y ) on Ω .
Revised: c [Equation] where { D 0 , D ij : 1 ≤ i + j ≤ 2 } are convolution operators and { c ij : 1 ≤ i + j ≤ 2 } are 2D arrays which approximate functions f ij ( x, y ) on Ω .

Original: The approximation is achieved using piecewise quadratic polynomial interpolation with smooth transitions at the boundaries of each piece.
Revised: The approximation is achieved using piecewise quadratic polynomial interpolation with smooth transitions at the boundaries of each piece.

Original: The approximation of ˜ f s is obtained by piecewise 4th order polynomial approximation over a regular grid of the interval [ − 30 , 30] with 40 grid points.
Revised: The approximation of ˜ f s is obtained by piecewise 4th order polynomial approximation over a regular grid of the interval [ − 30 , 30] with 40 grid points.

Original: The training and testing strategy is exactly the same as in Section 3.
Revised: The training and testing strategy is exactly the same as in Section 3.

Original: In our experiments, the size of the ﬁlters is 7 × 7 .
Revised: In our experiments, the size of the ﬁlters is 7 × 7 .

Original: The total number of trainable parameters for each δt -block is approximately 1 .
Revised: The total number of trainable parameters for each δt -block is approximately 1 .

Original: 2 k.
Revised: 2 k.

Original: 4. R ESULTS AND D ISCUSSIONS
Revised: 4. R ESULTS AND D ISCUSSIONS

Original: This section presents numerical results of the trained PDE-Net using the data set described in Section 4.1.
Revised: This section presents numerical results of the trained PDE-Net using the data set described in Section 4.1.

Original: We will observe how the trained PDE-Net performs in terms of prediction of dynamical behavior and identiﬁcation of the underlying PDE model.
Revised: We will observe how the trained PDE-Net performs in terms of prediction of dynamical behavior and identiﬁcation of the underlying PDE model.

Original: P REDICTING LONG - TIME DYNAMICS
Revised: P REDICTING LONG - TIME DYNAMICS

Original: We demonstrate the ability of the trained PDE-Net in prediction, which in the language of machine learning is the ability to generalize.
Revised: We demonstrate the ability of the trained PDE-Net in prediction, which in the language of machine learning is the ability to generalize.

Original: The testing method is exactly the same as the method described in Section 3.
Revised: The testing method is exactly the same as the method described in Section 3.

Original: Comparisons between PDE-Net and Frozen-PDE-Net are shown in Figure 8, where we can clearly see the advantage of learning the ﬁlters.
Revised: Comparisons between PDE-Net and Frozen-PDE-Net are shown in Figure 11, where we can clearly see the advantage of learning the ﬁlters.

Original: Long-time predictions of the PDE-Net is shown in Figure 9 and we visualize the predicted dynamics in Figure 10.
Revised: Long-time predictions of the PDE-Net is shown in Figure 12 and we visualize the predicted dynamics in Figure 13.

Original: To further demonstrate how well the learned PDE-Net generalizes, we generate initial values following (9) with highest frequency equal to 10, followed by adding noise (10).
Revised: To further demonstrate how well the learned PDE-Net generalizes, we generate initial values following (9) with highest frequency equal to 10, followed by adding noise (10).

Original: Note that the maximum allowable frequency in the training set is only 6.
Revised: Note that the maximum allowable frequency in the training set is only 6.

Original: The results of long-time prediction and the estimated dynamics are shown in Figure 11.
Revised: The results of long-time prediction and the estimated dynamics are shown in Figure 14.

Original: All these results show that the learned PDE-Net performs well in prediction.
Revised: All these results show that the learned PDE-Net performs well in prediction.

Original: D ISCOVERING THE HIDDEN EQUATION
Revised: D ISCOVERING THE HIDDEN EQUATION

Original: For the PDE (11), identifying the PDE amounts to ﬁnding the coefﬁcients { c ij : 1 ≤ i + j ≤ 2 } that approximate { f ij : 1 ≤ i + j ≤ 2 } , and ˜ f s that approximates f s .
Revised: For the PDE (11), identifying the PDE amounts to ﬁnding the coefﬁcients { c ij : 1 ≤ i + j ≤ 2 } that approximate { f ij : 1 ≤ i + j ≤ 2 } , and ˜ f s that approximates f s .

Original: The computed coefﬁcients { c ij : 1 ≤ i + j ≤ 2 } of the trained PDE-Net are shown in Figure 12, and the computed ˜ f s is shown in Figure 13 (left).
Revised: The computed coefﬁcients { c ij : 1 ≤ i + j ≤ 2 } of the trained PDE-Net are shown in Figure 15, and the computed ˜ f s is shown in Figure 16 (left).

Original: Note that the ﬁrst order terms are absent from the PDE (11), and the corresponding coefﬁcients learned by the PDE-Net are indeed close to zero.
Revised: Note that the ﬁrst order terms are absent from the PDE (11), and the corresponding coefﬁcients learned by the PDE-Net are indeed close to zero.

Original: The approximation of f s is more accurate near the center of the interval than near the boundary.
Revised: The approximation of f s is more accurate near the center of the interval than near the boundary.

Original: This is because the value of u in the data set is mostly distributed near the center (Figure 13(right)).
Revised: This is because the value of u in the data set is mostly distributed near the center (Figure 16(right)).

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: In this paper, we designed a deep feed-forward network, called the PDE-Net, to discover the hidden
Revised: In this paper, we designed a deep feed-forward network, called the PDE-Net, to discover the hidden

Original: PDE model from the observed dynamics and to predict the dynamical behavior.
Revised: PDE model from the observed dynamics and to predict the dynamical behavior.

Original: The PDE-Net consists of two major components which are jointly trained: to approximate differential operations by convolutions with properly constrained ﬁlters, and to approximate the nonlinear response by deep neural networks or other machine learning methods.
Revised: The PDE-Net consists of two major components which are jointly trained: to approximate differential operations by convolutions with properly constrained ﬁlters, and to approximate the nonlinear response by deep neural networks or other machine learning methods.

Original: The PDE-Net is suitable for learning PDEs as general as in (1).
Revised: The PDE-Net is suitable for learning PDEs as general as in (1).

Original: However, if we have a prior knowledge on the form of the response function F , we can easily adjust the network architecture by taking advantage of the additional information.
Revised: However, if we have a prior knowledge on the form of the response function F , we can easily adjust the network architecture by taking advantage of the additional information.

Original: This may simplify the training and improve the results.
Revised: This may simplify the training and improve the results.

Original: As an example, we considered a linear variable-coefﬁcient convection-diffusion equation.
Revised: As an example, we considered a linear variable-coefﬁcient convection-diffusion equation.

Original: The results show that the PDE-Net can uncover the hidden equation of the observed dynamics, and predict the dynamical behavior for a relatively long time, even in a noisy environment.
Revised: The results show that the PDE-Net can uncover the hidden equation of the observed dynamics, and predict the dynamical behavior for a relatively long time, even in a noisy environment.

Original: Furthermore, having deep structure (i.e. multiple δt -blocks) and larger learnable ﬁlters can improve the PDE-Net in terms of stability and can prolong reliable predictions.
Revised: Furthermore, having deep structure (i.e. multiple δt -blocks) and larger learnable ﬁlters can improve the PDE-Net in terms of stability and can prolong reliable predictions.

Original: As part of the future work, we will try the proposed framework on real data sets.
Revised: As part of the future work, we will try the proposed framework on real data sets.

Original: One of the important directions is to uncover hidden variables which cannot be measured by sensors directly, such as in data assimilation.
Revised: One of the important directions is to uncover hidden variables which cannot be measured by sensors directly, such as in data assimilation.

Original: Another interesting direction which is worth exploring is to learn stable and consistent numerical schemes for a given PDE model based on the architecture of the PDE-Net.
Revised: Another interesting direction which is worth exploring is to learn stable and consistent numerical schemes for a given PDE model based on the architecture of the PDE-Net