Original: E NFORCING PHYSICS - BASED ALGEBRAIC CONSTRAINTS FOR INFERENCE OF PDE MODELS ON UNSTRUCTURED GRIDS
Revised: E NFORCING PHYSICS - BASED ALGEBRAIC CONSTRAINTS FOR INFERENCE OF PDE MODELS ON UNSTRUCTURED GRIDS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The lack of relevant physical constraints in data-driven models of physical systems, such as neural network parameterized partial differential equations (PDEs), might lead to unrealistic modeling outcomes.
Revised: The lack of relevant physical constraints in data-driven models of physical systems, such as neural network parameterized partial differential equations (PDEs), might lead to unrealistic modeling outcomes.

Original: A majority of approaches to solving this problem are based on forcing a model to satisfy a set of equations representing physical constraints.
Revised: A majority of approaches to solving this problem are based on forcing a model to satisfy a set of equations representing physical constraints.

Original: Currently available approaches can enforce a very limited set of constraints and are applicable only to uniform spatial grids.
Revised: Currently available approaches can enforce a very limited set of constraints and are applicable only to uniform spatial grids.

Original: We propose a method for enforcing general pointwise, differential and integral constraints on unstructured spatial grids.
Revised: We propose a method for enforcing general pointwise, differential and integral constraints on unstructured spatial grids.

Original: Our method is based on representing a model’s output in terms of a function approximation and enforcing constraints on that approximation.
Revised: Our method is based on representing a model’s output in terms of a function approximation and enforcing constraints on that approximation.

Original: We demonstrate wide applicability and strong performance of our approach in data-driven learning of dynamical PDE systems and distributions of physical ﬁelds.
Revised: We demonstrate wide applicability and strong performance of our approach in data-driven learning of dynamical PDE systems and distributions of physical ﬁelds.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multiple works have shown the capability of neural networks to solve complex physical problems and learn the behavior of physical systems from data.
Revised: Multiple works have shown the capability of neural networks to solve complex physical problems and learn the behavior of physical systems from data.

Original: Examples include learning and solving ordinary differential equations (ODEs) [6], partial differential equations (PDEs) [28; 20] and rigid body dynamics [31; 5].
Revised: Examples include learning and solving ordinary differential equations (ODEs) [6], partial differential equations (PDEs) [28; 20] and rigid body dynamics [31; 5].

Original: Purely data-driven models are typically not forced to satisfy physical constraints of the system that generated the data.
Revised: Purely data-driven models are typically not forced to satisfy physical constraints of the system that generated the data.

Original: This might lead to unrealistic predictions that violate some known properties of the underlying physical system.
Revised: This might lead to unrealistic predictions that violate some known properties of the underlying physical system.

Original: Incorporation of relevant constraints allows to make a better use of the available data and makes predictions more physically plausible.
Revised: Incorporation of relevant constraints allows to make a better use of the available data and makes predictions more physically plausible.

Original: The ﬁeld dealing with physics-constrained learning is diverse and offers many approaches to adding constraints to models.
Revised: The ﬁeld dealing with physics-constrained learning is diverse and offers many approaches to adding constraints to models.

Original: We refer the reader to many reviews for details [30; 3; 36; 19].
Revised: We refer the reader to many reviews for details [30; 3; 36; 19].

Original: The approach we consider in this work is based on forcing a model to satisfy algebraic constraints represented by a set of equalities and inequalities.
Revised: The approach we consider in this work is based on forcing a model to satisfy algebraic constraints represented by a set of equalities and inequalities.

Original: This is the most commonly used approach which allows to represent a wide range of constraints and has been shown to work well in many cases [18; 17; 25].
Revised: This is the most commonly used approach which allows to represent a wide range of constraints and has been shown to work well in many cases [18; 17; 25].

Original: However, while many constraints can be represented algebraically, it is not always clear how to evaluate and enforce them.
Revised: However, while many constraints can be represented algebraically, it is not always clear how to evaluate and enforce them.

Original: Currently available approaches to enforcing algebraic constraints are limited to uniform grids and have a very narrow range of constraints they can enforce (e.g. only pointwise, or speciﬁc differential constraints), see Section 5 for details of related work.
Revised: Currently available approaches to enforcing algebraic constraints are limited to uniform grids and have a very narrow range of constraints they can enforce (e.g. only pointwise, or speciﬁc differential constraints), see Section 5 for details of related work.

Original: Such approaches can be readily applied to models based on convolutional neural networks (CNNs) but cannot be extended to recently developed models based on graph neural networks (GNNs)
Revised: Such approaches can be readily applied to models based on convolutional neural networks (CNNs) but cannot be extended to recently developed models based on graph neural networks (GNNs)

Original: [33; 27; 15] and other models working on unstructured grids.
Revised: [33; 27; 15] and other models working on unstructured grids.

Original: We propose a much more general method which allows to enforce pointwise, differential and integral constraints on unstructured spatial grids and demonstrate its applicability in learning of PDE-driven dynamical systems and distributions of physical ﬁelds.
Revised: We propose a much more general method which allows to enforce pointwise, differential and integral constraints on unstructured spatial grids and demonstrate its applicability in learning of PDE-driven dynamical systems and distributions of physical ﬁelds.

Original: The method is based on using a models’s output at the nodes of a grid to construct an interpolant and applying constraints directly to that interpolant (Section 3).
Revised: The method is based on using a models’s output at the nodes of a grid to construct an interpolant and applying constraints directly to that interpolant (Section 3).

Original: Code and data will be made publicly available.
Revised: Code and data will be made publicly available.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: PDE-driven dynamical systems.
Revised: PDE-driven dynamical systems.

Original: Many physical systems can be described in terms of PDEs.
Revised: Many physical systems can be described in terms of PDEs.

Original: Such systems are deﬁned on a bounded domain on which they evolve over time.
Revised: Such systems are deﬁned on a bounded domain on which they evolve over time.

Original: We consider continuous dynamical systems with state u ( t, x ) ∈ R p that evolves over time t ∈ R ≥ 0 and spatial locations x ∈ Ω ⊂ R D .
Revised: We consider continuous dynamical systems with state u ( t, x ) ∈ R p that evolves over time t ∈ R ≥ 0 and spatial locations x ∈ Ω ⊂ R D .

Original: For physical systems, D is typically limited to { 1 , 2 , 3 } although our method will work with any value of D .
Revised: For physical systems, D is typically limited to { 1 , 2 , 3 } although our method will work with any value of D .

Original: We assume the system is governed by an unknown PDE
Revised: We assume the system is governed by an unknown PDE

Original: [Equation]
Revised: [Equation]

Original: which describes the temporal evolution of the system in terms of the locations x , state u and its ﬁrst and higher-order partial derivatives
Revised: which describes the temporal evolution of the system in terms of the locations x , state u and its ﬁrst and higher-order partial derivatives

Original: w.r.t.
Revised: w.r.t.

Original: x .
Revised: x .

Original: The goal of a data-driven PDE model is to learn the dynamics F from data.
Revised: The goal of a data-driven PDE model is to learn the dynamics F from data.

Original: Data for learning F is collected by measuring the state of the system at observation locations ( x 1 , . . .
Revised: Data for learning F is collected by measuring the state of the system at observation locations ( x 1 , . . .

Original: , x N ) over increasing time points ( t 0 , . . .
Revised: , x N ) over increasing time points ( t 0 , . . .

Original: , t M ) .
Revised: , t M ) .

Original: This results in a dataset ( y ( t 0 ) , . . .
Revised: This results in a dataset ( y ( t 0 ) , . . .

Original: , y ( t M )) , where y ( t i ) =
Revised: , y ( t M )) , where y ( t i ) =

Original: ( u ( t i , x 1 ) , . . .
Revised: ( u ( t i , x 1 ) , . . .

Original: , u ( t i , x N )) is a collection of observations.
Revised: , u ( t i , x N )) is a collection of observations.

Original: The dataset is used to train the model to predict ( y ( t 1 ) , . .
Revised: The dataset is used to train the model to predict ( y ( t 1 ) , . .

Original: , y ( t M )) starting from the initial state y ( t 0 ) .
Revised: , y ( t M )) starting from the initial state y ( t 0 ) .

Original: Training is typically done by minimizing an average loss between the model’s predictions u ( t ) and the data y ( t ) .
Revised: Training is typically done by minimizing an average loss between the model’s predictions u ( t ) and the data y ( t ) .

Original: PDE models differ in restrictions they impose on time points (temporal grid) and observation locations (spatial grid).
Revised: PDE models differ in restrictions they impose on time points (temporal grid) and observation locations (spatial grid).

Original: Some models require both grids to be uniform [23], other models relax these requirements and allow arbitrary spatial [27] and spatio-temporal grids [15].
Revised: Some models require both grids to be uniform [23], other models relax these requirements and allow arbitrary spatial [27] and spatio-temporal grids [15].

Original: We build our algebraic constraints method using the model from [15] as the most general one.
Revised: We build our algebraic constraints method using the model from [15] as the most general one.

Original: The model is based on application of the method of lines [32] to Equation 1 which results into a system of ODEs
Revised: The model is based on application of the method of lines [32] to Equation 1 which results into a system of ODEs

Original: [Equation]
Revised: [Equation]

Original: which approximates the solution of Equation 1 at the observation locations x i using their neighboring points N ( i ) , where x N ( i ) and u N ( i ) are the neighbors’ positions and states respectively..
Revised: which approximates the solution of Equation 1 at the observation locations x i using their neighboring points N ( i ) , where x N ( i ) and u N ( i ) are the neighbors’ positions and states respectively, and u i is u ( t, x i ) .

Original: The approximate solution converges to the true solution as N increases.
Revised: The approximate solution converges to the true solution as N increases.

Original: The true dynamics F is approximated by a parametric model F θ whose parameters θ are learned by minimizing the difference between the model’s predictions
Revised: The true dynamics F is approximated by a parametric model F θ whose parameters θ are learned by minimizing the difference between the model’s predictions

Original: [Equation]
Revised: [Equation]

Original: and the data y ( t ) .
Revised: and the data y ( t ) .

Original: The integral in Equation 3 is solved using a numerical ODE solver.
Revised: The integral in Equation 3 is solved using a numerical ODE solver.

Original: In [15], the function F θ was represented by a graph neural network (GNN) which takes states and locations at an observation point i and its neighboring points N ( i ) .
Revised: In [15], the function F θ was represented by a graph neural network (GNN) which takes states and locations at an observation point i and its neighboring points N ( i ) .

Original: The observation points are connected into a grid using Delaunay triangulation which allows to naturally deﬁne N ( i ) as a set of points connected to the point i .
Revised: The observation points are connected into a grid using Delaunay triangulation which allows to naturally deﬁne N ( i ) as a set of points connected to the point i .

Original: However, F θ can be represented by other models and a different neighbor selection criterion can be used.
Revised: However, F θ can be represented by other models and a different neighbor selection criterion can be used.

Original: The model parameters θ are learned by minimizing the MSE between y ( t ) and u ( t )
Revised: The model parameters θ are learned by minimizing the MSE between y ( t ) and u ( t )

Original: [Equation]
Revised: [Equation]

Original: The gradient of L data w.r.t.
Revised: The gradient of L data w.r.t.

Original: θ is evaluated using the adjoint method as shown in [7].
Revised: θ is evaluated using the adjoint method as shown in [7].

Original:  Networks One of the tasks that we consider is learning distributions of physical ﬁelds.
Revised: Generative Adversarial Networks One of the tasks that we consider is learning distributions of physical ﬁelds.

Original: For that purpose we utilize generative adversarial networks (GANs).
Revised: For that purpose we utilize generative adversarial networks (GANs).

Original: A GAN is a generative model consisting of a generator and a discriminator [12].
Revised: A GAN is a generative model consisting of a generator and a discriminator [12].

Original: The generator, G , learns to transform a random variable Z ∼ p Z over a latent space Z to the data space Y in such a way that the discriminator, D , cannot tell the difference between samples generated by G and samples from the data distribution p data .
Revised: The generator, G , learns to transform a random variable Z ∼ p Z over a latent space Z to the data space Y in such a way that the discriminator, D , cannot tell the difference between samples generated by G and samples from the data distribution p data .

Original: Both, G and D are learned by solving the following minimax problem
Revised: Both, G and D are learned by solving the following minimax problem

Original: [Equation]
Revised: [Equation]

Original: Solution of this problem exists and is unique with the optimal generator perfectly mimicking the data distribution [12].
Revised: Solution of this problem exists and is unique with the optimal generator perfectly mimicking the data distribution [12].

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section we presents an approach to evaluating pointwise, differential and integral constraints on unstructured grids.
Revised: In this section we presents an approach to evaluating pointwise, differential and integral constraints on unstructured grids.

Original: Then, we demonstrate how this approach can be used to enforce arbitrary soft and linear hard constraints.
Revised: Then, we demonstrate how this approach can be used to enforce arbitrary soft and linear hard constraints.

Original: 3. E VALUATING CONSTRAINTS ON UNSTRUCTURED GRIDS
Revised: 3. E VALUATING CONSTRAINTS ON UNSTRUCTURED GRIDS

Original: We assume the data y ( t ) is available at observation points ( x 1 , . . .
Revised: We assume the data y ( t ) is available at observation points ( x 1 , . . .

Original: , x N ) and time points ( t 1 , . . .
Revised: , x N ) and time points ( t 1 , . . .

Original: , t M ) and that a model makes predictions u ( t ) at these points.
Revised: , t M ) and that a model makes predictions u ( t ) at these points.

Original: We assume the predictions to be evaluations of an unknown underlying function.
Revised: We assume the predictions to be evaluations of an unknown underlying function.

Original: Since the underlying function is unknown, we cannot impose constraints on it directly.
Revised: Since the underlying function is unknown, we cannot impose constraints on it directly.

Original: Instead, we approximate it by an interpolant u f ( t, x ) and impose constraints on u f ( t, x ) (Figure 1).
Revised: Instead, we approximate it by an interpolant u f ( t, x ) and impose constraints on u f ( t, x ) (Figure 1).

Original: The approximation is constructed from u ( t ) by placing a basis function at each x i and representing u f ( t, x ) as
Revised: The approximation is constructed from u ( t ) by placing a basis function at each x i and representing u f ( t, x ) as

Original: [Equation]
Revised: [Equation]

Original: where φ j is a scalar basis function at x j and α j ∈ R p cients α j ( t ) are obtained from u ( t ) (see Section 3.4)
Revised: where φ j is a scalar basis function at x j and α j ∈ R p cients α j ( t ) are obtained from u ( t ) (see Section 3.4)

Original: Next, we show how to evaluate constraints on u f ( t, x ) using basic building blocks.
Revised: Next, we show how to evaluate constraints on u f ( t, x ) using basic building blocks.

Original: To avoid cluttered notation, we consider equality constraints and assume u ( t, x ) , x ∈ R .
Revised: To avoid cluttered notation, we consider equality constraints and assume u ( t, x ) , x ∈ R .

Original: Generalization to inequality constraints, vector ﬁelds and higher spatial dimensions is straightforward.
Revised: Generalization to inequality constraints, vector ﬁelds and higher spatial dimensions is straightforward.

Original: Pointwise constraints.
Revised: Pointwise constraints.

Original: Consider points z
Revised: Consider points z

Original: = ( z 1 , . .
Revised: = ( z 1 , . .

Original: . , z K ) in Ω on which a pointwise constraint h ( u f ( t, z i ))
Revised: . , z K ) in Ω on which a pointwise constraint h ( u f ( t, z i ))

Original: = 0 should be evaluated.
Revised: = 0 should be evaluated.

Original: Assume the function h : R → R is representable in terms of a ﬁnite number of functions γ m ( u f ( t, z i )) :
Revised: Assume the function h : R → R is representable in terms of a ﬁnite number of functions γ m ( u f ( t, z i )) :

Original: R → R indexed by m .
Revised: R → R indexed by m .

Original: For example, should the constraint be h ( u f ) =
Revised: For example, should the constraint be h ( u f ) =

Original: 3 u f + u 2 f = 0 , then we would deﬁne γ 1 ( u f ) =
Revised: 3 u f + u 2 f = 0 , then we would deﬁne γ 1 ( u f ) =

Original: u f , γ 2 ( u f ) =
Revised: u f , γ 2 ( u f ) =

Original: u 2 f and h ( u f ) = 3 · γ 1 ( u f ) + γ 2 ( u f ) = 0 .
Revised: u 2 f and h ( u f ) = 3 · γ 1 ( u f ) + γ 2 ( u f ) = 0 .

Original: Then, h can be evaluated by evaluating each γ m as
Revised: Then, h can be evaluated by evaluating each γ m as

Original: [Equation]
Revised: [Equation]

Original: where α ( t ) =
Revised: where α ( t ) =

Original: ( α 1 ( t ) , . . .
Revised: ( α 1 ( t ) , . . .

Original: , α N ( t )) T , Φ is K -by- N matrix with elements Φ i,j = φ j
Revised: , α N ( t )) T , Φ is K -by- N matrix with elements Φ i,j = φ j

Original: ( z i ) , and Φ i, · is the i ’th row of Φ .
Revised: ( z i ) , and Φ i, · is the i ’th row of Φ .

Original: Differential constraints.
Revised: Differential constraints.

Original: Consider the same setup as before but now h ( u f ( t, z i ))
Revised: Consider the same setup as before but now h ( u f ( t, z i ))

Original: = 

Revised: =

Original: consists of differential operators and is representable in terms of a ﬁnite number of functions ∂
Revised: consists of differential operators and is representable in terms of a ﬁnite number of functions ∂

Original: q γ m ( u f ( t, z i ))
Revised: q γ m ( u f ( t, z i ))

Original: ∂
Revised: ∂

Original: z qi :
Revised: z qi :

Original: R → R indexed by m , where the derivative order q could be different for each m .
Revised: R → R indexed by m , where the derivative order q could be different for each m .

Original: For example, should the constraint be h ( u f ) =
Revised: For example, should the constraint be h ( u f ) =

Original: 3 u f + u f · ∂u 2 f ∂ x = 0 , then we would deﬁne γ 1 ( u f ) =
Revised: 3 u f + u f · ∂u 2 f ∂ x = 0 , then we would deﬁne γ 1 ( u f ) =

Original: u f , γ 2 ( u f ) =
Revised: u f , γ 2 ( u f ) =

Original: u 2 f and h ( u f ) = 3 · γ 1 ( u f ) + γ 1 ( u f ) · ∂γ 2 ( u f ) ∂ z = 0 .
Revised: u 2 f and h ( u f ) = 3 · γ 1 ( u f ) + γ 1 ( u f ) · ∂γ 2 ( u f ) ∂ z = 0 .

Original: Then, h can be evaluated by evaluating each ∂ q γ m ( u f ( t, z i ))
Revised: Then, h can be evaluated by evaluating each ∂ q γ m ( u f ( t, z i ))

Original: ∂
Revised: ∂

Original: z qi using the generalization of the chain rule (Appendix A) which contains only two types of terms.
Revised: z qi using the generalization of the chain rule (Appendix A) which contains only two types of terms.

Original: The ﬁrst type of terms dγ m du f , . . .
Revised: The ﬁrst type of terms dγ m du f , . . .

Original: , d q γ m du qf can be evaluated using automatic differentiation while the second type of terms ∂u f ∂
Revised: , d q γ m du qf can be evaluated using automatic differentiation while the second type of terms ∂u f ∂

Original: z i , . . . , ∂ q u f ∂ z qi can be evaluated as
Revised: z i , . . . , ∂ q u f ∂ z qi can be evaluated as

Original: [Equation]
Revised: [Equation]

Original: where Φ ( q ) i,j = ∂
Revised: where Φ ( q ) i,j = ∂

Original: q φ j
Revised: q φ j

Original: ( z i ) ∂
Revised: ( z i ) ∂

Original: z q i .
Revised: z q i .

Original: Mixed partial derivatives can be handled in a similar way (Appendix A).
Revised: Mixed partial derivatives can be handled in a similar way (Appendix A).

Original: Integral constraints.
Revised: Integral constraints.

Original: Consider the same setup as before but with h ( u f ( t, x ))
Revised: Consider the same setup as before but with h ( u f ( t, x ))

Original: [Equation]
Revised: [Equation]

Original: Ω τ ( u f ( t, x ))
Revised: Ω τ ( u f ( t, x ))

Original: d x = 0 , where the function τ : R → R is representable in terms of functions γ m ( u f ( t, z i )) :
Revised: d x = 0 , where the function τ : R → R is representable in terms of functions γ m ( u f ( t, z i )) :

Original: R → R similarly to the pointwise constraints.
Revised: R → R similarly to the pointwise constraints.

Original: Then, (cid:82) Ω τ ( u f ( t, x ))
Revised: Then, (cid:82) Ω τ ( u f ( t, x ))

Original: d x can be evaluated using a numerical integration technique, e.g. midpoint rule, Gaussian quadrature or Monte-Carlo integration, as
Revised: d x can be evaluated using a numerical integration technique, e.g. midpoint rule, Gaussian quadrature or Monte-Carlo integration, as

Original: [Equation]
Revised: [Equation]

Original: where K is the number of integration points, µ i are integration coefﬁcients which depend on the grid and integration method, and τ ( u f ( t, z i )) is evaluated as in Equation 7.
Revised: where K is the number of integration points, µ i are integration coefﬁcients which depend on the grid and integration method, and τ ( u f ( t, z i )) is evaluated as in Equation 7.

Original: 3. S OFT CONSTRAINTS
Revised: 3. S OFT CONSTRAINTS

Original: Soft constraints are implemented by minimizing the following loss L data + λr ( h ( u f )) , where λ ∈ R and L data is deﬁned as in Equation 4.
Revised: Soft constraints are implemented by minimizing the following loss L data + λr ( h ( u f )) , where λ ∈ R and L data is deﬁned as in Equation 4.

Original: We set r ( h ( u f ))
Revised: We set r ( h ( u f ))

Original: = 1 KM (cid:80)
Revised: = 1 KM (cid:80)

Original: Ki =1 (cid:80)
Revised: Ki =1 (cid:80)

Original: Mj
Revised: Mj

Original: =1 h ( u f ( t j , z i )) 2 for pointwise and differential constraints and r ( h ( u f ))
Revised: =1 h ( u f ( t j , z i )) 2 for pointwise and differential constraints and r ( h ( u f ))

Original: = 1 M (cid:80) Mj
Revised: = 1 M (cid:80) Mj

Original: =1 h ( u f ( t j , x )) 2 for integral constraints.
Revised: =1 h ( u f ( t j , x )) 2 for integral constraints.

Original: 3. H ARD CONSTRAINTS
Revised: 3. H ARD CONSTRAINTS

Original: Our method allows to implement hard constraints by projecting the interpolant u f ( t, x ) to a subset of functions which satisfy the required constraints.
Revised: Our method allows to implement hard constraints by projecting the interpolant u f ( t, x ) to a subset of functions which satisfy the required constraints.

Original: Namely, if u f ( t, x ) does not satisfy constraints g and h , it is projected to a subset of functions which satisfy the constraint by solving the following optimization problem
Revised: Namely, if u f ( t, x ) does not satisfy constraints g and h , it is projected to a subset of functions which satisfy the constraint by solving the following optimization problem

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the projection is denoted by ˆ u f ( t, x ) and V φ is spanned by the basis functions.
Revised: where the projection is denoted by ˆ u f ( t, x ) and V φ is spanned by the basis functions.

Original: Using the basis representation u f ( t, x ) = (cid:80) Ni =
Revised: Using the basis representation u f ( t, x ) = (cid:80) Ni =

Original: α i ( t ) φ i ( x ) and ˆ u f ( t, x ) = (cid:80) Ni
Revised: α i ( t ) φ i ( x ) and ˆ u f ( t, x ) = (cid:80) Ni

Original: =1 β
Revised: =1 β

Original: i ( t ) φ i ( x ) we can rewrite the optimization problem (10) as
Revised: i ( t ) φ i ( x ) we can rewrite the optimization problem (10) as

Original: [Equation]
Revised: [Equation]

Original: where β ( t ) = ( β 1 ( t ) , . .
Revised: where β ( t ) = ( β 1 ( t ) , . .

Original: . , β N ( t ))
Revised: . , β N ( t ))

Original: T and ˆΦ i,j = (cid:82) Ω
Revised: T and ˆΦ i,j = (cid:82) Ω

Original: φ i ( x ) φ j ( x ) d x .
Revised: φ i ( x ) φ j ( x ) d x .

Original: To train the model end-to-end, the problem (11) should be differentiable.
Revised: To train the model end-to-end, the problem (11) should be differentiable.

Original: Agrawal et.
Revised: Agrawal et.

Original: al.
Revised: al.

Original: [1] proposed differentiable convex optimization which could be used in this case if the problem (11) could be expressed in a DPP-compliant way (see [1]).
Revised: [1] proposed differentiable convex optimization which could be used in this case if the problem (11) could be expressed in a DPP-compliant way (see [1]).

Original: To do that, we restrict ourselves to constraints that can be expressed as an equality or inequality between Aβ ( t ) and b , where A is a constant matrix and b is a constant vector.
Revised: To do that, we restrict ourselves to constraints that can be expressed as an equality or inequality between Aβ ( t ) and b , where A is a constant matrix and b is a constant vector.

Original: This formulation admits pointwise, differential and integral constraints on untransformed u f .
Revised: This formulation admits pointwise, differential and integral constraints on untransformed u f .

Original: The objective function is convex since its Hessian is positive-semideﬁnite i.e. for any v ∈
Revised: The objective function is convex since its Hessian is positive-semideﬁnite i.e. for any v ∈

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This allows to solve the problem (11) and differentiate its solution β ∗ ( t ) w.r.t.
Revised: This allows to solve the problem (11) and differentiate its solution β ∗ ( t ) w.r.t.

Original: α ( t ) .
Revised: α ( t ) .

Original: The model parameters are found by minimizing the following loss function L data + λL proj , where λ ∈ R and L data is deﬁned as in Equation 4 but with u ( t i ) replaced by ˆu ( t i ) =
Revised: The model parameters are found by minimizing the following loss function L data + λL proj , where λ ∈ R and L data is deﬁned as in Equation 4 but with u ( t i ) replaced by ˆu ( t i ) =

Original: (ˆ u f ( t i , x 1 ) , . . .
Revised: (ˆ u f ( t i , x 1 ) , . . .

Original: , ˆ u f ( t i , x N )) .
Revised: , ˆ u f ( t i , x N )) .

Original: We set L proj = 1 NM (cid:80)
Revised: We set L proj = 1 NM (cid:80)

Original: Ni =1 (cid:80) Mj =1 (cid:107) u
Revised: Ni =1 (cid:80) Mj =1 (cid:107) u

Original: f
Revised: f

Original: ( t j , x i ) − ˆ u f ( t j , x i ) (cid:107) 22 .
Revised: ( t j , x i ) − ˆ u f ( t j , x i ) (cid:107) 22 .

Original: The second term makes the optimization procedure prefer models that predict u f close to the feasible set of the problem (11).
Revised: The second term makes the optimization procedure prefer models that predict u f close to the feasible set of the problem (11).

Original: We note that the proposed approach is currently limited to small-scale problems due to existing computational bottlenecks in the implementation of differentiable convex optimization [1].
Revised: We note that the proposed approach is currently limited to small-scale problems due to existing computational bottlenecks in the implementation of differentiable convex optimization [1].

Original: 3. B ASIS FUNCTIONS
Revised: 3. B ASIS FUNCTIONS

Original: Selecting appropriate basis is crucial for efﬁciency and applicability of the proposed method.
Revised: Selecting appropriate basis is crucial for efﬁciency and applicability of the proposed method.

Original: Ideally, the basis should allow efﬁcient construction of u f ( t, x ) from u ( t ) , contain no tunable parameters, and lead to sparse matrices Φ , Φ ( q ) and ˆΦ .
Revised: Ideally, the basis should allow efﬁcient construction of u f ( t, x ) from u ( t ) , contain no tunable parameters, and lead to sparse matrices Φ , Φ ( q ) and ˆΦ .

Original: We consider bases from two families: Lagrange basis functions and radial basis functions (RBFs).
Revised: We consider bases from two families: Lagrange basis functions and radial basis functions (RBFs).

Original: Lagrange basis functions do not have tunable parameters and have compact support which leads to sparse Φ , Φ ( q ) and ˆΦ .
Revised: Lagrange basis functions do not have tunable parameters and have compact support which leads to sparse Φ , Φ ( q ) and ˆΦ .

Original: For the piecewise linear basis the interpolant u f ( t, x ) can be constructed directly from the predictions by setting α ( t ) =
Revised: For the piecewise linear basis the interpolant u f ( t, x ) can be constructed directly from the predictions by setting α ( t ) =

Original: u ( t ) .
Revised: u ( t ) .

Original: However, constructing u f ( t, x ) for a higher order basis, e.g. piecewise quadratic, requires the model to make predictions not only at the observation points, but also at some extra points where the data is not available.
Revised: However, constructing u f ( t, x ) for a higher order basis, e.g. piecewise quadratic, requires the model to make predictions not only at the observation points, but also at some extra points where the data is not available.

Original: In Section 4 we demonstrate one approach to solving this problem.
Revised: In Section 4 we demonstrate one approach to solving this problem.

Original: After extending the state u ( t ) by predictions at the extra nodes, the coefﬁcients α ( t ) can be evaluated similarly to the piecewise linear basis.
Revised: After extending the state u ( t ) by predictions at the extra nodes, the coefﬁcients α ( t ) can be evaluated similarly to the piecewise linear basis.

Original: In this work we use piecewise linear (PWL) and piecewise quadratic (PWQ) bases.
Revised: In this work we use piecewise linear (PWL) and piecewise quadratic (PWQ) bases.

Original: Examples of PWL basis functions are shown in Figure 2.
Revised: Examples of PWL basis functions are shown in Figure 2.

Original: Radial basis functions have a wider range of properties.
Revised: Radial basis functions have a wider range of properties.

Original: Some RBFs have tunable parameters, some don’t.
Revised: Some RBFs have tunable parameters, some don’t.

Original: The matrices Φ , Φ ( q ) and ˆΦ evaluated with RBFs are typically dense, but RBFs with compact support exist (e.g. bump function).
Revised: The matrices Φ , Φ ( q ) and ˆΦ evaluated with RBFs are typically dense, but RBFs with compact support exist (e.g. bump function).

Original: The interpolant u f ( t, x ) can be constructed by evaluating α ( t ) =
Revised: The interpolant u f ( t, x ) can be constructed by evaluating α ( t ) =

Original: K − 1 u ( t ) , where K − 1 is the inverse of the interpolation matrix of the given RBF and
Revised: K − 1 u ( t ) , where K − 1 is the inverse of the interpolation matrix of the given RBF and

Original: K ij = φ ( (cid:107) x i − x j (cid:107) ) , where φ is an RBF and x i , x j are observation locations.
Revised: K ij = φ ( (cid:107) x i − x j (cid:107) ) , where φ is an RBF and x i , x j are observation locations.

Original: In this work we use the cubic RBF basis i.e. φ ( r ) =
Revised: In this work we use the cubic RBF basis i.e. φ ( r ) =

Original: r 3 .
Revised: r 3 .

Original: We use PyTorch [26] to handle sparse matrices and to evaluate K − 1 u ( t ) in a differentiable way.
Revised: We use PyTorch [26] to handle sparse matrices and to evaluate K − 1 u ( t ) in a differentiable way.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In the following experiments we use the relative error between the data y ( t ) and model predictions u ( t ) deﬁned as (cid:107) y ( t )
Revised: In the following experiments we use the relative error between the data y ( t ) and model predictions u ( t ) deﬁned as (cid:107) y ( t )

Original: [Equation]
Revised: [Equation]

Original: ( t
Revised: ( t

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:107) y
Revised: (cid:107) y

Original: ( t
Revised: ( t

Original: [Equation]
Revised: [Equation]

Original: (cid:107)and consider only soft constraints.
Revised: (cid:107)and consider only soft constraints.

Original: We present an experiment with hard constraints implemented as shown in Section 3.3 in Appendix D. Data generation is described in
Revised: We present an experiment with hard constraints implemented as shown in Section 3.3 in Appendix D. Data generation is described in

Original: Appendix B. Training, testing and modeling details are in Appendix C. All experiments were run on a single NVIDIA Quadro P5000 GPU.
Revised: Appendix B. Training, testing and modeling details are in Appendix C. All experiments were run on a single NVIDIA Quadro P5000 GPU.

Original: All errors bars represent one standard deviation of the results over ﬁve random seeds.
Revised: All errors bars represent one standard deviation of the results over ﬁve random seeds.

Original: 4. R EPLACING EXISTING METHODS
Revised: 4. R EPLACING EXISTING METHODS

Original: In this experiment we take existing models which incorporate physics-based constraints in training and replace their constraint enforcing approaches with ours.
Revised: In this experiment we take existing models which incorporate physics-based constraints in training and replace their constraint enforcing approaches with ours.

Original: We consider two works.
Revised: We consider two works.

Original: First, [37] which trains a GAN to produce divergence-free vector ﬁelds using zero-divergence constraint.
Revised: First, [37] which trains a GAN to produce divergence-free vector ﬁelds using zero-divergence constraint.

Original: Second, [10] which predicts warping ﬁelds driving the evolution of sea surface temperature by observing snapshots of the temperature over time while enforcing gradient and divergence constraints on the warping ﬁelds (see Appendix C for more details).
Revised: Second, [10] which predicts warping ﬁelds driving the evolution of sea surface temperature by observing snapshots of the temperature over time while enforcing gradient and divergence constraints on the warping ﬁelds (see Appendix C for more details).

Original: Both models work on uniform grids which allows them to evaluate constraints using ﬁnite differences.
Revised: Both models work on uniform grids which allows them to evaluate constraints using ﬁnite differences.

Original: For comparison, we replace ﬁnite differences with our method and observe how it changes the models’ performance.
Revised: For comparison, we replace ﬁnite differences with our method and observe how it changes the models’ performance.

Original: In both cases we use the PWL basis.
Revised: In both cases we use the PWL basis.

Original: For [37] we track the mean divergence and discriminator loss.
Revised: For [37] we track the mean divergence and discriminator loss.

Original: Results of the original approach are as follows: mean divergence 0 .
Revised: Results of the original approach are as follows: mean divergence 0 .

Original: 079 and discriminator loss 0 .
Revised: 079 and discriminator loss 0 .

Original: With our method the mean divergence was 0 .
Revised: With our method the mean divergence was 0 .

Original: and the discriminator loss was 0 .
Revised: and the discriminator loss was 0 .

Original: Both approaches results in similar discriminator losses but our approach produces a smaller mean divergence (smaller is better).
Revised: Both approaches results in similar discriminator losses but our approach produces a smaller mean divergence (smaller is better).

Original: Our method increased the runtime per epoch by 6 %.
Revised: Our method increased the runtime per epoch by 6 %.

Original: For [10] we track the total, divergence and smoothness losses which, with the original approach, were 0 .
Revised: For [10] we track the total, divergence and smoothness losses which, with the original approach, were 0 .

Original: 139 , 8 . 4 · 10 − 5 and 1 . 51 · 10 − 4 , respectively.
Revised: 139 , 8 . 4 · 10 − 5 and 1 . 51 · 10 − 4 , respectively.

Original: With our approach the losses were 0 .
Revised: With our approach the losses were 0 .

Original: 139 , 8 . 3 · 10 −and 1 . 51 · 10 − 4 , respectively.
Revised: 139 , 8 . 3 · 10 −and 1 . 51 · 10 − 4 , respectively.

Original: Both methods produce very similar results.
Revised: Both methods produce very similar results.

Original: Our method increased the runtime per epoch by 30 %.
Revised: Our method increased the runtime per epoch by 30 %.

Original: Overall, replacing existing constraint enforcing approaches by ours on data from uniform grids resulted in comparable model performance, except for runtime which was slightly increased.
Revised: Overall, replacing existing constraint enforcing approaches by ours on data from uniform grids resulted in comparable model performance, except for runtime which was slightly increased.

Original: 4. C AHN -H ILLIARD EQUATION WITH AN INTEGRAL CONSTRAINT
Revised: 4. C AHN -H ILLIARD EQUATION WITH AN INTEGRAL CONSTRAINT

Original: We start with the 1D Cahn-Hilliard equation
Revised: We start with the 1D Cahn-Hilliard equation

Original: [Equation]
Revised: [Equation]

Original: (13) which is known to conserve the state u i.e. h ( u ) = (cid:82) Ω u ( t, x ) dx − C = 

Revised: (13) which is known to conserve the state u i.e. h ( u ) = (cid:82) Ω u ( t, x ) dx − C =

Original: at all time points, where
Revised: at all time points, where

Original: C = (cid:82) Ω u (0 , x ) dx is a constant.
Revised: C = (cid:82) Ω u (0 , x ) dx is a constant.

Original: This is an example of a conservation law which are abundant in nature and are important class of constraints that data-driven models of physical systems should satisfy.
Revised: This is an example of a conservation law which are abundant in nature and are important class of constraints that data-driven models of physical systems should satisfy.

Original: Conservation laws can be expressed in differential and integral forms and this experiment demonstrates how the integral form can be enforced.
Revised: Conservation laws can be expressed in differential and integral forms and this experiment demonstrates how the integral form can be enforced.

Original: The constraint is evaluated using the midpoint rule as shown in the previous section with a single γ 1 being the identity function.
Revised: The constraint is evaluated using the midpoint rule as shown in the previous section with a single γ 1 being the identity function.

Original: We use PWL, PWQ and cubic RBF bases and compare the results to an unconstrained model.
Revised: We use PWL, PWQ and cubic RBF bases and compare the results to an unconstrained model.

Original: For training we use 30 , 60 and 120 simulations while the test set consist of 60 simulations.
Revised: For training we use 30 , 60 and 120 simulations while the test set consist of 60 simulations.

Original: Simulations in the training/test data last for 0 . 0015 /
Revised: Simulations in the training/test data last for 0 . 0015 /

Original: 0030 seconds and contain 50 / 100 uniformly spaced time points.
Revised: 0030 seconds and contain 50 / 100 uniformly spaced time points.

Original: The full spatial grid consists of 101 uniformly spaced nodes.
Revised: The full spatial grid consists of 101 uniformly spaced nodes.

Original: We randomly sample 50 %, 75 % and 100 % of the nodes and train/test on the resulting (irregular) spatial grid.
Revised: We randomly sample 50 %, 75 % and 100 % of the nodes and train/test on the resulting (irregular) spatial grid.

Original: Training and testing is done with identical spatial grids.
Revised: Training and testing is done with identical spatial grids.

Original: An example of a spatial grid with 50 % of nodes is shown in Figure 3.
Revised: An example of a spatial grid with 50 % of nodes is shown in Figure 3.

Original: We evaluate the constraint on a uniform grid with 200 nodes placed on top of the original grid.
Revised: We evaluate the constraint on a uniform grid with 200 nodes placed on top of the original grid.

Original: To learn the dynamics of the system we use the model from [15] (Section 2).
Revised: To learn the dynamics of the system we use the model from [15] (Section 2).

Original: We found that using a GNN produced poor results.
Revised: We found that using a GNN produced poor results.

Original: For that reason we represented the function F θ with a multiplayer perceptron (MLP) which updates the state of each node based on the states of all other nodes in the grid (results for a GNN are in Appendix E).
Revised: For that reason we represented the function F θ with a multiplayer perceptron (MLP) which updates the state of each node based on the states of all other nodes in the grid (results for a GNN are in Appendix E).

Original: The MLP contains two hidden layers with Leaky ReLU nonlinearities.
Revised: The MLP contains two hidden layers with Leaky ReLU nonlinearities.

Original: The number of hidden neurons was set to the number of nodes in the grid.
Revised: The number of hidden neurons was set to the number of nodes in the grid.

Original: The coefﬁcients α ( t ) for the PWL and cubic bases can be evaluated directly from the model predictions at the grid nodes.
Revised: The coefﬁcients α ( t ) for the PWL and cubic bases can be evaluated directly from the model predictions at the grid nodes.

Original: But the PWQ basis requires extra predictions to be available between the nodes.
Revised: But the PWQ basis requires extra predictions to be available between the nodes.

Original: This is problematic since there is no data at these points to guide the model’s predictions.
Revised: This is problematic since there is no data at these points to guide the model’s predictions.

Original: To solve this problem we introduce a small MLP which is applied to consecutive pairs of nodes.
Revised: To solve this problem we introduce a small MLP which is applied to consecutive pairs of nodes.

Original: The
Revised: The

Original: MLP takes the states at both nodes and the distance between them as the input and estimates the state at the midpoint between the two nodes.
Revised: MLP takes the states at both nodes and the distance between them as the input and estimates the state at the midpoint between the two nodes.

Original: The MLP is trained jointly with the main model and uses only the constraint-related loss term during training.
Revised: The MLP is trained jointly with the main model and uses only the constraint-related loss term during training.

Original: For testing, we construct the interpolant u f ( t, x ) using the thin plate spline basis ( φ ( r ) =
Revised: For testing, we construct the interpolant u f ( t, x ) using the thin plate spline basis ( φ ( r ) =

Original: r 2 log r ) and evaluate the constraint on that interpolant.
Revised: r 2 log r ) and evaluate the constraint on that interpolant.

Original: This allows to make a fair comparison between the unconstrained model and different bases and avoid biasing or oveﬁtting to bases used for training.
Revised: This allows to make a fair comparison between the unconstrained model and different bases and avoid biasing or oveﬁtting to bases used for training.

Original: Figure 4 shows results of the experiment.
Revised: Figure 4 shows results of the experiment.

Original: We observe that changing the node fraction does not signiﬁcantly affect the relative errors but has noticeable effect on constraint violations, especially for the unconstrained model.
Revised: We observe that changing the node fraction does not signiﬁcantly affect the relative errors but has noticeable effect on constraint violations, especially for the unconstrained model.

Original: Constrained models tend to show similar or better performance than the unconstrained model.
Revised: Constrained models tend to show similar or better performance than the unconstrained model.

Original: Among all bases, the cubic basis consistently results in lower relative errors and constraint violations.
Revised: Among all bases, the cubic basis consistently results in lower relative errors and constraint violations.

Original: However, the simpler PWL basis often performs on par with the cubic basis, especially on denser spatial grids.
Revised: However, the simpler PWL basis often performs on par with the cubic basis, especially on denser spatial grids.

Original: We also observe that coarsening of the grid increases the constraint violation gap between constrained and unconstrained models and that this gap seems to not close as we increase the amount of training data.
Revised: We also observe that coarsening of the grid increases the constraint violation gap between constrained and unconstrained models and that this gap seems to not close as we increase the amount of training data.

Original: The PWQ basis performs rather poorly on ﬁne grids which is likely due to a suboptimal approach to evaluating the state at the extra nodes.
Revised: The PWQ basis performs rather poorly on ﬁne grids which is likely due to a suboptimal approach to evaluating the state at the extra nodes.

Original: A better approach could consider not only pairs of points but also larger neighborhoods.
Revised: A better approach could consider not only pairs of points but also larger neighborhoods.

Original: Nonetheless, the PWQ basis achieves good performance on coarse grids which shows that piecewise bases of higher order could potentially be used to enforce constraints.
Revised: Nonetheless, the PWQ basis achieves good performance on coarse grids which shows that piecewise bases of higher order could potentially be used to enforce constraints.

Original: This will allow to scale to grids with a large number of nodes due to sparsity of the constraint matrices and efﬁcient evaluation of α .
Revised: This will allow to scale to grids with a large number of nodes due to sparsity of the constraint matrices and efﬁcient evaluation of α .

Original: 4. H EAT EQUATION WITH A MONOTONICITY CONSTRAINT
Revised: 4. H EAT EQUATION WITH A MONOTONICITY CONSTRAINT

Original: We impose constraints on a 2D system governed by the heat equation ∂u∂t = ∇ 2 u for which the generated initial conditions (ICs) are monotone in one direction.
Revised: We impose constraints on a 2D system governed by the heat equation ∂u∂t = ∇ 2 u for which the generated initial conditions (ICs) are monotone in one direction.

Original: Since the ICs are monotone, the state u remains monotone at all time points as well.
Revised: Since the ICs are monotone, the state u remains monotone at all time points as well.

Original: We enforce the monotonicity constraint as ∂u∂x ≥ 0 .
Revised: We enforce the monotonicity constraint as ∂u∂x ≥ 0 .

Original: The constraint is evaluated as shown in the previous section with γ 1 being the identity function.
Revised: The constraint is evaluated as shown in the previous section with γ 1 being the identity function.

Original: For training we use 15 , 30 and 90 simulations while the test set consist of 120 simulations.
Revised: For training we use 15 , 30 and 90 simulations while the test set consist of 120 simulations.

Original: Simulations in the training/test data last for 0 .
Revised: Simulations in the training/test data last for 0 .

Original: 2 seconds and contain 21 / 41 uniformly spaced time points.
Revised: 2 seconds and contain 21 / 41 uniformly spaced time points.

Original: The full spatial grid consists of 1087 nodes.
Revised: The full spatial grid consists of 1087 nodes.

Original: We randomly sample 33%, 66% and 100% of the nodes and train/test on the resulting (irregular) spatial grid.
Revised: We randomly sample 33%, 66% and 100% of the nodes and train/test on the resulting (irregular) spatial grid.

Original: Training and testing is done with identical spatial grids.
Revised: Training and testing is done with identical spatial grids.

Original: Spatial grid with 100% of nodes is shown in
Revised: Spatial grid with 100% of nodes is shown in

Original: To learn the dynamics of the system we use the model from [15] directly with the messaging and aggregation networks being MLPs with a single hidden layer consisting of 60 neurons with Tanh nonlinearities and the input/output sizes of 4 / 40 and 41 / 1 respectively.
Revised: To learn the dynamics of the system we use the model from [15] directly with the messaging and aggregation networks being MLPs with a single hidden layer consisting of 60 neurons with Tanh nonlinearities and the input/output sizes of 4 / 40 and 41 / 1 respectively.

Original: During testing, we use predictions of the models to construct an interpolant u f ( t, x ) using the thin plate spline basis and evaluate the constraint on that interpolant.
Revised: During testing, we use predictions of the models to construct an interpolant u f ( t, x ) using the thin plate spline basis and evaluate the constraint on that interpolant.

Original: This allows to make a fair comparison between the unconstrained model and different bases.
Revised: This allows to make a fair comparison between the unconstrained model and different bases.

Original: Figure 7 shows results of the experiment.
Revised: Figure 7 shows results of the experiment.

Original: We observe that changing the node fraction equally increases relative errors of all models and has noticeable effect on constraint violations, especially for the unconstrained model.
Revised: We observe that changing the node fraction equally increases relative errors of all models and has noticeable effect on constraint violations, especially for the unconstrained model.

Original: Constrained models tend to show slightly higher or comparable relative errors but noticeably lower constraint violations than the unconstrained model.
Revised: Constrained models tend to show slightly higher or comparable relative errors but noticeably lower constraint violations than the unconstrained model.

Original: The cubic and PWL bases perform equally well in this case.
Revised: The cubic and PWL bases perform equally well in this case.

Original: Similarly to the experiment in the previous section, we observe that coarsening of the grid introduces a larger constraint violation gap between constrained and unconstrained models and that this gap seems to not close as we increase the amount of training data.
Revised: Similarly to the experiment in the previous section, we observe that coarsening of the grid introduces a larger constraint violation gap between constrained and unconstrained models and that this gap seems to not close as we increase the amount of training data.

Original: Figure 6 shows qualitative difference between predictions of constrained and unconstrained models.
Revised: Figure 6 shows qualitative difference between predictions of constrained and unconstrained models.

Original: It can be noted that predictions from the constrained model have noticeably smoother contours thus making the ﬁeld more monotonous in the horizontal direction.
Revised: It can be noted that predictions from the constrained model have noticeably smoother contours thus making the ﬁeld more monotonous in the horizontal direction.

Original: 4. L EARNING DISTRIBUTIONS OF PHYSICAL FIELDS
Revised: 4. L EARNING DISTRIBUTIONS OF PHYSICAL FIELDS

Original: are
Revised: are

Original: We demonstrate the effect of adding constraints to a GAN when learning distributions of physical ﬁelds on unstructured grids.
Revised: We demonstrate the effect of adding constraints to a GAN when learning distributions of physical ﬁelds on unstructured grids.

Original: We use Wasserstein GAN (WGAN)
Revised: We use Wasserstein GAN (WGAN)

Original: [2] as a more stable variant of a GAN.
Revised: [2] as a more stable variant of a GAN.

Original: We use MLPs as a generator and discriminator.
Revised: We use MLPs as a generator and discriminator.

Original: Unconstrained and constrained models are trained for 1.2M iterations.
Revised: Unconstrained and constrained models are trained for 1.2M iterations.

Original: Constraints are enabled only after 600k iterations.
Revised: Constraints are enabled only after 600k iterations.

Original: Constrained models are trained similarly to the unconstrained ones but with a modiﬁed generator loss deﬁned as L G + λ ln (1 + L C ) , where
Revised: Constrained models are trained similarly to the unconstrained ones but with a modiﬁed generator loss deﬁned as L G + λ ln (1 + L C ) , where

Original: L G is the standard generator loss and L C is the constraint-based loss.
Revised: L G is the standard generator loss and L C is the constraint-based loss.

Original: We deﬁne L C as the mean value of h ( u ) 2 , where h is a constraint evaluated at the centroid of each cell in the grid.
Revised: We deﬁne L C as the mean value of h ( u ) 2 , where h is a constraint evaluated at the centroid of each cell in the grid.

Original: 4.4. Z ERO - DIVERGENCE FIELDS
Revised: 4.4. Z ERO - DIVERGENCE FIELDS

Original: Divergence-free vector ﬁelds are often encountered in solutions of ﬂuid dynamics problems.
Revised: Divergence-free vector ﬁelds are often encountered in solutions of ﬂuid dynamics problems.

Original: The divergence-free constraint on a vector ﬁeld u ( x, y )
Revised: The divergence-free constraint on a vector ﬁeld u ( x, y )

Original: = ( u 1 ( x, y ) , u 2 ( x, y ))
Revised: = ( u 1 ( x, y ) , u 2 ( x, y ))

Original: T is deﬁned as h ( u ) =
Revised: T is deﬁned as h ( u ) =

Original: ∂u 1 ∂x + ∂u 2 ∂y = 0 .
Revised: ∂u 1 ∂x + ∂u 2 ∂y = 0 .

Original: The constraint is enforced using the PWL basis.
Revised: The constraint is enforced using the PWL basis.

Original: We generated a dataset with 10k divergence-free ﬁelds on an unstructured grid with 1050 nodes (Figure 14) and used a WGAN to learn a distribution over such ﬁelds.
Revised: We generated a dataset with 10k divergence-free ﬁelds on an unstructured grid with 1050 nodes (Figure 14) and used a WGAN to learn a distribution over such ﬁelds.

Original: Note that the generated ﬁelds are not entirely divergence-free but have small residual divergence due to discretization errors.
Revised: Note that the generated ﬁelds are not entirely divergence-free but have small residual divergence due to discretization errors.

Original: Figure 9a shows that there is a clear difference in the quality of the samples generated by the unconstrained and constrained models.
Revised: Figure 9a shows that there is a clear difference in the quality of the samples generated by the unconstrained and constrained models.

Original: Samples from the constrained model are smoother and more similar to the data.
Revised: Samples from the constrained model are smoother and more similar to the data.

Original: Quantitative comparison of the samples presented in Figure 8 shows that the constrained model generates ﬁelds that have much lower constraint violation and divergence distribution very similar to that of the data.
Revised: Quantitative comparison of the samples presented in Figure 8 shows that the constrained model generates ﬁelds that have much lower constraint violation and divergence distribution very similar to that of the data.

Original: 4.4. Z ERO -L APLACIAN FIELDS
Revised: 4.4. Z ERO -L APLACIAN FIELDS

Original: Fields with zero Laplacian represent solutions to some PDEs, for example the steady-state heat equation.
Revised: Fields with zero Laplacian represent solutions to some PDEs, for example the steady-state heat equation.

Original: The zero-Laplacian constraint on a scalar ﬁeld u ( x, y ) is deﬁned as h ( u ) =
Revised: The zero-Laplacian constraint on a scalar ﬁeld u ( x, y ) is deﬁned as h ( u ) =

Original: [Equation]
Revised: [Equation]

Original: 2 = 0 .
Revised: 2 = 0 .

Original: The constraint is enforced using the cubic basis as the PWL basis has zero second derivatives
Revised: The constraint is enforced using the cubic basis as the PWL basis has zero second derivatives

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: everywhere.
Revised: everywhere.

Original: We generated a dataset with 10k Laplacian-free ﬁelds on an unstructured grid withnodes (Figure 14) and used a WGAN to learn a distribution over such ﬁelds.
Revised: We generated a dataset with 10k Laplacian-free ﬁelds on an unstructured grid withnodes (Figure 14) and used a WGAN to learn a distribution over such ﬁelds.

Original: Note that the generated ﬁelds are not entirely Laplacian-free due to discretization errors.
Revised: Note that the generated ﬁelds are not entirely Laplacian-free due to discretization errors.

Original: Results of the experiment are shown in Figures 9b and 8.
Revised: Results of the experiment are shown in Figures 9b and 8.

Original: Similarly to the divergence-free case, visual quality of the ﬁelds generated by the constrained model is signiﬁcantly better than for the unconstrained model.
Revised: Similarly to the divergence-free case, visual quality of the ﬁelds generated by the constrained model is signiﬁcantly better than for the unconstrained model.

Original: Quantitative comparison of the samples presented in Figure 8 shows that the constrained model generates ﬁelds that have much lower constraint violation and Laplacian distribution very similar to that of the data.
Revised: Quantitative comparison of the samples presented in Figure 8 shows that the constrained model generates ﬁelds that have much lower constraint violation and Laplacian distribution very similar to that of the data.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Soft constraints.
Revised: Soft constraints.

Original: Soft constraints are widely used due to being relatively easy to implement.
Revised: Soft constraints are widely used due to being relatively easy to implement.

Original: Examples include lake temperature prediction [18; 16], trafﬁc simulation [22], ﬂuid and climate modeling [11; 10; 4], where constraints are evaluated pointwise or using ﬁnite differences.
Revised: Examples include lake temperature prediction [18; 16], trafﬁc simulation [22], ﬂuid and climate modeling [11; 10; 4], where constraints are evaluated pointwise or using ﬁnite differences.

Original: Hard constraints.
Revised: Hard constraints.

Original: Approaches to implementing hard constraints are diverse and can be categorized as processing the output of an unconstrained model [4; 17; 24; 34] and designing a model that produces feasible predictions by default [23; 25; 14; 9; 13; 8; 38].
Revised: Approaches to implementing hard constraints are diverse and can be categorized as processing the output of an unconstrained model [4; 17; 24; 34] and designing a model that produces feasible predictions by default [23; 25; 14; 9; 13; 8; 38].

Original: Constrained PDE models Current approaches to enforcing soft [10; 11] and hard [21; 25; 17] constraints are limited to speciﬁc types of constraints and spatial grids.
Revised: Constrained PDE models Current approaches to enforcing soft [10; 11] and hard [21; 25; 17] constraints are limited to speciﬁc types of constraints and spatial grids.

Original: For example, [25; 17] implement only hard differential constraints and both are limited to uniform grids.
Revised: For example, [25; 17] implement only hard differential constraints and both are limited to uniform grids.

Original: Uniform grids allow to evaluate constraints efﬁciently e.g. using ﬁnite differences [10; 21; 25] or fast Fourier transform [17] but assuming that the data lies on a uniform grid might be limiting.
Revised: Uniform grids allow to evaluate constraints efﬁciently e.g. using ﬁnite differences [10; 21; 25] or fast Fourier transform [17] but assuming that the data lies on a uniform grid might be limiting.

Original: Constrained GANs Works such as [37; 17] showed how physics-based constraints beneﬁt training and quality of the generated samples but are also limited to uniform grids.
Revised: Constrained GANs Works such as [37; 17] showed how physics-based constraints beneﬁt training and quality of the generated samples but are also limited to uniform grids.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented a general approach to enforcing algebraic constraints on unstructured grids and showed how it can be used to enforce soft and hard constraints.
Revised: We presented a general approach to enforcing algebraic constraints on unstructured grids and showed how it can be used to enforce soft and hard constraints.

Original: We demonstrated applicability of the approach to learning of PDE-driven dynamical systems and distributions of physical ﬁelds.
Revised: We demonstrated applicability of the approach to learning of PDE-driven dynamical systems and distributions of physical ﬁelds.

Original: We considered two families of basis functions for constructing the interpolant and showed how Lagrange basis functions of order higher than one can be used.
Revised: We considered two families of basis functions for constructing the interpolant and showed how Lagrange basis functions of order higher than one can be used.

Original: Our method allows to drop the unrealistic assumption about uniformity of spatial grids and shows promising results on various tasks.
Revised: Our method allows to drop the unrealistic assumption about uniformity of spatial grids and shows promising results on various tasks.

Original: R EPRODUCIBILITY STATEMENT
Revised: R EPRODUCIBILITY STATEMENT

Original: All details required to reproduce the experiments are provided in Section 4 and Appendices.
Revised: All details required to reproduce the experiments are provided in Section 4 and Appendices.

Original: Code and data used to run the experiments will be made publicly available after the review process.
Revised: Code and data used to run the experiments will be made publicly available after the review process.

Original: A G ENERALIZED CHAIN RULE AND HANDLING MIXED PARTIAL DERIVATIVES .
Revised: A G ENERALIZED CHAIN RULE AND HANDLING MIXED PARTIAL DERIVATIVES .

Original: Let y
Revised: Let y

Original: =
Revised: =

Original: g
Revised: g

Original: ( x 1 , . . .
Revised: ( x 1 , . . .

Original: , x n ) with all arguments being either identical, distinct or grouped.
Revised: , x n ) with all arguments being either identical, distinct or grouped.

Original: Then, partial derivatives of f ( y ) can be evaluated using the Faà di Bruno’s formula
Revised: Then, partial derivatives of f ( y ) can be evaluated using the Faà di Bruno’s formula

Original: [Equation]
Revised: [Equation]

Original: where Π is the set of all partitions of the set { 1 , . .
Revised: where Π is the set of all partitions of the set { 1 , . .

Original: . , n } , B runs through elements of the partition π , f ( m ) denotes m ’th derivative, and | · | is cardinality.
Revised: . , n } , B runs through elements of the partition π , f ( m ) denotes m ’th derivative, and | · | is cardinality.

Original: The formula consists of two terms: f ( | π | ) ( y ) , which can be evaluated using automatic differentiation, and
Revised: The formula consists of two terms: f ( | π | ) ( y ) , which can be evaluated using automatic differentiation, and

Original: [Equation]
Revised: [Equation]

Original: j , which can be evaluated as shown in Equation 8.
Revised: j , which can be evaluated as shown in Equation 8.

Original: In case that all x 1 , . . .
Revised: In case that all x 1 , . . .

Original: , x n are identical, the mixed derivative ∂ n f ( y ) ∂x
Revised: , x n are identical, the mixed derivative ∂ n f ( y ) ∂x

Original: [Equation]
Revised: [Equation]

Original: ∂x n reduces to ∂ n f ( y ) ∂x n 1 .
Revised: ∂x n reduces to ∂ n f ( y ) ∂x n 1 .

Original: B D ATA GENERATION
Revised: B D ATA GENERATION

Original: In all cases we run simulation on a ﬁne grid and then interpolate the results to a coarser grid represented as the "full grid" in the experiments.
Revised: In all cases we run simulation on a ﬁne grid and then interpolate the results to a coarser grid represented as the "full grid" in the experiments.

Original: B. C AHN -H ILLIARD EQUATION WITH AN INTEGRAL CONSTRAINT
Revised: B. C AHN -H ILLIARD EQUATION WITH AN INTEGRAL CONSTRAINT

Original: Training and testing data was obtained by solving
Revised: Training and testing data was obtained by solving

Original: [Equation]
Revised: [Equation]

Original: (14) on a unit interval with periodic boundary conditions and (cid:15) = 0 . 04 .
Revised: (14) on a unit interval with periodic boundary conditions and (cid:15) = 0 . 04 .

Original: The domain was represented by a uniform grid with 100 nodes and the time step was set to 1 . 

Revised: The domain was represented by a uniform grid with 100 nodes and the time step was set to 1 .

Original: e-
Revised: e-

Original: 6 sec.
Revised: 6 sec.

Original: The initial conditions u 0 ( x ) were generated as follows
Revised: The initial conditions u 0 ( x ) were generated as follows

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where λ i , γ i ∼ Unif ( − 1 , 1) and s ∼ Unif (0 , 1) .
Revised: where λ i , γ i ∼ Unif ( − 1 , 1) and s ∼ Unif (0 , 1) .

Original: Examples of the simulations are shown in Figure 10.
Revised: Examples of the simulations are shown in Figure 10.

Original: B. H EAT EQUATION WITH A MONOTONICITY CONSTRAINT
Revised: B. H EAT EQUATION WITH A MONOTONICITY CONSTRAINT

Original: Training and testing data was obtained by solving
Revised: Training and testing data was obtained by solving

Original: [Equation]
Revised: [Equation]

Original: 17) on a unit square with zero Neumann boundary conditions and D = 0 .
Revised: 17) on a unit square with zero Neumann boundary conditions and D = 0 .

Original: The domain was represented by an unstructured grid with 2971 nodes and the time step was set to 0 .
Revised: The domain was represented by an unstructured grid with 2971 nodes and the time step was set to 0 .

Original: 001 sec.
Revised: 001 sec.

Original: The initial conditions u 0 ( x ) were generated as
Revised: The initial conditions u 0 ( x ) were generated as

Original: [Equation]
Revised: [Equation]

Original: where ω i ∼ Unif (0 . 1 , 1 . 1)
Revised: where ω i ∼ Unif (0 . 1 , 1 . 1)

Original: , λ i , γ i , s ∼ Unif ( − 1 , 1) .
Revised: , λ i , γ i , s ∼ Unif ( − 1 , 1) .

Original: Examples of the simulations are shown in Figure 11.
Revised: Examples of the simulations are shown in Figure 11.

Original: B. GAN WITH A DIVERGENCE CONSTRAINT
Revised: B. GAN WITH A DIVERGENCE CONSTRAINT

Original: The data was generated by sampling random velocity ﬁelds and then projecting them to the space of divergence-free ﬁelds.
Revised: The data was generated by sampling random velocity ﬁelds and then projecting them to the space of divergence-free ﬁelds.

Original: The procedure was as follows.
Revised: The procedure was as follows.

Original: First, a random velocity ﬁeld u 0 ( x, y ) was generated on a unit square by generating each component i as
Revised: First, a random velocity ﬁeld u 0 ( x, y ) was generated on a unit square by generating each component i as

Original: [Equation]
Revised: [Equation]

Original: where N = 10 and λ kl , γ kl ∼ N (0 , 1)
Revised: where N = 10 and λ kl , γ kl ∼ N (0 , 1)

Original: Then, the divergence-free component of u 0 ( x, y ) , denoted by u ∗ 0 ( x, y ) , was extracted by using the projection method by solving ∇ · u 0 = ∇ 2 φ for φ and then evaluating u ∗ 0 ( x, y )
Revised: Then, the divergence-free component of u 0 ( x, y ) , denoted by u ∗ 0 ( x, y ) , was extracted by using the projection method by solving ∇ · u 0 = ∇ 2 φ for φ and then evaluating u ∗ 0 ( x, y )

Original: =
Revised: =

Original: u 0 ( x, y ) − ∇ φ .
Revised: u 0 ( x, y ) − ∇ φ .

Original: Finally, the data was scaled to [ − 1 , 1] .
Revised: Finally, the data was scaled to [ − 1 , 1] .

Original: B. GAN WITH A L APLACIAN CONSTRAINT
Revised: B. GAN WITH A L APLACIAN CONSTRAINT

Original: The data was generated by solving
Revised: The data was generated by solving

Original: [Equation]
Revised: [Equation]

Original: (24) on a unit square with Dirichlet boundary conditions.
Revised: (24) on a unit square with Dirichlet boundary conditions.

Original: The domain was represented by an unstructured grid with 2971 nodes.
Revised: The domain was represented by an unstructured grid with 2971 nodes.

Original: The boundary conditions were generated by generating random functions u 0 ( x ) and using their boundary values as the boundary conditions.
Revised: The boundary conditions were generated by generating random functions u 0 ( x ) and using their boundary values as the boundary conditions.

Original: The functions u 0 ( x ) were generated as
Revised: The functions u 0 ( x ) were generated as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where N = 5 and λ kl , γ kl ∼ N (0 , 1) .
Revised: where N = 5 and λ kl , γ kl ∼ N (0 , 1) .

Original: The
Revised: The

Original: C M ODELS , TRAINING AND TESTING C. R EPLACING EXISTING METHODS
Revised: C M ODELS , TRAINING AND TESTING C. R EPLACING EXISTING METHODS

Original: For our comparisons we considered experiments from two works.
Revised: For our comparisons we considered experiments from two works.

Original: Next, we provide some details about these experiments.
Revised: Next, we provide some details about these experiments.

Original: The ﬁrst experiment was taken from [37] Section 3.2.
Revised: The ﬁrst experiment was taken from [37] Section 3.2.

Original: The experiment shows how soft physics-based constraints affect predictions of a GAN learning a distribution of divergence-free ﬁelds.
Revised: The experiment shows how soft physics-based constraints affect predictions of a GAN learning a distribution of divergence-free ﬁelds.

Original: The data is generated on a uniform grid which allows to evaluate divergence using ﬁnite differences.
Revised: The data is generated on a uniform grid which allows to evaluate divergence using ﬁnite differences.

Original: The constraint is enforced through an extra loss term which penalizes violation of the constraint.
Revised: The constraint is enforced through an extra loss term which penalizes violation of the constraint.

Original: The performance metric used is the Frobenius norm of the divergence averaged over all ﬁelds in a batch.
Revised: The performance metric used is the Frobenius norm of the divergence averaged over all ﬁelds in a batch.

Original: For training we used code provided by the authors with original parameters.
Revised: For training we used code provided by the authors with original parameters.

Original: We replaced ﬁnite differences in the constrained evaluation function with our method.
Revised: We replaced ﬁnite differences in the constrained evaluation function with our method.

Original: The second experiment was taken from [10].
Revised: The second experiment was taken from [10].

Original: This work deals with the task of predicting sea surface temperatures at future times given snapshots of the temperature over current and previous times.
Revised: This work deals with the task of predicting sea surface temperatures at future times given snapshots of the temperature over current and previous times.

Original: The model proposed by the authors accomplishes this tasks by taking a sequence of surface temperatures at times t
Revised: The model proposed by the authors accomplishes this tasks by taking a sequence of surface temperatures at times t

Original: i − k , . . .
Revised: i − k , . . .

Original: , t i and predicting the underlying motion ﬁeld which is then used to predict the temperature at time t
Revised: , t i and predicting the underlying motion ﬁeld which is then used to predict the temperature at time t

Original: i +1 .
Revised: i +1 .

Original: Insights about physical properties of the motion ﬁeld were used to constrain the model’s predictions.
Revised: Insights about physical properties of the motion ﬁeld were used to constrain the model’s predictions.

Original: Constraints are imposed on divergence, magnitude and gradients of the motion ﬁeld.
Revised: Constraints are imposed on divergence, magnitude and gradients of the motion ﬁeld.

Original: The data is generated on a uniform grid which allows to evaluate the constraints using ﬁnite differences.
Revised: The data is generated on a uniform grid which allows to evaluate the constraints using ﬁnite differences.

Original: The constraints are enforced through extra loss terms which penalize violation of the constraints.
Revised: The constraints are enforced through extra loss terms which penalize violation of the constraints.

Original: Performance metrics that were used are MSE between the data and model predictions, smoothness loss and divergence loss.
Revised: Performance metrics that were used are MSE between the data and model predictions, smoothness loss and divergence loss.

Original: For training we used code provided by the authors with original parameters.
Revised: For training we used code provided by the authors with original parameters.

Original: We replaced ﬁnite differences in the constrained evaluation function with our method.
Revised: We replaced ﬁnite differences in the constrained evaluation function with our method.

Original: C. C AHN -H ILLIARD EQUATION WITH AN INTEGRAL CONSTRAINT
Revised: C. C AHN -H ILLIARD EQUATION WITH AN INTEGRAL CONSTRAINT

Original: In all experiments with the Cahn-Hilliard equation we represent the dynamics function F θ by an MLP with 2 hidden layers and LeakyReLU nonlinearities (negative slope 0.2).
Revised: In all experiments with the Cahn-Hilliard equation we represent the dynamics function F θ by an MLP with 2 hidden layers and LeakyReLU nonlinearities (negative slope 0.2).

Original: The number of neurons in each layer was set to the number of nodes in the spatial grid on which the model was trained.
Revised: The number of neurons in each layer was set to the number of nodes in the spatial grid on which the model was trained.

Original: The predictions u ( t ) were obtained by simulating the system forward in time using adaptive Heun solver from torchdiffeq package [6] with rtol and atol set to 1 . 

Revised: The predictions u ( t ) were obtained by simulating the system forward in time using adaptive Heun solver from torchdiffeq package [6] with rtol and atol set to 1 .

Original: e- 5 and 1 . 

Revised: e- 5 and 1 .

Original: e-
Revised: e-

Original: 5 respectively.
Revised: 5 respectively.

Original: All models were trained for 1500 epochs using Rprop optimizer [29] with learning rate set to 1 . 

Revised: All models were trained for 1500 epochs using Rprop optimizer [29] with learning rate set to 1 .

Original: · 10 − 6 and batch size set to the number of simulations in the training set.
Revised: · 10 − 6 and batch size set to the number of simulations in the training set.

Original: Mean squared error was used as the loss function.
Revised: Mean squared error was used as the loss function.

Original: Spatial and temporal grids in the testing data were the same as in the training data.
Revised: Spatial and temporal grids in the testing data were the same as in the training data.

Original: We set λ = 2 .
Revised: We set λ = 2 .

Original: C. H EAT EQUATION WITH A MONOTONICITY CONSTRAINT
Revised: C. H EAT EQUATION WITH A MONOTONICITY CONSTRAINT

Original: In all experiments with the heat equation we represent the dynamics function F θ by a GNN with the messaging and aggregation networks being MLPs with a single hidden layer consisting of 60 neurons with Tanh nonlinearities and the input/output sizes of 4 / 40 and 41 / 1 respectively.
Revised: In all experiments with the heat equation we represent the dynamics function F θ by a GNN with the messaging and aggregation networks being MLPs with a single hidden layer consisting of 60 neurons with Tanh nonlinearities and the input/output sizes of 4 / 40 and 41 / 1 respectively.

Original: The predictions u ( t ) were obtained by simulating the system forward in time using adaptive Heun solver from torchdiffeq package [6] with rtol and atol set to 1 . 

Revised: The predictions u ( t ) were obtained by simulating the system forward in time using adaptive Heun solver from torchdiffeq package [6] with rtol and atol set to 1 .

Original: e- 5 and 1 . 

Revised: e- 5 and 1 .

Original: e-
Revised: e-

Original: 5 respectively.
Revised: 5 respectively.

Original: All models were trained for 750 epochs using Rprop optimizer [29] with learning rate set to 1 . 

Revised: All models were trained for 750 epochs using Rprop optimizer [29] with learning rate set to 1 .

Original: · 10 − 6 and batch size set to the number of simulations in the training set.
Revised: · 10 − 6 and batch size set to the number of simulations in the training set.

Original: Mean squared error was used as the loss function.
Revised: Mean squared error was used as the loss function.

Original: Spatial and temporal grids in the testing data were the same as in the training data.
Revised: Spatial and temporal grids in the testing data were the same as in the training data.

Original: We set λ = 0 .
Revised: We set λ = 0 .

Original: C.
Revised: C.

Original: L EARNING DISTRIBUTIONS OF PHYSICAL FIELDS
Revised: L EARNING DISTRIBUTIONS OF PHYSICAL FIELDS

Original: In both cases we used identical architectures and training process for the constrained and unconstrained models.
Revised: In both cases we used identical architectures and training process for the constrained and unconstrained models.

Original: Both models were trained for 1 . 2 M iterations using the same random seed.
Revised: Both models were trained for 1 . 2 M iterations using the same random seed.

Original: Constraints in the constrained model were enabled only after 600 k iterations.
Revised: Constraints in the constrained model were enabled only after 600 k iterations.

Original: The base distribution was set to a 128 -dimensional isotropic standard normal.
Revised: The base distribution was set to a 128 -dimensional isotropic standard normal.

Original: Models were trained using RMSProp optimizer [35] with batch size and learning rate set to 64 and 0 .
Revised: Models were trained using RMSProp optimizer [35] with batch size and learning rate set to 64 and 0 .

Original: 00001 respectively.
Revised: 00001 respectively.

Original: The discriminator’s weights were clipped to [ − 0 . 01 , 0 . 01] .
Revised: The discriminator’s weights were clipped to [ − 0 . 01 , 0 . 01] .

Original: C.4. Z ERO - DIVERGENCE FIELDS
Revised: C.4. Z ERO - DIVERGENCE FIELDS

Original: We used MLPs as a discriminator and generator.
Revised: We used MLPs as a discriminator and generator.

Original: The discriminator consisted of 3 hidden layers of sizes 1024 - 512 - 256 with LeakyReLU nonlinearities (negative slope 0.2) and input/output size of 2010 and 1 respectively.
Revised: The discriminator consisted of 3 hidden layers of sizes 1024 - 512 - 256 with LeakyReLU nonlinearities (negative slope 0.2) and input/output size of 2010 and 1 respectively.

Original: The generator consisted of 3 hidden layers of sizes 256 - 512 - 1024 with
Revised: The generator consisted of 3 hidden layers of sizes 256 - 512 - 1024 with

Original: LeakyReLU nonlinearities (negative slope 0.2), input/output size of 128 and 2010 respectively, and a ﬁnal hyperbolic tangent nonlinearity applied to the output.
Revised: LeakyReLU nonlinearities (negative slope 0.2), input/output size of 128 and 2010 respectively, and a ﬁnal hyperbolic tangent nonlinearity applied to the output.

Original: We set λ = 0 .
Revised: We set λ = 0 .

Original: C.4. Z ERO -L APLACIAN FIELDS
Revised: C.4. Z ERO -L APLACIAN FIELDS

Original: We used MLPs as a discriminator and generator.
Revised: We used MLPs as a discriminator and generator.

Original: The discriminator consisted of 3 hidden layers of sizes 1024 - 512 - 256 with LeakyReLU nonlinearities (negative slope 0.2) and input/output size of 1086 and 1 respectively.
Revised: The discriminator consisted of 3 hidden layers of sizes 1024 - 512 - 256 with LeakyReLU nonlinearities (negative slope 0.2) and input/output size of 1086 and 1 respectively.

Original: The generator consisted of 3 hidden layers of sizes 256 - 512 - 1024 with
Revised: The generator consisted of 3 hidden layers of sizes 256 - 512 - 1024 with

Original: LeakyReLU nonlinearities (negative slope 0.2), input/output size of 128 and 1086 respectively, and sigmoid function applied to the output.
Revised: LeakyReLU nonlinearities (negative slope 0.2), input/output size of 128 and 1086 respectively, and sigmoid function applied to the output.

Original: We set λ = 0 . 0075 .
Revised: We set λ = 0 . 0075 .

Original: D C AHN -H ILLIARD EQUATION WITH HARD INTEGRAL CONSTRAINTS
Revised: D C AHN -H ILLIARD EQUATION WITH HARD INTEGRAL CONSTRAINTS

Original: Here we demonstrate how the approach to enforcing hard constraints described in Section 3.3 can be used to enforce integral constraints on a nonuniform grid.
Revised: Here we demonstrate how the approach to enforcing hard constraints described in Section 3.3 can be used to enforce integral constraints on a nonuniform grid.

Original: We use the same setup as in Section 4.2 with 30 training simulations and 50 % of nodes in the grid.
Revised: We use the same setup as in Section 4.2 with 30 training simulations and 50 % of nodes in the grid.

Original: We compare three models: unconstrained model, model with soft constraint and model with hard constraint.
Revised: We compare three models: unconstrained model, model with soft constraint and model with hard constraint.

Original: We use the PWL basis during training and testing.
Revised: We use the PWL basis during training and testing.

Original: Table 1 shows that relative errors of all three models are practically similar but, as Figuredemonstrates, constraint violations differ signiﬁcantly.
Revised: Table 1 shows that relative errors of all three models are practically similar but, as Figuredemonstrates, constraint violations differ signiﬁcantly.

Original: We see that constraint violations of the model with hard constraint are zero at all time points as expected.
Revised: We see that constraint violations of the model with hard constraint are zero at all time points as expected.

Original: Being able to produce predictions that satisfy some constraints exactly might be very useful for some applications, however, as we mention in Section 3.3, currently this approach to enforcing hard constraints is limited to systems with a relatively small number of nodes and is signiﬁcantly slower than models with soft constraints.
Revised: Being able to produce predictions that satisfy some constraints exactly might be very useful for some applications, however, as we mention in Section 3.3, currently this approach to enforcing hard constraints is limited to systems with a relatively small number of nodes and is signiﬁcantly slower than models with soft constraints.

Original: We report training times in Table 1.
Revised: We report training times in Table 1.

Original: [Table]
Revised: [Table]

Original: E L EARNING C AHN -H ILLIARD EQUATION WITH GNN S
Revised: E L EARNING C AHN -H ILLIARD EQUATION WITH GNN S

Original: We use the same setup as in Section 4.2 with 75 % of nodes in the grid but trained the models forepochs.
Revised: We use the same setup as in Section 4.2 with 75 % of nodes in the grid but trained the models forepochs.

Original: Instead of an MLP we use a GNN with messaging and aggregation networks being MLPs with two hidden layers of size 64 and LeakyReLU nonlinearities (negative slope 0.2).
Revised: Instead of an MLP we use a GNN with messaging and aggregation networks being MLPs with two hidden layers of size 64 and LeakyReLU nonlinearities (negative slope 0.2).

Original: For each node, the GNN evaluates the output as
Revised: For each node, the GNN evaluates the output as

Original: [Equation]
Revised: [Equation]

Original: where u proj i and u proj j are linear projections of the state at nodes i and j and x proj ij is a linear projection of the pair consisting of the distance between nodes i and j and a unit vector pointing from node i to node j .
Revised: where u proj i and u proj j are linear projections of the state at nodes i and j and x proj ij is a linear projection of the pair consisting of the distance between nodes i and j and a unit vector pointing from node i to node j .

Original: All projections have dimension 16 .
Revised: All projections have dimension 16 .

Original: We compare constrained and unconstrained models.
Revised: We compare constrained and unconstrained models.

Original: We use the PWL, PWQ and cubic RBF bases.
Revised: We use the PWL, PWQ and cubic RBF bases.

Original: Results of the experiment are shown in Figure 13.
Revised: Results of the experiment are shown in Figure 13.

Original: The ﬁgure shows that relative errors and constraint violations of all models are signiﬁcantly higher than for MLP-based models.
Revised: The ﬁgure shows that relative errors and constraint violations of all models are signiﬁcantly higher than for MLP-based models.

Original: [Figure]
Revised: [Figure]