Original: On the Relationships between Graph Neural Networks for the Simulation of Physical Systems and Classical Numerical Methods
Revised: On the Relationships between Graph Neural Networks for the Simulation of Physical Systems and Classical Numerical Methods

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: Recent developments in Machine Learning approaches for modelling physical systems have begun to mirror the past development of numerical methods in the computational sciences.
Revised: Recent developments in Machine Learning approaches for modelling physical systems have begun to mirror the past development of numerical methods in the computational sciences.

Original: In this survey we begin by providing an example of thisdevelopment with the parallels between  graph neural network acceleration for physical simulations andthe development of particle-based approaches.
Revised: In this survey we begin by providing an example of this with the parallels between the development trajectories of graph neural network acceleration for physical simulations and particle-based approaches.

Original: We then give an overview of simulation approaches, which have not yet found their way into state-of-the-art Machine Learning methods and hold the potential to  Machine Learning approaches more accurate and more efficient.
Revised: We then give an overview of simulation approaches, which have not yet found their way into state-ofthe-art Machine Learning methods and hold the potential to make Machine Learning approaches more accurate and more efficient.

Original: We conclude by presenting an outlook on the potential of these approaches for making Machine Learning models for science more efficient.
Revised: We conclude by presenting an outlook on the potential of these approaches for making Machine Learning models for science more efficient.

Original: 1. Introduction
Revised: 1. Introduction

Original: Recent years have seen an ever-larger push towards the application of Machine Learning to problems from the physical sciences such as Molecular Dynamics (Musaelian et al., 2022b), coarse-graining
Revised: Recent years have seen an ever-larger push towards the application of Machine Learning to problems from the physical sciences such as Molecular Dynamics (Musaelian et al., 2022b), coarse-graining

Original: (Wang et al., 2022), the timeevolution of incompressible fluid flows (Wang et al., 2020), learning governing equations from data (Cranmer et al., 2020), large-scale GPT-class models for chemistry (Frey et al., 2022), and the acceleration of numerical simulations with machine learning techniques (Kochkov et al., 2021).
Revised: (Wang et al., 2022), the time-evolution of incompressible fluid flows (Wang et al., 2020), learning governing equations from data (Brunton et al., 2016; Cranmer et al., 2020), large-scale transformer models for chemistry (Frey et al., 2022), and the acceleration of numerical simulations with machine learning techniques (Kochkov et al., 2021).

Original: All of these algorithms build on the infrastructure underpinning modern Machine Learning in combing state-of-the-art approaches with a deep understanding of the physical problems at hand.
Revised: All of these algorithms build on the infrastructure underpinning modern Machine Learning in combing state-of-the-art approaches with a deep understanding of the physical problems at hand.

Original: This begs the questions if there exist more insights and tricks hidden in existing, classical approaches in the physical sciences which have the potential to maybe not only make the algorithm for the particular problem class more efficient, but maybe even Machine Learning in gen- eral?
Revised: This begs the questions if there exist more insights and tricks hidden in existing, classical approaches in the physical sciences which have the potential to maybe not only make the algorithm for the particular problem class more efficient, but maybe even Machine Learning in general?

Original: Inspired by recent theoretical advances in the algorithmic alignment between graph neural networks (GNNs), and dynamical programming (Xu et al., 2020; Veli ˇ ckovi ´ c et al., 2020), we surmise that the extension of this analysis to classical PDE solvers, and the physical considerations they incorporate enables us to learn from the development trajectory in the physical sciences, and inform the development of new algorithms.
Revised: Inspired by recent theoretical advances in the algorithmic alignment between Graph Neural Networks (GNNs) and dynamic programming (Xu et al., 2020; Veli ˇ ckovi ´ c et al., 2020), we surmise that the extension of this analysis to classical PDE solvers, and the physical considerations they incorporate, enables us to learn from the development trajectory in the physical sciences to inform the development of new algorithms.

Original: In this workshop paper we make the following contributions towards this goal:
Revised: In this workshop paper we make the following contributions towards this goal:

Original: • A comparison of the development of graph-based learned solvers, and the proximity of their development ideas to the development ofMolecular Dynamics, to Smoothed Particle Hydrodynamics  in the physical sciences.
Revised: • A comparison of the development of graph-based learned solvers, and the proximity of their development ideas to the development of Smoothed Particle Hydrodynamics starting from Molecular Dynamics in the physical sciences.

Original: • An analysis of classical numerical solvers, and their algorithmic features to inform new ideas for new algorithms.
Revised: • An analysis of classical numerical solvers, and their algorithmic features to inform new ideas for new algorithms.

Original: 2. MeshGraphNets and its relation to classical methods
Revised: 2. MeshGraphNets and its relation to classical methods

Original: An excellent example of the parallels between the development of Machine Learning methods for the sciences, and the development of classical approaches is the recent development of graph-based simulators.
Revised: An excellent example of the parallels between the development of Machine Learning methods for the sciences and the development of classical approaches is the recent development of graph-based simulators.

Original: When we relate their inherent assumptions, and techniques to the development of particle-based methods, starting with Molecular Dynamics, a great many parallels arise.
Revised: When we relate their inherent assumptions and techniques to the development of particle-based methods, starting with Molecular Dynamics, a great many parallels arise.

Original: For an impression of the scales the classical methods operate on, and where graph-based simulators are placed in relation, please refer to Figure 1.
Revised: For an impression of the scales the classical methods operate on, and where graph-based simulators are placed in relation, please refer to Figure 1.

Original: 2.1. The Graph Network Building Block
Revised: In this section, we analyze the structure of two of the first mature learned solvers (GNS (Sanchez-Gonzalez et al., 2020), MeshGraphNets (Pfaff et al., 2021)) and how these two approaches align with three of the classical methods (MD, FPM, SPH). We select these learned algorithms because they were one of the first of their kind to show promising results on real world data.

Revised: Also, GNS is trained directly on SPH data which further motivates an algorithmic comparison.

Revised: 2.1. Graph Neural Network-based Approaches to Simulation

Original: Adopting the graph formulation of (Battaglia et al., 2018), the main  choicesin graph networks are the choice of updatefunction, and aggregation-function.
Revised: The Graph Network (GN) (Battaglia et al., 2018) is a framework that generalizes graph-based learning and specifically the Graph Neural Network (GNN) architecture by Scarselli et al. However, in this work, we use the terms GN and GNN interchangeably. Adopting the Graph Network formulation, the main design choices are the choice of update-function, and aggregation-function.

Original: For physics-informed modeling this affords us the ability to blur the line between classical methods, and graph-based methods by including biases such as CNNs for non-regular grids, as well as encoding physical laws into our network structure with the help of spatial equivariance/invariance, local interactions, the superposition principe, and differential equations.
Revised: For physicsinformed modeling this gives us the ability to blur the line between classical methods and graph-based methods by including biases similar to CNNs for non-regular grids, as well as encoding physical laws into our network structure with the help of spatial equivariance/invariance, local interactions, the superposition principle, and differential equations.

Original: E.g. translational equivariance can easily be incorporated using relative positions between neighbouring nodes, or the superposition principle can be encoded in graphs by using the summation aggregation over the representation of forces as edge features.
Revised: E.g. translational equivariance can easily be incorporated using relative positions between neighboring nodes, or the superposition principle can be encoded in graphs by using the summation aggregation over the representation of forces as edge features.

Original: 2.2. MeshGraphNets and their Predecessor

Original: Viewing MeshGraphNets (Pfaff et al., 2021) from a physicsmotivated perspective, we argue that MeshGraphNets originate from Molecular Dynamics.
Revised: Viewing MeshGraphNets (Pfaff et al., 2021) from a physicsmotivated perspective, we argue that MeshGraphNets originate from Molecular Dynamics.

Original: To present this argument in all its clarity, we have to begin with its predecessor; the Graph Network-based Simulators (GNS) (SanchezGonzalez et al., 2020).
Revised: To present this argument in all its clarity, we have to begin with its predecessor: the Graph Network-based Simulators (GNS) (SanchezGonzalez et al., 2020).

Original: 2.2.1. G RAPH N ETWORK - BASED S IMULATORS
Revised: 2.1.1. G RAPH N ETWORK - BASED S IMULATORS

Original: The graph networks-based simulator builds on the encoderprocessor-decoder approach, where graph networks are applied iteratively on the encoded space.
Revised: The Graph Networks-based Simulator builds on the encoderprocessor-decoder approach, where Graph Networks are applied iteratively on the encoded space.

Original: Proving GNS’ ability to simulate systems with up to 85k particles, their approach can be summarized as follows
Revised: Proving GNS’ ability to simulate systems with up to 85k particles, their approach can be summarized as follows.

Revised: Let X t denote the states of a particle system at time t .

Revised: X might contain the position, velocity, type of particle, or any other physical information specific to a material particle.

Revised: A set of k + 1 subsequent past states

Original: [Equation]
Revised: [Equation]

Original:  The core task isthen to  learn the differential operator d θ , which approximates the dynamicsd θ : X t k −→ Y t k ,
Revised: if given to the network. The core task is to then learn the differential operator d θ , which approximates the dynamics

Original: [Equation]
Revised: [Equation]

Revised: Here, Y t is the acceleration, which is used to obtain the next state X t +1 via integration using a deterministic ”Update” routine, e.g. semi-implicit Euler scheme.

Original: This differential operator  is learned with the encoderprocessor-decoder approach where the encoder takes into 10 previous states, and encodes them into a graph.
Revised: The differential operator d θ is learned with the encoder-processor-decoder approach where the encoder takes in 1 to 10 previous states, and encodes them into a graph.

Revised: This graph consists of nodes - latent representation of the states X - and edges - between each pair of particles closer than some cut-off radius there is another latent vector, which initially contains the distance or displacement information.

Original: The processor is then a multilayer graph network of which the[Figure] exact number of message-passing graph networks is a hyperparameter.
Revised: The processor is then a multilayer Graph Network of which the exact number of message-passing Graph Networks is a hyperparameter.

Original: The resulting is then decoded back to physical space.
Revised: The result on the graph-space is then decoded back to physical space.

Original: The loss isthen computed as the mean-squared error between the learned acceleration, and the target acceleration.
Revised: The loss is computed as the mean-squared error between the learned acceleration, and the target acceleration.

Original: While the approach showed promising results for fluid simulations, and fluid-solid interactions, it struggled on deforming meshes, such as thin shells.
Revised: While the approach showed promising results for fluid simulations, and fluid-solid interactions, it struggled on deforming meshes, such as thin shells.

Original: 2.2.2. M ESH G RAPH N ETS
Revised: 2.1.2. M ESH G RAPH N ETS

Original: To better represent meshesthe MeshGraphNets (Pfaff et al., 2021) supplemented the graph network simulation with an additional set of edges to define a mesh on which interactions can be learned.
Revised: To better represent meshes MeshGraphNets (Pfaff et al., 2021) supplemented the Graph Network simulation with an additional set of edges to define a mesh, on which interactions can be learned.

Original: Closely related to the superposition principle in physics, i.e. splitting a complicated function into the sum of multiple simpler ones, the interaction function is split into the interaction of mesh-type edges, and collision-type edges.
Revised: Closely related to the superposition principle in physics, the principle of splitting a complicated function into the sum of multiple simpler ones, the interaction function is split into the interaction of mesh-type edges and collision-type edges.

Original: Following the widespread use of remeshing in engineering, MeshGraphNets has included the ability for adaptive remeshing which allows it to model a wider spectrum of dynamics.
Revised: Following the widespread use of remeshing in engineering, MeshGraphNets have the ability to adaptively remesh to model a wider spectrum of dynamics.

Original: Mesh deformation without adaptive remeshing leads to the loss of high frequency information 2.
Revised: Mesh deformation without adaptive remeshing would lead to the loss of high frequency information.

Original: The  output vector can be extended with additional components to also predict further targets, such as the stress field
Revised: The last major improvement of MeshGraphNets over GNS is extending the output vector Y with additional components to also predict further targets, such as the stress field. In difference to the Graph Network-based Simulators, the input here includes a predefined mesh and the output is extended to contain dynamical features like pressure. Update

Original: In difference to the Graph Network-based Simulators the input
Revised: Learned one-step simulator Iterative rollout Encoder Processor

Original: here includes a predefined mesh
Revised: Message passing x L

Original: and
Revised: Decoder World-space edges Mesh-space edges Cloth mesh nodes World-space messages Mesh-space messages

Original: the{+Decoded accelerations E M E W V e 0 Mij e 
+}
Revised: {+Decoded accelerations E M E W V e 0 Mij e 
+}

Original: output is extended to contain dynamical features like pressure.
Revised: Wij q ti q t +1 i p i p i M t

Original: From Molecular Dynamics to Graph Network-Based Simulators
Revised: M t + M t + Obstacle mesh nodes G 2.2. Similarities between the Development Trajectories of Particle-based Methods and Graph Neural Network-based Approaches to Simulations

Original: Beginning with  the original particle-based method,Molecular Dynamics, we will now outline the similarities between the development  of MeshGraphNets, and the development of particle-based methods in physics.
Revised: Beginning with Molecular Dynamics, the earliest and most fundamental particle-based method, we will now outline the similarities between the development trajectories, and the derivations inherent to them, of MeshGraphNets and the development of particle-based methods in physics.

Original: 2.3.1. M OLECULAR D YNAMICS
Revised: 2.2.1. S IMILARITIES TO M OLECULAR D YNAMICS

Original: Molecular Dynamics is a widely used simulation method which generates the trajectories of an N-body atomic system.
Revised: Molecular Dynamics is a widely used simulation method which generates the trajectories of an N-body atomic system.

Original: For the sake of intellectual clarity we restrict ourselves to its simplest form, the unconstrained Hamiltonian mechanics description.
Revised: For the sake of intellectual clarity we restrict ourselves to its simplest form, the unconstrained Hamiltonian mechanics description.

Original: The construction of connections, and edges is one of the clearest similarities between Molecular Dynamics and
Revised: The construction of connections, and edges is one of the clearest similarities between Molecular Dynamics and

Original: MeshGraphNets.
Revised: MeshGraphNets.

Original: Both can potentially have a mesh as an input, and both compute the interactions based on spatial distances up to a fixed threshold.
Revised: Both can potentially have a mesh as an input, and both compute the interactions based on spatial distances up to a fixed threshold.

Original: Iterative updates, or the repeated application of graph network layers in the MeshGraphNets, extend the effective interaction radius beyond the immediate neighbourhood of a particle such that all particles can be interacted with.
Revised: Iterative updates, or the repeated application of Graph Network layers in the MeshGraphNets, extend the effective interaction radius beyond the immediate neighbourhood of a particle such that all particles can be interacted with.

Original: Both approaches are at the same time translationally invariant w.r.t.
Revised: Both approaches are at the same time translationally invariant w.r.t.

Original: accelerations, and permutation equivariant w.r.t.
Revised: accelerations, and permutation equivariant w.r.t.

Original: the particles, and use a symlectic time-integrator.
Revised: the particles, and use a symplectic time-integrator.

Original: While there are theoretical reasons for this choice in Molecular Dynamics, it is choice of convenience in the context of learned approaches.
Revised: While there are theoretical reasons for this choice in Molecular Dynamics, it is choice of convenience in the context of learned approaches.

Original: The main difference between the two approaches lies in the computation of the accelerations.
Revised: The main difference between the two approaches lies in the computation of the accelerations.

Original: In Molecular Dynamics the derivative of a predefined potential function is evaluated, whereas a learned model is used in the Graph Network-based Simulators.
Revised: In Molecular Dynamics the derivative of a predefined potential function is evaluated, whereas a learned model is used in the Graph Network-based Simulators.

Original: 2.3.2. S MOOTHED P ARTICLE H YDRODYNAMICS
Revised: 2.2.2. S IMILARITIES TO S MOOTHED P ARTICLE H YDRODYNAMICS

Original: A closer relative to the Graph Network-based Simulators is the Smoothed Particle Hydrodynamics algorithm  (Lucy, 1977; Gingold & Monaghan, 1977) originating from astrophysics.
Revised: A closer relative to the Graph Network-based Simulators is the Smoothed Particle Hydrodynamics algorithm originating from astrophysics (Lucy, 1977; Gingold & Monaghan, 1977).

Original: Smoothed Particle Hydrodynamics discretizes the governing equations of fluid dynamics, the Navier-Stokes equations, with kernels such that the discrete particles follow
Revised: Smoothed Particle Hydrodynamics discretizes the governing equations of fluid dynamics, the Navier-Stokes equations, with kernels such that the discrete particles follow

Original: Newtonian mechanics with the equivalent of a prescribed molecular potential.
Revised: Newtonian mechanics with the equivalent of a prescribed molecular potential.

Original: Both, Smoothed Particle Hydrodynamics, and Graph Network-based Simulators obey the continuum assumption, whereas Molecular Dynamics presumes a discrete particle distribution, and is constrained to extremely short time intervals.
Revised: Both, Smoothed Particle Hydrodynamics, and Graph Network-based Simulators obey the continuum assumption, whereas Molecular Dynamics presumes a discrete particle distribution, and is constrained to extremely short time intervals.

Original: 2.3.3. T HE D IFFERENCES
Revised: 2.2.3. T HE D IFFERENCES

Original: Summarizing the key differences between the closely related approaches, Molecular Dynamics and Smoothed Particle Hydrodynamics both take the velocity as an input, whereas Graph-based approaches require a history of the coordinates to approximate the same velocity.
Revised: Summarizing the key differences between the closely related approaches, Molecular Dynamics and Smoothed Particle Hydrodynamics both take one past state X t as an input, whereas Graph-based approaches require a history of k states X t 0: K .

Original: Molecular Dynamics encodes geometric relations in the potential, MeshGraphNets encode the geometry in the mesh, while there exists no direct way for inclusion in the other two approaches.
Revised: Molecular Dynamics encodes geometric relations in the potential, MeshGraphNets encode the geometry in the mesh, while there exists no direct way for inclusion in the other two approaches.

Original: Molecular Dynamics, and Smoothed Particle Hydrodynamics explicitly encode physical laws, for learned methods all these parameters and relations have to be learned from data.
Revised: Molecular Dynamics, and Smoothed Particle Hydrodynamics explicitly encode physical laws, for learned methods all these parameters and relations have to be learned from data.

Original: 2.4. From Graph Network-based Simulators to MeshGraphNets

Original: A key advancement of MeshGraphNets, coming from the
Revised: A key advancement of MeshGraphNets, coming from the

Original: Graph Network-based Simulators, is the explicit superimposition of solutions on both sets of edges, which far outperforms the implicit distinction of interactions.
Revised: Graph Network-based Simulators, is the explicit superimposition of solutions on both sets of edges, which far outperforms the implicit distinction of interactions.

Original: This approach is equally applicable to all conventional particle-, and mesh-based simulations in engineering.
Revised: This approach is equally applicable to all conventional particle-, and mesh-based simulations in engineering.

Original: Borrowing the
Revised: Borrowing the

Original: Fluid Particle Model from fluid mechanics, we can then connect the classical methods with the learned approaches by viewing meshes and particles as the same thing under the fluid-particle paradigm.
Revised: Fluid Particle Model from fluid mechanics, we can subsequently connect the classical methods with the learned approaches by viewing meshes and particles as the same entity under the fluid-particle paradigm.

Original: 2.4.1. T HE F LUID P ARTICLE M ODEL
Revised: 2.2.4. C ONNECTING M ESH G RAPH N ETS TO G RAPH N EURAL N ETWORK - BASED S IMULATIONS WITH THE F LUID P ARTICLE M ODEL

Original: The Fluid Particle Model (Espanol, 1998) is a mesoscopic
Revised: The Fluid Particle Model (Espanol, 1998) is a mesoscopic

Original: Newtonian model, as seen in fig 1, situated on an intermediate scale between the microscopic Molecular Dynamics, and the macroscopic Smoothed Particle Hydrodynamics.
Revised: Newtonian model, as seen in Figure 1, situated on an intermediate scale between the microscopic Molecular Dynamics and the macroscopic Smoothed Particle Hydrodynamics.

Original: It views particles from the point of view of a Voronoi tesselation of the molecular fluid, see Figure 3.
Revised: It views particles from the point of view of a Voronoi tesselation of the molecular fluid, see Figure 3.

Original: The Voronoi tesselation coarse-grains the atomistic system to a pseudoparticle system with ensembles of atoms in thermal equilibrium summarized as pseudoparticles.
Revised: The Voronoi tesselation coarse-grains the atomistic system to a pseudoparticle system with ensembles of atoms in thermal equilibrium summarized as pseudoparticles.

Original: This pseudoparticle construction is closely related to the MeshGraphNets construction, where each mesh node also corresponds to the cell center of a simulated pseudoparticle.
Revised: This pseudoparticle construction is closely related to the MeshGraphNets construction, where each mesh node also corresponds to the cell center of a simulated pseudoparticle.

Original: With Smoothed
Revised: Smoothed

Original: Particle Hydrodynamics, as well as Dissipative Particle Dynamics (Hoogerbrugge & Koelman, 1992) also operating on pseudoparticles.
Revised: Particle Hydrodynamics as well as Dissipative Particle Dynamics (Hoogerbrugge & Koelman, 1992) also both operate on pseudoparticles.

Original: All of these approaches share that they have to presume a large enough quantity of  pseudoparticles to be viewed as a thermodynamic system.
Revised: All of these approaches share that they have to presume a large enough number of atoms per pseudoparticles to be viewed as a thermodynamic system.

Original: Especially in Dissipative Particle Dynamics one injects
Revised: Especially in Dissipative Particle Dynamics one injects

Original: Gaussian noise to approximate a physical system, just as is done for Graph Network-based Simulators, and MeshGraphNets to stabilize the training.
Revised: Gaussian noise to approximate a physical system, just as is done for Graph Network-based Simulators and MeshGraphNets to stabilize the training.

Original: We surmise that this injection of noise into graph-based simulators amounts to forcing the learned model to predict the true output despite the noisy inputs, hence leading the model to converge to the central limit of the estimated conditional distribution of the acceleration.
Revised: We surmise that this injection of noise into graph-based simulators amounts to forcing the learned model to predict the true output despite the noisy inputs, hence leading the model to converge to the central limit of the estimated conditional distribution of the acceleration.

Original: The construction of Voronoi tesselations governs that the size of the cells is to be inversely proportional to variations in their properties, hence leading to more sampling in regions with high property variation.
Revised: The construction of Voronoi tesselations governs that the size of the cells is to be inversely proportional to variations in their properties, hence leading to more sampling in regions with high property variation.

Original: The very same argument based on the curvature as a heuristic is being used to derive the mesh refinement of the MeshGraphNets algorithm.
Revised: The very same argument based on the curvature as a heuristic is being used to derive the mesh refinement of the MeshGraphNets algorithm.

Original: 3. Relation to Numerical Schemes
Revised: 3. Relation to Numerical Schemes

Original: After the recent success of Neural ODEs solvers (Chen et al., 2018), it has taken almost four years to start considering Neural PDEs in general (Brandstetter et al., 2022).
Revised: After the recent success of Neural ODEs solvers (Chen et al., 2018), it has taken almost four years to start considering Neural PDEs in general (Brandstetter et al., 2022).

Original: By definition, PDEs deal with derivatives of multiple variables, compared to ODEs having one variable.
Revised: By definition, PDEs deal with derivatives of multiple variables, compared to ODEs having one variable.

Original: As a result, typical numerical approximations of PDEs are much more diverse depending on the peculiarities of the PDE of interest.
Revised: As a result, typical numerical approximations of PDEs are much more diverse depending on the peculiarities of the PDE of interest.

Original: Typical PDE solvers operating on grids (Eulerian description) include Finite Difference Methods (FDM), Finite Volume Methods (FVM), and Finite Element Methods (FEM), whereas other methods follow the trajectory of irregularly spaced points (Lagrangian description) like Smoothed Particle Hydrodynamics (SPH), Fluid Particle Model (FPM),
Revised: Typical PDE solvers operating on grids (Eulerian description) include Finite Difference Methods (FDM), Finite Volume Methods (FVM), and Finite Element Methods (FEM), whereas other methods follow the trajectory of irregularly spaced points (Lagrangian description) like Smoothed Particle Hydrodynamics (SPH), Fluid Particle Model (FPM),

Original: Dissipative Particle Dynamics (DPD) (Hoogerbrugge &
Revised: Dissipative Particle Dynamics (DPD) (Hoogerbrugge &

Original: Koelman, 1992), Volume of Fluid Method (VOF) (Hirt &
Revised: Koelman, 1992), Volume of Fluid Method (VOF) (Hirt &

Original: Nichols, 1981), Particle-in-Cell (PIC) (Brackbill & Ruppel, 1986), Material Point Method (MPM) (Sulsky et al., 1993), Discrete Element Method (DEM) (Cundall & Strack, 1979), and Meshless FEM (MFEM).
Revised: Nichols, 1981), Particle-in-Cell (PIC) (Brackbill & Ruppel, 1986), Material Point Method (MPM) (Sulsky et al., 1993), Discrete Element Method (DEM) (Cundall & Strack, 1979), and Meshless FEM (MFEM).

Original: Finally, there are also approaches to solving PDEs without any discretization as in Sawhney et al.
Revised: Finally, there are also approaches to solving PDEs without any discretization as in Sawhney et al.

Original: Each of these methods works best for a specific type of PDE, boundary/initial conditions, and parameter range.
Revised: Each of these methods works best for a specific type of PDE, boundary/initial conditions, and parameter range.

Original: Here, we compare concepts from these classical methods to state-of-the-art learned algorithms.
Revised: In this section we compare concepts from these classical methods to state-of-the-art learned algorithms.

Original: 3.1. Data augmentation with white noise
Revised: 3.1. Data augmentation with white noise

Original: Two popular papers corrupting training inputs with additive Gaussian noise include Sanchez-Gonzalez et al.
Revised: Two popular papers corrupting training inputs with additive Gaussian noise include Sanchez-Gonzalez et al.

Original: Pfaff et al.
Revised: Pfaff et al.

Original: (2021), as described before.
Revised: (2021), as described before.

Original: The goal of this approach is to force the model to deal with accumulating noise leading to a distribution shift during longer rollouts.
Revised: The goal of this approach is to force the model to deal with accumulating noise leading to a distribution shift during longer rollouts.

Original: Thus, the noise acts as an effective regularization technique, which in these two papers allows for much longer trajectories than seen during training.
Revised: Thus, the noise acts as an effective regularization technique, which in these two papers allows for much longer trajectories than seen during training.

Original: However, one major issue with this approach is that the scale of the noise is represented by two new hyperparameters, which have to be tuned manually (Pfaff et al.
Revised: However, one major issue with this approach is that the scale of the noise is represented by two new hyperparameters, which have to be tuned manually (Pfaff et al.

Original: (2021), Appendix 2.2).
Revised: (2021), Appendix 2.2).

Original: A perspective on noise injection coming from the physical sciences is to see it through the lens of mesoscopic particle methods like the Fluid Particle Model and Dissipative
Revised: A perspective on noise injection coming from the physical sciences is to see it through the lens of mesoscopic particle methods like the Fluid Particle Model and Dissipative

Original: Particle Dynamics, in which the noise originates from the
Revised: Particle Dynamics, in which the noise originates from the

Original: Brownian motion at small scales.
Revised: Brownian motion at small scales.

Original: Although GNS and MeshGraphNets operate on scales too large for the relevance of
Revised: Although GNS and MeshGraphNets operate on scales too large for the relevance of

Original: Brownian motion, the Fluid Particle Model provides a principled way of relating particle size and noise scale.
Revised: Brownian motion, the Fluid Particle Model provides a principled way of relating particle size and noise scale.

Original: The underlying considerations from statistical mechanics might aid to a better understanding of the influence of training noise and in turn make approaches based on it more efficient.
Revised: The underlying considerations from statistical mechanics might aid to a better understanding of the influence of training noise and in turn make approaches based on it more efficient.

Original: 3.2. Data augmentation by multi-step loss
Revised: 3.2. Data augmentation by multi-step loss

Original: Another way of dealing with the distribution shift is by training a model to correct its own mistakes via some form of a multi-step loss, i.e. during training a short trajectory is generated and the loss is summed over one or multiple past steps (Tompson et al., 2017;
Revised: Another way of dealing with the distribution shift is by training a model to correct its own mistakes via some form of a multi-step loss, i.e. during training a short trajectory is generated and the loss is summed over one or multiple past steps (Tompson et al., 2017;

Original: Um et al., 2020; Ummenhofer et al., 2020; Brandstetter et al., 2022).
Revised: Um et al., 2020; Ummenhofer et al., 2020; Brandstetter et al., 2022).

Original: The results on this vary with some researchers reporting better performance than with noise injection (Brandstetter et al., 2022), while others report the opposite experience (Sanchez-Gonzalez et al., 2020).
Revised: The results on this vary with some researchers reporting better performance than with noise injection (Brandstetter et al., 2022), while others report the opposite experience (Sanchez-Gonzalez et al., 2020).

Original: Looking at classical solvers for something related to the multi-step loss, it is natural to think of adaptive time integrators used by default in ODE routines like ODE45 in Matlab (Dormand & Prince, 1980).
Revised: Looking at classical solvers for something related to the multi-step loss, it is natural to think of adaptive time integrators used by default in ODE routines like ODE45 in Matlab (Dormand & Prince, 1980).

Original: Adaptive integrators work by generating two short trajectories of the same time length, but with different step sizes, and as long as the outcome with larger steps differs within some bounds, then the step size is increased.
Revised: Adaptive integrators work by generating two short trajectories of the same time length, but with different step sizes, and as long as the outcome with larger steps differs within some bounds, then the step size is increased.

Original: This guarantees some level of long-term rollout stability just as attempted with the multi-step loss, but the multi-step loss forces the network to implicitly correct for future deviations of the trajectory without actually changing the step size.
Revised: This guarantees some level of long-term rollout stability just as attempted with the multi-step loss, but the multi-step loss forces the network to implicitly correct for future deviations of the trajectory without actually changing the step size.

Original: The adaptive step-size idea has gained popularity in ML with the  Neural ODE paper (Chen et al., 2018).
Revised: The adaptive step-size idea has gained popularity in ML with the introduction of Neural ODEs (Chen et al., 2018).

Original: 3.3. Equivariance bias
Revised: 3.3. Equivariance bias

Original: Numerical PDE solvers come in two flavors: stencil-based and kernel-based, both of which are equivariant to translation, rotation, and reflection in space (Euclidean group equivariance), as well as translation in time (by Noether’s theorem).
Revised: Numerical PDE solvers come in two flavors: stencil-based and kernel-based, both of which are equivariant to translation, rotation, and reflection in space (Euclidean group equivariance), as well as translation in time (by Noether’s theorem).

Original: These properties arise from the conservation of energy, which is a fundamental principle in physics.
Revised: These properties arise from the conservation of energy, which is a fundamental principle in physics.

Original: While equivariance, with respect to the Euclidean group, has been around for a couple of years on grids (Weiler et al., 2018), its extension to the grid-free (Lagrangian) setting is gaining popularity just recently (Brandstetter et al., 2021;
Revised: While equivariance, with respect to the Euclidean group, has been around for a couple of years on grids (Weiler et al., 2018), its extension to the grid-free (Lagrangian) setting is gaining popularity just recently (Brandstetter et al., 2021;

Original: Sch¨utt et al., 2021; Batzner et al., 2022; Musaelian et al., 2022a).
Revised: Sch¨utt et al., 2021; Batzner et al., 2022; Musaelian et al., 2022a).

Original: Here, we talk about equivariance in terms of a neural net operation on vectors, which rotates the output exactly the same way as the input is rotated, as opposed to working with scalar values, which is called an invariant operation, e.g. SchNet (Sch¨utt et al., 2017).
Revised: Here, we talk about equivariance in terms of a neural net operation on vectors, which rotates the output exactly the same way as the input is rotated, as opposed to working with scalar values, which is called an invariant operation, e.g. SchNet (Sch ¨ utt et al., 2017).

Original: The performance boost by including equivariant features is significant and reaches up to an order of magnitude compared to invariant methods (Batzner et al., 2022).
Revised: The performance boost by including equivariant features is significant and reaches up to an order of magnitude compared to invariant methods (Batzner et al., 2022).

Original: 3.4. Input multiple past steps
Revised: 3.4. Input multiple past steps

Original: Another common performance improvement in neural net training is observed by stacking multiple past states as an input (Sanchez-Gonzalez et al., 2020; Pfaff et al., 2021;
Revised: Another common performance improvement in neural net training is observed by stacking multiple past states as an input (Sanchez-Gonzalez et al., 2020; Pfaff et al., 2021;

Original: Brandstetter et al., 2022).
Revised: Brandstetter et al., 2022).

Original: One argument supporting this approach is overfitting prevention by inputting more data (Pfaff et al., 2021).
Revised: One argument supporting this approach is overfitting prevention by inputting more data (Pfaff et al., 2021).

Original: Looking at conventional solvers we very rarely see multiple past states as input and this is done for materials with memory property, e.g. rheological fluids or ”smart” materials.
Revised: Looking at conventional solvers we very rarely see multiple past states as input and this is done for materials with memory property, e.g. rheological fluids or ”smart” materials.

Original: Thus, providing multiple past states implicitly assumes that there is some nonphysical non-Markovian retardation process, which in most cases does not correspond to the physics used for training data generated.
Revised: Thus, providing multiple past states implicitly assumes that there is some nonphysical non-Markovian retardation process, which in most cases does not correspond to the physics used for training data generated.

Original: The only physical justification of a multi-step input we are aware of arises if we train the model to learn a coarsegrained representation of the system.
Revised: The only physical justification of a multi-step input we are aware of arises if we train the model to learn a coarsegrained representation of the system.

Original: Li et al.
Revised: Li et al.

Original: (2015) showed that explicit memory effects are necessary in Dissipative
Revised: (2015) showed that explicit memory effects are necessary in Dissipative

Original: Particle Dynamics for the correct coarse-graining of a complex dynamical system using the Mori-Zwanzig formalism.
Revised: Particle Dynamics for the correct coarse-graining of a complex dynamical system using the Mori-Zwanzig formalism.

Original: Given that papers like GNS and MeshGraphNets do not make use of coarse-graining, it is questionable why we observe improvement in performance and whether this trick generalizes well to different settings.
Revised: Given that papers like GNS and MeshGraphNets do not make use of coarse-graining, it is questionable why we observe improvement in performance and whether this trick generalizes well to different settings.

Original: 3.5. Spatial multi-scale modeling
Revised: 3.5. Spatial multi-scale modeling

Original: Conventional multi-scale methods include, among others, all types of coarse-graining, Wavelet-based methods (e.g.
Revised: Conventional multi-scale methods include, among others, all types of coarse-graining, Wavelet-based methods (e.g.

Original: Kim et al.
Revised: Kim et al.

Original: and the Fast Multipole Method (Rokhlin, 1985).
Revised: and the Fast Multipole Method (Rokhlin, 1985).

Original: Graph networks seem especially suitable for tasks like coarse-graining as they are designed to work on unstructured domains, opposed for example to approaches using
Revised: Graph Networks seem especially suitable for tasks like coarse-graining as they are designed to work on unstructured domains, opposed for example to approaches using

Original: Wavelet or Fourier transforms, which require regular grids.
Revised: Wavelet or Fourier transforms, which require regular grids.

Original: GNNs seem especially promising with many applications in Molecular Dynamics (Husic et al., 2020) and engineering (Migus et al., 2022;
Revised: GNNs seem especially promising with many applications in Molecular Dynamics (Husic et al., 2020) and engineering (Lino et al., 2021;

Original: Valencia et al., 2022;  Han et al., 2022).
Revised: Valencia et al., 2022; Migus et al., 2022; Han et al., 2022).

Original: It is particularly interesting to see works like Migus et al.
Revised: It is particularly interesting to see works like Migus et al.

Original: (2022) inspired by multi-resolution methods and Valencia et al.
Revised: (2022) inspired by multi-resolution methods and Valencia et al.

Original: resembling geometric coarse-graining by weighted averaging.
Revised: resembling geometric coarse-graining by weighted averaging.

Original: All these methods rely on the fact that physical systems exhibit multi-scale behavior, meaning that the trajectory of a particle depends on its closest neighbors, but also on more far-reaching weaker forces.
Revised: All these methods rely on the fact that physical systems exhibit multi-scale behavior, meaning that the trajectory of a particle depends on its closest neighbors, but also on more far-reaching weaker forces.

Original: Splitting the scales and combining their contributions can greatly reduce computation.
Revised: Splitting the scales and combining their contributions can greatly reduce computation.

Original: One of the great advantages of GNNs is their capability to operate on irregularly spaced data, which is necessary for most coarse-graining approaches.
Revised: One of the great advantages of GNNs is their capability to operate on irregularly spaced data, which is necessary for most coarse-graining approaches.

Original: 3.6. Locality of interactions
Revised: 3.6. Locality of interactions

Original: In most cases, graph-based approaches to solving PDEs define the edges in the graph, based on an interaction radius.
Revised: In most cases, graph-based approaches to solving PDEs define the edges in the graph, based on an interaction radius.

Original: Methods using the Graph Network architecture (Battaglia et al., 2018) effectively expand the receptive field of each node with every further layer, in the extreme case resulting in the phenomenon known as over-smoothing.
Revised: Methods using the Graph Network architecture (Battaglia et al., 2018) effectively expand the receptive field of each node with every further layer, in the extreme case resulting in the phenomenon known as over-smoothing.

Original: But if we keep the number of layers reasonably low, the receptive field will always be larger compared to a conventional simulation with the same radius.
Revised: But if we keep the number of layers reasonably low, the receptive field will always be larger compared to a conventional simulation with the same radius.

Original: Until recently, it was thought that a large receptive field is the reason for the success of learned simulators, but Musaelian et al.
Revised: Until recently, it was thought that a large receptive field is the reason for the success of learned simulators, but Musaelian et al.

Original: (2022a) question that assumption.
Revised: (2022a) question that assumption.

Original: In this paper, an equivariant graph network with fixed interaction neighbors performs on a par with the very similar Graph Network-based method NequIP (Batzner et al., 2022) on molecular property prediction tasks.
Revised: In this paper, an equivariant graph network with fixed interaction neighbors performs on a par with the very similar Graph Network-based method NequIP (Batzner et al., 2022) on molecular property prediction tasks.

Original: This finding supports the physics-based argument about the locality of interactions.
Revised: This finding supports the physics-based argument about the locality of interactions.

Original: 3.7. Mesh vs Particle
Revised: 3.7. Mesh vs Particle

Original: GNN-based simulation approaches offer the flexibility to combine particles and meshes out-of-the-box.
Revised: GNN-based simulation approaches offer the flexibility to combine particles and meshes out-of-the-box.

Original: If we then train one neural network to reproduce the results of a Finite
Revised: If we then train one neural network to reproduce the results of a Finite

Original: Element solution on a mesh and Smoothed Particle Hydrodynamics solution over particles, this is where learned methods really shine.
Revised: Element solution on a mesh and Smoothed Particle Hydrodynamics solution over particles, this is where learned methods really shine.

Original: This was achieved with the MeshGraphNets framework (Pfaff et al., 2021).
Revised: This was achieved with the MeshGraphNets framework (Pfaff et al., 2021).

Original: We argue that the transition from particles to meshes is a direct result of a coarse-graining procedure using Voronoi tessellation, which is related to the derivation of the Fluid Particle Model.
Revised: We argue that the transition from particles to meshes is a direct result of a coarse-graining procedure using Voronoi tessellation, which is related to the derivation of the Fluid Particle Model.

Original: The main assumption in this derivation is that each mesh cell should be small enough that it can be treated as being in equilibrium - similar to the assumption made when discretizing a domain with points.
Revised: The main assumption in this derivation is that each mesh cell should be small enough that it can be treated as being in equilibrium - similar to the assumption made when discretizing a domain with points.

Original: 3.8. Stencils
Revised: 3.8. Stencils

Original: We talk about stencils when operating on regular grids.
Revised: We talk about stencils when operating on regular grids.

Original: Although this is not the main strength of GNNs, there are some useful concepts from stencil-based simulations, which are conventionally nontrivial to generalize to particles, but can easily be adapted with GNNs.
Revised: Although this is not the main strength of GNNs, there are some useful concepts from stencil-based simulations, which are conventionally nontrivial to generalize to particles, but can easily be adapted with GNNs.

Original: Brandstetter et al.
Revised: Brandstetter et al.

Original: (2022) state that their paper is motivated by the observation that the Weighted Essentially Non-Oscillatory scheme (WENO) (Shu, 1998) can be written as a special case of a GNN.
Revised: (2022) state that their paper is motivated by the observation that the Weighted Essentially Non-Oscillatory scheme (WENO) (Shu, 1998) can be written as a special case of a GNN.

Original: Another work, inspired by the general idea of the Finite Volume Method, looking at the fluxes at the left and right cell boundary, was developed by (Praditia et al., 2021).
Revised: Another work, inspired by the general idea of the Finite Volume Method, looking at the fluxes at the left and right cell boundary, was developed by Praditia et al.

Original: And finally, a very interesting work inspired by the Finite Element Method in terms of weighting the contributions of neighboring cells by their volume was presented by (Lienen & G¨unnemann, 2022).
Revised: Inspired by the Finite Element Method, finite element networks were introduced by weighting the contributions of neighbouring cells by their volume, as is done in Finite Element analysis (Lienen & G ¨ unnemann, 2022).

Original: 3.9. Integration schemes
Revised: 3.9. Integration schemes

Original: In addition to the time-step adaptation mentioned in relation to multi-step losses, another topic investigated in literature is the order of the integrator (Sanchez-Gonzalez et al., 2019).
Revised: In addition to the time-step adaptation mentioned in relation to multi-step losses, another topic investigated in literature is the order of the integrator (Sanchez-Gonzalez et al., 2019).

Original: This work points to the fact that higher order integrators lead to much better robustness, with respect to the choice of an integration time step.
Revised: This work points to the fact that higher order integrators lead to much better robustness, with respect to the choice of an integration time step.

Original: Another interesting question discussed in this paper is whether symplectic integrators improve performance of a learned Hamiltonian neural net.
Revised: Another interesting question discussed in this paper is whether symplectic integrators improve performance of a learned Hamiltonian neural net.

Original: The answer seems to be that the symplectic property is much less important than the order of the integrator, which is in contrast with conventional Molecular Dynamics integrators, which work extremely poorly if not symplectic.
Revised: The answer seems to be that the symplectic property is much less important than the order of the integrator, which is in contrast with conventional Molecular Dynamics integrators, which work extremely poorly if not symplectic.

Original: 4. Untapped Ideas from Classical Approaches
Revised: 4. Untapped Ideas from Classical Approaches

Original: In this subsection, we introduce potentially useful ideas from conventional differential equation solvers in science, which to the best of our knowledge have not been adapted in main-stream learned PDE solvers yet.
Revised: In this subsection, we introduce potentially useful ideas from conventional differential equation solvers in science, which to the best of our knowledge have not been adapted in main-stream learned PDE solvers yet.

Revised: Figure 4 is a collection of these concepts in the form of a word cloud.

Original: 4.1. Noise during inference
Revised: 4.1. Noise during inference

Original: Adding noise to the inputs during training has proven to be useful, but has not been done during testing.
Revised: Adding noise to the inputs during training has proven to be useful, but has not been done during testing.

Original: One idea would be to use noise during inference to emulate Brownian motion.
Revised: One idea would be to use noise during inference to emulate Brownian motion.

Original: And one further topic we already mentioned is the relation of the noise scale to particle mass.
Revised: And one further topic we already mentioned is the relation of the noise scale to particle mass.

Original: From mesoscopic methods and the Fluctuation-dissipation theorem we would expect the noise to scale as 1 / √ m if a coarser representation is used.
Revised: From mesoscopic methods and the Fluctuation-dissipation theorem we would expect the noise to scale as 1 / √ m if a coarser representation is used.

Original: 4.2. Multiple time steps
Revised: 4.2. Multiple time steps

Original: Learned Molecular Dynamics simulations stick to using only the last past state and doing the same for larger-scale simulations might partially explain the unphysical behavior of the GNS method demonstrated in Klimesch et al.
Revised: Learned Molecular Dynamics simulations stick to using only the last past state and doing the same for larger-scale simulations might partially explain the unphysical behavior of the GNS method demonstrated in Klimesch et al.

Original: For coarse-graining though a longer history might be helpful.
Revised: For coarse-graining though a longer history might be helpful.

Original: 4.3. Feature Engineering
Revised: 4.3. Feature Engineering

Original: From the Volume of Fluid Method we could adapt the idea of including features corresponding to the ratio of different material, if we are interested in simulating multi-material flows.
Revised: From the Volume of Fluid Method we could adapt the idea of including features corresponding to the ratio of different material, if we are interested in simulating multi-material flows.

Original: The Discrete Element Method suggests encoding much more features like rotational degree of freedom (in magnetic field or simulating friction), stateful contact information (contact simulations), and often complicated geometry (for non-spherical, e.g. granular particles).
Revised: The Discrete Element Method suggests encoding much more features like rotational degree of freedom (in magnetic field or simulating friction), stateful contact information (contact simulations), and often complicated geometry (for non-spherical, e.g. granular particles).

Original: Inspired by shock-capturing methods used routinely for the solution of nonlinear fluid dynamics problems (Ketcheson et al., 2020), one could think of further hand-tuned node features indicating the presence of a shock.
Revised: Inspired by shock-capturing methods used routinely for the solution of nonlinear fluid dynamics problems (Ketcheson et al., 2020), one could think of further hand-tuned node features indicating the presence of a shock.

Original: 4.4. Particles and Grid
Revised: 4.4. Particles and Grid

Original: There are a number of methods using the best of both particle and grid worlds like the Particle-in-Cell method and its successor Material Point Method.
Revised: There are a number of methods using the best of both particle and grid worlds like the Particle-in-Cell method and its successor Material Point Method.

Original: The idea of updating the node features and from time to time also based on the grid cell they belong to, might speed up simulations and is worth exploring.
Revised: The idea of updating the node features and from time to time also based on the grid cell they belong to, might speed up simulations and is worth exploring.

Original: Now, if we restrict ourselves to regularly spaced particles, respectively grid cells, our solver toolkit becomes much richer with methods like the Fast Fourier Transform (which has already seen great success with the Fourier Neural Operator (Li et al., 2020)) and the Wavelet Transform (as used in the PDE-Net (Long et al., 2018)) at our disposal, as mentioned above in the context of multi-scale modeling.
Revised: Now, if we restrict ourselves to regularly spaced particles, respectively grid cells, our solver toolkit becomes much richer with methods like the Fast Fourier Transform (which has already seen great success with the Fourier Neural Operator (Li et al., 2020)) and the Wavelet Transform (as used in the PDE-Net (Long et al., 2018)) at our disposal, as mentioned above in the context of multi-scale modeling.

Original: 4.5. Integrator
Revised: 4.5. Integrator

Original: Taking the perspective of Neural ODEs (Chen et al., 2018) with the neural network learning the perfect acceleration, one could arguably expect the next evolutionary step to be the combination of learned integrators with adaptive integration schemes.
Revised: Taking the perspective of Neural ODEs (Chen et al., 2018) with the neural network learning the perfect acceleration, one could arguably expect the next evolutionary step to be the combination of learned integrators with adaptive integration schemes.

Original: Incorporating insights from classical numerical methods, one should possibly seek to define an equivalent stability criterion for learned methods as the
Revised: Incorporating insights from classical numerical methods, one should possibly seek to define an equivalent stability criterion for learned methods as the

Original: Courant-Friedrichs-Lewy (CFL) condition for classical numerical methods.
Revised: Courant-Friedrichs-Lewy (CFL) condition for classical numerical methods.

Original: This would in turn aid in bounding the time, and subsequently explore time steps smaller than the critical value.
Revised: This would in turn aid in bounding the time, and subsequently explore time steps smaller than the critical value.

Original: 5. Conclusion & Discussion
Revised: 5. Conclusion & Discussion

Original: In this article, we claim that studying classical PDE solvers and their past development offers a direct path to the acceleration of the development of learned PDE solvers.
Revised: In this article, we claim that studying classical PDE solvers and their past development offers a direct path to the acceleration of the development of learned PDE solvers.

Original: Example in literature show that biasing a learned solver by means of architectural design, data augmentation, feature engineering, etc.
Revised: Examples in literature show that biasing a learned solver by means of architectural design, data augmentation, feature engineering, etc.

Original: incorporating existing knowledge from classical solvers can greatly improve performance, explainability, and data-efficiency.
Revised: incorporating existing knowledge from classical solvers can greatly improve performance, explainability, and data-efficiency.

Original: In Section 2 we show how this development has already subconsciously played out in the development of graph-based learned solvers following the same development as particlebased methods such as Molecular Dynamics, Smoothed
Revised: In Section 2 we show how this development has already subconsciously played out in the development of graph-based learned solvers following the same development as particlebased methods such as Molecular Dynamics, Smoothed

Original: Particle Hydrodynamics, and the Fluid-Particle Model.
Revised: Particle Hydrodynamics, and the Fluid-Particle Model.

Original: This investigation is revisited for algorithmic comparisons, and illustrations of the limitations of classical solvers later on.
Revised: This investigation is revisited for algorithmic comparisons and illustrations of the limitations of classical solvers later on.

Original: In
Revised: In

Original: Section 3 we then focus on ideas from classical approaches which have found their way into recent learned solver literature, and discuss the physical interpretation of these developments.
Revised: Section 3 we then focus on ideas from classical approaches which have found their way into recent learned solver literature, and discuss the physical interpretation of these developments.

Original: In the discussed examples the included physically motivated biases are used to improve robustness w.r.t.
Revised: In the discussed examples, the included physically motivated biases are used to improve robustness w.r.t.

Original: hyperparameter choices, lower errors, and faster inference.
Revised: hyperparameter choices, lower errors, and speed-up inference.

Original: Section 4 takes a glimpse into a possible version of the future with ideas which have, to the best of our knowledge, not yet been integrated in learned methods.
Revised: Section 4 takes a glimpse into a possible version of the future with ideas which have, to the best of our knowledge, not yet been integrated in learned methods.

Original: Given the elaborate history of classical methods, and the short, but highly dynamic history of learned approaches, there is still a lot of potential to be realized within the latter by incorporating insights from the former.
Revised: Given the elaborate history of classical methods, and the short, but highly dynamic history of learned approaches, there is still a lot of potential to be realized within the latter by incorporating insights from the former.

Original: Going further, many exciting problems in the physical sciences, such as simulations involving multiple spatial scales, multiple temporal scales, non-Newtonian fluids, or phasechanging materials, are heavily data-constrained and will hence have to rely on insights from classical methods for
Revised: Going further, many exciting problems in the physical sciences, such as simulations involving multiple spatial scales, multiple temporal scales, non-Newtonian fluids, or phasechanging materials, are heavily data-constrained and will hence have to rely on insights from classical methods for

Original: Machine Learning approaches to become feasible.
Revised: Machine Learning approaches to become feasible.