Original: CROM: C ONTINUOUS R EDUCED -O RDER M ODELING OF PDE S U SING I MPLICIT N EURAL R EPRESENTATIONS
Revised: CROM: C ONTINUOUS R EDUCED -O RDER M ODELING OF PDE S U SING I MPLICIT N EURAL R EPRESENTATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The long runtime of high-ﬁdelity partial differential equation (PDE) solvers makes them unsuitable for time-critical applications.
Revised: The long runtime of high-fidelity partial differential equation (PDE) solvers makes them unsuitable for time-critical applications.

Original: We propose to accelerate PDE solvers using reduced-order modeling (ROM).
Revised: We propose to accelerate PDE solvers using reduced-order modeling (ROM).

Original: Whereas prior ROM approaches reduce the dimensionality of discretized vector ﬁelds, our continuous reducedorder modeling (CROM) approach builds asmooth, low-dimensional manifold of the continuous vector ﬁelds themselves, not their discretization.
Revised: Whereas prior ROM approaches reduce the dimensionality of discretized vector fields, our continuous reducedorder modeling (CROM) approach builds a low-dimensional embedding of the continuous vector fields themselves, not their discretization.

Original: We represent this reduced manifold using continuously differentiable neural ﬁelds, which may train on any and all available numerical solutions of the continuous system, even when they are obtained using diverse methods or discretizations.
Revised: We represent this reduced manifold using continuously differentiable neural fields, which may train on any and all available numerical solutions of the continuous system, even when they are obtained using diverse methods or discretizations.

Original: We validate our approach on an extensive range of PDEs with training data from voxel grids, meshes, and point clouds.
Revised: We validate our approach on an extensive range of PDEs with training data from voxel grids, meshes, and point clouds.

Original: Compared to prior discretization-dependent ROM methods, such as linear subspace proper orthogonal decomposition (POD) and nonlinear manifold neural-network-based autoencoders, CROM features higher accuracy, lower memory consumption, dynamically adaptive resolutions, and applicability to any discretization.
Revised: Compared to prior discretization-dependent ROM methods, such as linear subspace proper orthogonal decomposition (POD) and nonlinear manifold neural-network-based autoencoders, CROM features higher accuracy, lower memory consumption, dynamically adaptive resolutions, and applicability to any discretization.

Original: For equal latent space dimension, CROM exhibits 79 × and 49 × better accuracy, and 39 × and 132 × smaller memory footprint, than POD and autoencoder methods, respectively.
Revised: For equal latent space dimension, CROM exhibits 79 × and 49 × better accuracy, and 39 × and 132 × smaller memory footprint, than POD and autoencoder methods, respectively.

Original: Experiments demonstrate 109 × and 89 × wall-clock speedups over unreduced models on CPUs and GPUs, respectively.
Revised: Experiments demonstrate 109 × and 89 × wall-clock speedups over unreduced models on CPUs and GPUs, respectively.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many scientiﬁc and engineering models are posed as partial differential equations (PDEs) of the form
Revised: Many scientific and engineering models are posed as partial differential equations (PDEs) of the form

Original: [Equation]
Revised: [Equation]

Original: subject to initial and boundary conditions.
Revised: subject to initial and boundary conditions.

Original: Here f is a spatiotemporal dependent, multidimensional continuous vector ﬁeld, such as temperature, velocity, or displacement; ∇ and ˙( · ) are the spatial and temporal gradients; Ω ⊂ R m and T ⊂ R are the spatial and temporal domains, respectively.
Revised: Here f is a spatiotemporal dependent, multidimensional continuous vector field, such as temperature, velocity, or displacement; ∇ and ˙( · ) are the spatial and temporal gradients; Ω ⊂ R m and T ⊂ R are the spatial and temporal domains, respectively.

Original: We may solve for f by discretizing in space, f ( x , t )
Revised: We may solve for f by discretizing in space, f ( x , t )

Original: ≈ f P ( x , t ) =
Revised: ≈ f P ( x , t ) =

Original: (cid:80) Pi =
Revised: (cid:80) Pi =

Original: 1 a i ( t ) N i ( x ) , transforming the continuous spatial representation to a ( P · d )-dimensional vector whose coefﬁcients a i ( t ) :
Revised: 1 a i ( t ) N i ( x ) , transforming the continuous spatial representation to a ( P · d )-dimensional vector whose coefficients a i ( t ) :

Original: T → R d and the corresponding basis functions
Revised: T → R d and the corresponding basis functions

Original: N i ( x ) : Ω → R (e.g., polynomial basis, fourier basis) approximate the continuous solution.
Revised: N i ( x ) : Ω → R (e.g., polynomial basis, fourier basis) approximate the continuous solution.

Original: For instance, if N i is the linear ﬁnite element basis, the coefﬁcients a i ( t ) =
Revised: For instance, if N i is the linear finite element basis, the coefficients a i ( t ) =

Original: f ( x i , t ) are ﬁeld values at spatial samples 
Revised: f ( x i , t ) are field values at spatial samples x

Original: x i (Hughes, 2012).
Revised: i (Hughes, 2012).

Original: After introducing temporal samples { t n } Tn =0 , we temporally evolve the solution by solving for P unknowns { a i ( t n +1 ) } given the previous state { a i ( t n ) } .
Revised: After introducing temporal samples { t n } Tn =0 , we temporally evolve the solution by solving for P unknowns { a i ( t n +1 ) } given the previous state { a i ( t n ) } .

Original: Unfortunately, when P is large, processing and memory costs of these full-order solves become intractable.
Revised: Unfortunately, when P is large, processing and memory costs of these full-order solves become intractable.

Original: To alleviate this computational burden, prior model reduction techniques (Berkooz et al., 1993; Willcox & Peraire, 2002;
Revised: To alleviate this computational burden, prior model reduction techniques (Berkooz et al., 1993; Willcox & Peraire, 2002;

Original: Benner et al., 2015) construct a low-dimensional manifold g P : R r (cid:55)→ R Pd , with r (cid:28) Pd , such that every  latent space vector q ( t ) ∈ R r maps to a discrete ﬁeld g P ( q ) (cid:55)→ ( a 1 , . . .
Revised: Benner et al., 2015) construct a manifold-parameterization function g P : R r (cid:55)→ R Pd , with r ≪ Pd , such that every low-dimensional latent space vector q ( t ) ∈ R r maps to a discrete field g P ( q ) (cid:55)→ ( a 1 , . . .

Original: , a P ) T .
Revised: , a P ) T .

Original: For instance, for linear ﬁnite elements (Barbiˇc & James, 2005), g P ( q ) (cid:55)→ (cid:0) f
Revised: For instance, for linear finite elements (Barbiˇc & James, 2005), g P ( q ) (cid:55)→ (cid:0) f

Original: ( x 1 , t ) , . . .
Revised: ( x 1 , t ) , . . .

Original: f ( x P , t ) (cid:1) T , as depicted in Figure 1a.
Revised: f ( x P , t ) (cid:1) T , as depicted in Figure 1a.

Original: ROM saves computation because it requires evolving only r (cid:28) Pd latent space variables.
Revised: ROM saves computation because it requires evolving only r ≪ Pd latent space variables.

Original: [Figure] Since existing ROM approaches apply to already-discretized ﬁelds, model training and PDE solving are tied to the dimension and discretization type of the training data, causing key limitations: 
Revised: Since existing ROM approaches apply to already-discretized fields, model training and PDE solving are tied to the dimension and discretization type of the training data, causing key limitations: [Figure]

Original: Discretization dependence.
Revised: Discretization dependence.

Original: If we alter the training simulation resolution ( P ) or the discretization types (e.g., meshes to point clouds), we must also alter the architecture and numbers of parameters.
Revised: If we alter the training simulation resolution ( P ) or the discretization types (e.g., meshes to point clouds), we must also alter the architecture and numbers of parameters.

Original: Memory scaling.
Revised: Memory scaling.

Original: Memory footprint grows with discretization resolution P .
Revised: Memory footprint grows with discretization resolution P .

Original: Fixed discretization.
Revised: Fixed discretization.

Original: We cannot dynamically adapt spatial resolution P , discretization type, or basis function N i during latent-space-PDE solves, e.g., dynamic remeshing (Peraire et al., 1987).
Revised: We cannot dynamically adapt spatial resolution P , discretization type, or basis function N i during latent-space-PDE solves, e.g., dynamic remeshing (Peraire et al., 1987).

Original: Altogether these problems arise because the architecture of g P ( q ) is tied to the discretization ( a 1 , . .
Revised: Altogether these problems arise because the architecture of g P ( q ) is tied to the discretization ( a 1 , . .

Original: . , a P ) T .
Revised: . , a P ) T .

Original: Introducing a discretization-independent architecture In an alternative point of departure, we train a low-dimensional manifold g ( x , q )
Revised: Introducing a discretization-independent architecture In an alternative point of departure, we train a manifold-parameterization function g ( x , q )

Original: ≈ f ( x , t ) to approximate the continuous ﬁeld itself, not its discretization (see Figure 1b).
Revised: ≈ f ( x , t ) to approximate the continuous field itself, not its discretization (see Figure 1b).

Original: Note that the domain and co-domain of g are continuous domains: they do not depend on the choice of discretization(s) used at any stage of the process, i.e., during preparation of training data, nor during latent-space-PDE solving.
Revised: Note that the domain and co-domain of g are continuous domains: they do not depend on the choice of discretization(s) used at any stage of the process, i.e., during preparation of training data, nor during latent-space-PDE solving.

Original: In this sense, the manifold architecture is discretization independent.
Revised: In this sense, the manifoldparameterization architecture is discretization independent.

Original: In our implementation, g is embodied as an implicit neural representation (Park et al., 2019; Chen & Zhang, 2019; Mescheder et al., 2019), also known as a neural ﬁeld, yielding a smooth and analytically-differentiable manifold.
Revised: In our implementation, g is embodied as an implicit neural representation (Park et al., 2019; Chen & Zhang, 2019; Mescheder et al., 2019), also known as a neural field, yielding a smooth and analytically-differentiable manifold-parameterization.

Original: This representation’s memory footprint depends on the complexity of ﬁelds produced by the PDE, not the discretization resolution .
Revised: This representation’s memory footprint depends on the complexity of fields produced by the PDE, not the discretization resolution .

Original: After training, we evolve the latent variables, as governed by the PDE, for previously-unexplored parameters.
Revised: After training, we evolve the latent variables, as governed by the PDE, for previously-unexplored parameters.

Original: Unlike approaches that discard the PDE after training, we evaluate the original PDE at a small number of domain points at every time integration step.
Revised: Unlike approaches that discard the PDE after training, we evaluate the original PDE at a small number of domain points at every time integration step.

Original: We validate our approach on classic
Revised: We validate our approach on classic

Original: PDEs with discretized data from voxels, meshes, and point clouds.
Revised: PDEs with discretized data from voxels, meshes, and point clouds.

Original: In comparison to the full-order model, our approach reduces the number of spatial degrees of freedom, memory, and computational cost.
Revised: In comparison to the full-order model, our approach reduces the number of spatial degrees of freedom, memory, and computational cost.

Original: In comparison to prior linear and nonlinear discretization-dependent model reduction methods, our method exhibits higher accuracy and consumes less memory.
Revised: In comparison to prior linear and nonlinear discretization-dependent model reduction methods, our method exhibits higher accuracy and consumes less memory.

Original: To highlight another beneﬁt of being discretization-agnostic, we demonstrate an elasticity simulation that adapts mesh resolution.
Revised: To highlight another benefit of being discretization-agnostic, we demonstrate an elasticity simulation that adapts mesh resolution.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Reduced-Order Modeling for PDEs.
Revised: Reduced-Order Modeling for PDEs.

Original: Early works on identifying a low-dimensional latent space focused on linear methods (Berkooz et al., 1993; Holmes et al., 2012), e.g., proper orthogonal decomposition (POD) or principal component analysis (PCA).
Revised: Early works on identifying a low-dimensional latent space focused on linear methods (Berkooz et al., 1993; Holmes et al., 2012), e.g., proper orthogonal decomposition (POD) or principal component analysis (PCA).

Original: Recent nonlinear manifolds (Fulton et al., 2019; Lee & Carlberg, 2020), often constructed via autoencoder neural networks, have been shown to signiﬁcantly outperform their linear counterparts on slowly decaying Kolmogorov n-width problems (Peherstorfer, 2022).
Revised: Recent nonlinear manifolds (Fulton et al., 2019; Lee & Carlberg, 2020), often constructed via autoencoder neural networks, have been shown to significantly outperform their linear counterparts on slowly decaying Kolmogorov n-width problems (Peherstorfer, 2022).

Original: Most of these prior works exclusively focus on building a latent space for the already-dicretized vector ﬁelds.
Revised: Most of these prior works exclusively focus on building a latent space for the already-dicretized vector fields.

Original: Chen et al.
Revised: Chen et al.

Original: (2021); Pan et al.
Revised: (2021); Pan et al.

Original: and our work attempt to construct the latent space for the continuous vectors themselves.
Revised: and our work attempt to construct the latent space for the continuous vectors themselves.

Original: However, Chen et al.
Revised: However, Chen et al.

Original: (2021)’s treatment specializes in the material point method (discretization) and elasticity (PDE); our general treatment is both discretization and PDE agnostic.
Revised: (2021)’s treatment specializes in the material point method (discretization) and elasticity (PDE); our general treatment is both discretization and PDE agnostic.

Original: Likewise, Pan et al.
Revised: Likewise, Pan et al.

Original: (2022) trains a discretization-agnostic latent space for PDE data; we further solve the PDEs in the reduced space via rapid latent space traversal.
Revised: (2022) trains a discretization-agnostic latent space for PDE data; we further solve the PDEs in the reduced space via rapid latent space traversal.

Original: Additional references to ROM are listed in Appendix A.
Revised: Additional references to ROM are listed in Appendix A.

Original: Implicit neural representations use (fully-connected) neural networks to represent arbitrary vector ﬁelds.
Revised: Implicit neural representations use (fully-connected) neural networks to represent arbitrary vector fields.

Original: While the Euclidean space spatial coordinates always form part of the input to the network, it is also common to have a latent space vector to complement the rest of the input.
Revised: While the Euclidean space spatial coordinates always form part of the input to the network, it is also common to have a latent space vector to complement the rest of the input.

Original: Different latent space vectors correspond to different states of the continuous vector ﬁeld (see Figure 1b), e.g. different geometries (Park et al., 2019; Chen & Zhang, 2019; Mescheder et al., 2019) or different radiance ﬁelds (Mildenhall et al., 2020).
Revised: Different latent space vectors correspond to different states of the continuous vector field (see Figure 1b), e.g. different geometries (Park et al., 2019; Chen & Zhang, 2019; Mescheder et al., 2019) or different radiance fields (Mildenhall et al., 2020).

Original: A key contribution of our work is nonlinearly traversing the latent space of neural representations under an explicit PDE constraint.
Revised: A key contribution of our work is nonlinearly traversing the latent space of neural representations under an explicit PDE constraint.

Original: Machine learning (ML) for PDEs.
Revised: Machine learning (ML) for PDEs.

Original: Physics-informed neural networks (PINNs) (Raissi et al., 2019;
Revised: Physics-informed neural networks (PINNs) (Raissi et al., 2019;

Original: Sitzmann et al., 2020b) demonstrate that PDEs can be accurately solved via neural representations.
Revised: Sitzmann et al., 2020b) demonstrate that PDEs can be accurately solved via neural representations.

Original: Notably, PINN enables prediction and discovery from incomplete models and incomplete data (Karniadakis et al., 2021).
Revised: Notably, PINN enables prediction and discovery from incomplete models and incomplete data (Karniadakis et al., 2021).

Original: However, the degrees of freedom involved in their approaches are still
Revised: However, the degrees of freedom involved in their approaches are still

Original: Pd , and the underlying gradient-descent-based solver is often computationally more expensive than traditional solvers (see Table 1 by Zehnder et al.
Revised: Pd , and the underlying gradient-descent-based solver is often computationally more expensive than traditional solvers (see Table 1 by Zehnder et al.

Original: By contrast, our goal is building a computationally more efﬁcient solution that solves for only r degrees of freedom ( r (cid:28) Pd ).
Revised: By contrast, our goal is building a computationally more efficient solution that solves for only r degrees of freedom ( r ≪ Pd ).

Original: In fact, our approach can be viewed as an extension of PINN for model reduction.
Revised: In fact, our approach can be viewed as an extension of PINN for model reduction.

Original: Setting the latent space vector q ( t ) in our formulation (see Figure 1b) to the time variable t recovers the exact formulation of PINN.
Revised: Setting the latent space vector q ( t ) in our formulation (see Figure 1b) to the time variable t recovers the exact formulation of PINN.

Original: In addition to PINN, Sanchez-Gonzalez et al.
Revised: In addition to PINN, Sanchez-Gonzalez et al.

Original: (2020) show graph neural network (GNN) architectures are also capable of learning PDEs.
Revised: (2020) show graph neural network (GNN) architectures are also capable of learning PDEs.

Original: Yet, like PINNs, GNNs also do not offer dimension reduction.
Revised: Yet, like PINNs, GNNs also do not offer dimension reduction.

Original:  M ETHOD : O VERVIEW AND M ANIFOLD C ONSTRUCTION
Revised:  M ETHOD : O VERVIEW AND M ANIFOLD C ONSTRUCTION

Original: Overview Our goal is to efﬁciently obtain the solution of Equation (1).
Revised: Overview Our goal is to efficiently obtain the solution of Equation (1).

Original: We begin by constructing a low dimensional manifold (see below), after which we solve PDEs by time-integrating the dynamics of the manifold’s latent space vector (see Section 4).
Revised: We begin by constructing a low-dimensional embedding and the corresponding manifold-parameterization function (see below), after which we solve PDEs by time-integrating the dynamics of the manifold’s latent space vector (see Section 4).

Original: As we demonstrate in examples from various scientiﬁc disciplines (see Section 5), this general method is applicable regardless of the discretization of the training data (e.g., voxel grids, meshes, point clouds) or the discretization deemed most useful for evaluating the gradients (e.g., physical forces) when solving PDEs on the constructed manifold.
Revised: As we demonstrate in examples from various scientific disciplines (see Section 5), this general method is applicable regardless of the discretization of the training data (e.g., voxel grids, meshes, point clouds) or the discretization deemed most useful for evaluating the gradients (e.g., physical forces) when solving PDEs on the constructed manifold.

Original: Low-dimensional Manifold Construction As depicted in Figure 1b, we seek a manifold g ( x , q ) ,
Revised: Manifold Construction As depicted in Figure 1b, we seek a manifold-parameterization function g ( x , q ) ,

Original: [Equation]
Revised: [Equation]

Original: that well approximates the continuous ﬁeld f ( x , t ; µ ) throughout the spatiotemporal domain Ω × T , and for a workable range of problem parameters µ ∈ D .
Revised: that well approximates the continuous field f ( x , t ; µ ) throughout the spatiotemporal domain Ω × T , and for a workable range of problem parameters µ ∈ D .

Original: For ease of exposition, we omit the dependencies of q and f on the problem parameters µ .
Revised: For ease of exposition, we omit the dependencies of q and f on the problem parameters µ .

Original: Here D is an arbitrary parameter space (e.g., material properties, external forces, user settings).
Revised: Here D is an arbitrary parameter space (e.g., material properties, external forces, user settings).

Original: For scenarios that do not feature trivial parameterizations, e.g., external force via crowd-sourcing (Barbiˇc & James, 2005), D can also be implicitly deﬁned.
Revised: For scenarios that do not feature trivial parameterizations, e.g., external force via crowd-sourcing (Barbiˇc & James, 2005), D can also be implicitly defined.

Original: What distinguishes our approach from typical model reduction is that g takes the position x ∈ Ω as an input.
Revised: What distinguishes our approach from typical model reduction is that g takes the position x ∈ Ω as an input.

Original: Thus, unlike prior approaches that infer only discrete coefﬁcients, our approach infers ﬁeld values at arbitrary domain positions x .
Revised: Thus, unlike prior approaches that infer only discrete coefficients, our approach infers field values at arbitrary domain positions x .

Original: We parameterize g with a neural network g θ g whose weights θ g satisfy the minimization problem
Revised: We parameterize g with a neural network g θ g whose weights θ g satisfy the minimization problem

Original: [Equation]
Revised: [Equation]

Original: where D train ⊂ D is the training set, and q ( t n ) is the latent space vector shared among all spatial samples.
Revised: where D train ⊂ D is the training set, and q ( t n ) is the latent space vector shared among all spatial samples.

Original: This objective aims to reproduce all the ﬁeld values present in the training data, generated
Revised: This objective aims to reproduce all the field values present in the training data, generated

Original: [Figure]
Revised: [Figure]

Original: via full-order PDE solutions.
Revised: via full-order PDE solutions.

Original: Notably, our approach imposes no limit on the discretization strategy of the PDE solver.
Revised: Notably, our approach imposes no limit on the discretization strategy of the PDE solver.

Original: For instance, this framework is applicable to training data from both ﬁnite difference methods and ﬁnite element methods as well as both voxel grids and meshes.
Revised: For instance, this framework is applicable to training data from both finite difference methods and finite element methods as well as both voxel grids and meshes.

Original: There are commonly two approaches to deﬁne the latent space vector q : the auto-decoder approach (Park et al., 2019) that trains q along with g θ g and the encoder approach (Chen & Zhang, 2019;
Revised: There are commonly two approaches to define the latent space vector q : the auto-decoder approach (Park et al., 2019) that trains q along with g θ g and the encoder approach (Chen & Zhang, 2019;

Original: Mescheder et al., 2019) that trains a separate network to output q .
Revised: Mescheder et al., 2019) that trains a separate network to output q .

Original: While both approaches work for our application, we adopt the latter.
Revised: While both approaches work for our application, we adopt the latter.

Original: The encoder network e θ e with weights θ e takes an input vector constructed by concatenating all the discrete degrees of freedom from the training data and outputs a latent space vector (see Figure 2):
Revised: The encoder network e θ e with weights θ e takes an input vector constructed by concatenating all the discrete degrees of freedom from the training data and outputs a latent space vector (see Figure 2):

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We emphasize that this discretization-dependent encoder (Xie et al., 2021) ismerely a tool for training the smoothly varying latent space.
Revised: We emphasize that this discretization-dependent encoder (Xie et al., 2021) is a tool for training the smoothly varying latent space and for determining the initial latent space vector from the discretized training data.

Original: The implicit neural representation g θ g remains a discretizationagnostic architecture.
Revised: Otherwise, the implicit neural representation g θ g remains a discretization-agnostic architecture.

Original: Adding the encoder, Equation (3) now becomes
Revised: Adding the encoder, Equation (3) now becomes

Original: [Equation]
Revised: [Equation]

Original: Figure 2 illustrates the training pipeline.
Revised: Figure 2 illustrates the training pipeline.

Original: Please refer to Appendix D for network and training details and Appendix F for hyperparameter selection.
Revised: Please refer to Appendix D for network and training details and Appendix F for hyperparameter selection.

Original:  M ETHOD : L ATENT S PACE D YNAMICS
Revised:  M ETHOD : L ATENT S PACE D YNAMICS

Original: After the manifold is constructed, we compute  latent space dynamics ( q n (cid:55)→ q n +1 ) in three steps (see Figure 3 and appendix Q): (1) network inference, (2) PDE time-stepping, and (3) network inversion.
Revised: After the manifold is constructed, we compute the low-dimensional latent space dynamics ( q n (cid:55)→ q n +1 ) in three steps (see Figure 3 and appendix Q): (1) network inference, (2) PDE time-stepping, and (3) network inversion.

Original: The neural network strictly serves as a kinematic spatial representation in the maps from and to the latent space (steps 1 and 3, respectively).
Revised: The neural network strictly serves as a kinematic spatial representation in the maps from and to the latent space (steps 1 and 3, respectively).

Original: The time integration (step 2) itself uses the original PDE, not a neural network approximation thereof.
Revised: The time integration (step 2) itself uses the original PDE, not a neural network approximation thereof.

Original: The ROM community has demonstrated that these three steps can yield strong long-time stability even on stiff and chaotic dynamical systems (Carlberg et al., 2013; 2017).
Revised: The ROM community has demonstrated that these three steps can yield strong long-time stability even on stiff and chaotic dynamical systems (Carlberg et al., 2013; 2017).

Original: Appendices O and P demonstrate smooth latent space trajectories and empirical stability analysis using this approach.
Revised: Appendices O and P demonstrate smooth latent space trajectories and empirical stability analysis using this approach.

Original: [Figure]
Revised: [Figure]

Original: Commonly shared among all three steps are “integration samples”, a ﬁnite set of spatial domain points M :=
Revised: Commonly shared among all three steps are “integration samples”, a finite set of spatial domain points M :=

Original: { y j ∈ Ω | 1 ≤ j ≤ |M|} chosen at the user’s discretion (see Section 4.4).
Revised: { y j ∈ Ω | 1 ≤ j ≤ |M|} chosen at the user’s discretion (see Section 4.4).

Original: These samples need not coincide with the previously mentioned full-order ﬁnite element discretization samples { x i } Pi =1 .
Revised: These samples need not coincide with the previously mentioned full-order finite element discretization samples { x i } Pi =1 .

Original: 4. S TEP 1: N ETWORK I NFERENCE
Revised: 4. S TEP 1: N ETWORK I NFERENCE

Original: We ﬁrst aim to gather all the full-space spatiotemporal information ( ∀ y ∈ M ) necessary for PDE time integration.
Revised: We first aim to gather all the full-space spatiotemporal information ( ∀ y ∈ M ) necessary for PDE time integration.

Original: The function value f itself can be evaluated via inferencing of the neural network f
Revised: The function value f itself can be evaluated via inferencing of the neural network f

Original: ( y , t n ) =
Revised: ( y , t n ) =

Original: g θ g ( y , q n ) .
Revised: g θ g ( y , q n ) .

Original: The spatial and temporal gradients are computed either by differentiating the network, ∇ f ( y , t n ) = ∇ y g θ g and ˙
Revised: The spatial and temporal gradients are computed either by differentiating the network, ∇ f ( y , t n ) = ∇ y g θ g and ˙

Original: f ( y , t n ) =
Revised: f ( y , t n ) =

Original: ∂ g θg ∂ q ˙ q n , respectively, or by numerical approximation.
Revised: ∂ g θg ∂ q ˙ q n , respectively, or by numerical approximation.

Original: Higher-order gradients may be generalized in a similar manner.
Revised: Higher-order gradients may be generalized in a similar manner.

Original: Further details on gradient computation are listed in Appendix E.
Revised: Further details on gradient computation are listed in Appendix E.

Original: 4. S TEP 2: PDE T IME - STEPPING
Revised: 4. S TEP 2: PDE T IME - STEPPING

Original: We now evolve time from t n to t n +1 .
Revised: We now evolve time from t n to t n +1 .

Original: Unlike end-to-end learning-based latent space dynamics methods (Lusch et al., 2018), we evaluate time derivatives using the exact PDE (1), not a learned surrogate.
Revised: Unlike end-to-end learning-based latent space dynamics methods (Lusch et al., 2018), we evaluate time derivatives using the exact PDE (1), not a learned surrogate.

Original: At each integration point y , we evaluate the temporal derivative by solving the PDE (1) for ˙
Revised: At each integration point y , we evaluate the temporal derivative by solving the PDE (1) for ˙

Original: f n +1 ( y ) :
Revised: f n +1 ( y ) :

Original: [Equation]
Revised: [Equation]

Original: We evolve the conﬁguration to time t n +1 = t n + ∆ t using the chosen explicit time integration method I F subject to given boundary conditions, e.g., Runge-Kutta methods (Dormand & Prince, 1980):
Revised: We evolve the configuration to time t n +1 = t n + ∆ t using the chosen explicit time integration method I F subject to given boundary conditions, e.g., Runge-Kutta methods (Dormand & Prince, 1980):

Original: [Equation]
Revised: [Equation]

Original: While this work focuses on explicit time integration, we can also extend the framework for implicit time integration (Carlberg et al., 2017).
Revised: While this work focuses on explicit time integration, we can also extend the framework for implicit time integration (Carlberg et al., 2017).

Original: 4. S TEP 3: N ETWORK I NVERSION
Revised: 4. S TEP 3: N ETWORK I NVERSION

Original: We project back onto the reduced manifold by ﬁnding the corresponding input q n +1 that best matches the evolved conﬁguration f n +1 in a least-squares sense: (Quarteroni et al., 2014)
Revised: We project back onto the reduced manifold by finding the corresponding input q n +1 that best matches the evolved configuration f n +1 in a least-squares sense: (Quarteroni et al., 2014)

Original: [Equation]
Revised: [Equation]

Original: The objective is similar to the training loss found in Equation (3), but with two dimensions signiﬁcantly reduced: the dimension of the unknown q n +1 , and the summation bound |M| .
Revised: The objective is similar to the training loss found in Equation (3), but with two dimensions significantly reduced: the dimension of the unknown q n +1 , and the summation bound |M| .

Original: Consequently, instead of using a stochastic gradient descent type of method, such as the auto-decoder scheme by Park et al.
Revised: Consequently, instead of using a stochastic gradient descent type of method, such as the auto-decoder scheme by Park et al.

Original: (2019), we achieve rapid inversion using the Gauss-Newton algorithm (Nocedal & Wright, 2006) with conditionally quadratic convergence.
Revised: (2019), we achieve rapid inversion using the Gauss-Newton algorithm (Nocedal & Wright, 2006) with conditionally quadratic convergence.

Original: Further details are listed in Appendix C.
Revised: Further details are listed in Appendix C.

Original: [Figure]
Revised: [Figure]

Original: 4. S PATIAL S AMPLE R EDUCTION
Revised: 4. S PATIAL S AMPLE R EDUCTION

Original: The least squares formulation from Equation (7) iswell-posed if r ≤ d |M| .
Revised: A necessary condition for the well-posedness of the least squares formulation from Equation (7) is r ≤ d |M| .

Original: Since the lowdimensional manifold construction guarantees that r (cid:28) Pd , we choose rd ≤ |M| (cid:28) P .
Revised: Since the manifold-parameterization function construction guarantees that r ≪ Pd , we choose rd ≤ |M| ≪ P .

Original: To obtain the next-time step f ( y , t n +1 ) , ∀ y ∈ M necessary for the least squares solves, we only require PDE updates (Section 4.2) and spatiotemporal data (Section 4.1) at these |M| samples.
Revised: To obtain the next-time step f ( y , t n +1 ) , ∀ y ∈ M necessary for the least squares solves, we only require PDE updates (Section 4.2) and spatiotemporal data (Section 4.1) at these |M| samples.

Original: As such, the entire latent space dynamics framework (Figure 3) requires only |M| samples, compared to the full-order solver’s P samples.
Revised: As such, the entire latent space dynamics framework (Figure 3) requires only |M| samples, compared to the full-order solver’s P samples.

Original: Hyper-reduction approaches like this have captured a wide range of real-world scenarios, including massive elasticity deformations (Fulton et al., 2019) and large turbulent ﬂows (Grimberg et al., 2021).
Revised: Hyper-reduction approaches like this have captured a wide range of real-world scenarios, including massive elasticity deformations (Fulton et al., 2019) and large turbulent flows (Grimberg et al., 2021).

Original: Unlike our discretization-independent approach, prior methods only support hyper-reduction samples that coincide with the full-order discretization.
Revised: Unlike our discretization-independent approach, prior methods only support hyper-reduction samples that coincide with the full-order discretization.

Original: A naive selection of the integration samples can lead to inaccurate latent space dynamics, even if |M| ≥ rd ; refer to Figure 4 for the failure case of uniform sampling.
Revised: A naive selection of the integration samples can lead to inaccurate latent space dynamics, even if |M| ≥ rd ; refer to Figure 4 for the failure case of uniform sampling.

Original: As noted in the hyper-reduction literature, stochastic sampling can eliminate such errors (Carlberg, 2011).
Revised: As noted in the hyper-reduction literature, stochastic sampling can eliminate such errors (Carlberg, 2011).

Original: To better control hyperreduction error, we draw inspirations from the cubature approach by An et al.
Revised: To better control hyperreduction error, we draw inspirations from the cubature approach by An et al.

Original: and propose a greedy algorithm that augments the sample set to meet a target residual (see Appendix B).
Revised: and propose a greedy algorithm that augments the sample set to meet a target residual (see Appendix B).

Revised: This strategy iteratively adds spatial samples to the sampling set until the target residual is met.

Revised: In this case, the user decides the target accuracy, not the sample set size |M| .

Revised: Alternatively, the user can terminate the algorithm early once it reaches a chosen sample set size |M| .

Original: Results of our sampling approach are shown in Figures 4, 8 and 20.
Revised: Results of our sampling approach are shown in Figures 4, 8 and 20.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We analyze the proposed framework on classic PDEs, with training data produced using a variety of discretizations (voxel grids, meshes, and point clouds).
Revised: We analyze the proposed framework on classic PDEs, with training data produced using a variety of discretizations (voxel grids, meshes, and point clouds).

Original: Unless otherwise noted, for each PDE, we delineate a testing set where D test ⊂ D with D train ∩ D test = ∅ .
Revised: Unless otherwise noted, for each PDE, we delineate a testing set where D test ⊂ D with D train ∩ D test = ∅ .

Original: We construct the manifold (Section 3) with data from D train , and then validate the latent space dynamics (Section 4) on D test .
Revised: We construct the manifold (Section 3) with data from D train , and then validate the latent space dynamics (Section 4) on D test .

Original: We compare our approach with prior discretization-dependent ROM methods, including POD (Berkooz et al., 1993; Holmes et al., 2012) and neural-network-based autoencoder approaches (Fulton et al., 2019; Lee & Carlberg, 2020;
Revised: We compare our approach with prior discretization-dependent ROM methods, including POD (Berkooz et al., 1993; Holmes et al., 2012) and neural-network-based autoencoder approaches (Fulton et al., 2019; Lee & Carlberg, 2020;

Original: Shen et al., 2021).
Revised: Shen et al., 2021), under the same latent space dimensions.

Original: Additional implementation and reproducibility details are listed from Appendix G to Appendix L. Experiment statistics are summarized in Table 1.
Revised: Additional implementation and reproducibility details are listed from Appendix G to Appendix L. Experiment statistics are summarized in Table 1.

Original: The temporal evolutions of the PDEs are best illustrated via the supplementary video .
Revised: The temporal evolutions of the PDEs are best illustrated via the supplementary video .

Original: Thermodynamics , ∂u∂t − ν ( x ) ∂ 2 u ∂x 2 = 0 .
Revised: Thermodynamics , ∂u∂t − ν ( x ) ∂ 2 u ∂x 2 = 0 .

Original: Temperature u is governed by a one-dimensional heat equation.
Revised: Temperature u is governed by a one-dimensional heat equation.

Original: ν describes the spatially-varying diffusion speed.
Revised: ν describes the spatially-varying diffusion speed.

Original: Figure 4 displays our approach’s ability to use very few integration samples and to capture conservation of energy.
Revised: Figure 4 displays our approach’s ability to use very few integration samples and to capture conservation of energy.

Original: Image Processing , ∂u∂t − ν ( x ) ∇ 2 u = 0 .
Revised: Image Processing , ∂u∂t − ν ( x ) ∇ 2 u = 0 .

Original: We model image blurring with the 2D diffusion equation (Perona & Malik, 1990).
Revised: We model image blurring with the 2D diffusion equation (Perona & Malik, 1990).

Original: Figure 5 shows that under the same latent space dimension ( r = 3 ), our method is more accurate than POD (Berkooz et al., 1993; Holmes et al., 2012), both visually and quantitatively.
Revised: Figure 5 shows that under the same latent space dimension ( r = 3 ), our method is more accurate than POD (Berkooz et al., 1993; Holmes et al., 2012), both visually and quantitatively.

Original: With itsmanifold architecture independent of pixel count, CROM uses an order of magnitude less memory than POD (see Figure 5c). 
Revised: With its architecture independent of pixel count, CROM uses an order of magnitude less memory than POD (see Figure 5c). [Figure] [Figure]

Original: Transport dominated systems .
Revised: Transport dominated systems .

Original: Next, we examine two transport-dominated slowly decaying Kolmogorov n-widths problems, the Advection Equation and Burgers’ Equation, where classic model reduction techniques often struggle (Peherstorfer, 2022):[Figure] [Figure]
Revised: Next, we examine two transport-dominated slowly decaying Kolmogorov n-widths problems, the Advection Equation and Burgers’ Equation, where classic model reduction techniques often struggle (Peherstorfer, 2022):

Original: Advection Equation , ∂u∂t + ( a · ∇ ) u = 0 .
Revised: Advection Equation , ∂u∂t + ( a · ∇ ) u = 0 .

Original: Here u is the advected quantity and a is the advection velocity.
Revised: Here u is the advected quantity and a is the advection velocity.

Original: Figure 6a depicts CROM’s favorable trajectory tracking relative to both POD and the convolutional autoencoder (CAE) of Lee & Carlberg (2020), for equal latent space dimension (also see Figure 21).
Revised: Figure 6a depicts CROM’s favorable trajectory tracking relative to both POD and the convolutional autoencoder (CAE) of Lee & Carlberg (2020), for equal latent space dimension (also see Figure 21).

Original: Burgers’ Equation , 
Revised: Burgers’ Equation , ∂w∂t + ∂ 0 .

Original: [Equation]
Revised: 5 w 2 ∂x = 0 .

Original:  Figure 6b shows that CROM more accurately captures the nonlinear dynamics than both POD and CAE (also see Figure 22).
Revised: 02 e µ D x . Figure 6b shows that CROM more accurately captures the nonlinear dynamics than both POD and CAE (also see Figure 22).

Original: Additionally, CROM also uses 12 × less memory than CAE (see Figure 26).
Revised: Additionally, CROM also uses 12 × less memory than CAE (see Figure 27).

Original: These accuracy and memory advantages are consistent with the implicit neural representation literature (Chen & Zhang, 2019).
Revised: These accuracy and memory advantages are consistent with the implicit neural representation literature (Chen & Zhang, 2019).

Original: Whereas CAE is applicable only to voxel grid data, CROM is applicable to data from any discretization , as highlighted by these tetrahedral mesh and point cloud data examples.
Revised: Whereas CAE is applicable only to voxel grid data, CROM is applicable to data from any discretization , as highlighted by these tetrahedral mesh and point cloud data examples.

Original: Solid Mechanics , ρ 0 ¨ φ = ∇ · P ( ∇ φ ) + ρ 0 B .
Revised: Solid Mechanics , ρ 0 ¨ ϕ = ∇ · P ( ∇ ϕ ) + ρ 0 B .

Original: We solve the second-order elastodynamics equation for the deformation map φ of soft bodies, where ρ 0 is the initial density, P is the ﬁrst Piola–Kirchhoff stress, and B is the body force density (see Figures 7 to 10).
Revised: We solve the second-order elastodynamics equation for the deformation map ϕ of soft bodies, where ρ 0 is the initial density, P is the first Piola–Kirchhoff stress, and B is the body force density (see Figures 7 to 10).

Original: Adaptive resolution.
Revised: Adaptive resolution.

Original: Prior ROM methods (see Figure 1a) ﬁx the discretization at onset, precluding the dynamic adaptivity approaches that beneﬁt problems with evolving complexity.
Revised: Prior ROM methods (see Figure 1a) fix the discretization at onset, precluding the dynamic adaptivity approaches that benefit problems with evolving complexity.

Original: When modeling a falling block (see Figure 7), prior ROM methods must begin with a high-resolution mesh, despite that the high-resolution is required only during contact (Figure 7a and b).
Revised: When modeling a falling block (see Figure 7), prior ROM methods must begin with a high-resolution mesh, despite that the high-resolution is required only during contact (Figure 7a and b).

Original: Our discretization-independent representation allows the time integrator to freely adapt the spatial discretization throughout the dynamic evolution.
Revised: Our discretization-independent representation allows the time integrator to freely adapt the spatial discretization throughout the dynamic evolution.

Original: For instance, we can employ a coarse mesh ( P = 8 vertices) to economize computation during the rigid falling phase (Figure 7c), and a high-resolution mesh ( P = 2 , 065 ) to capture details during contact (Figure 7c), yielding 35 × –48 × wall-clock speedups over ﬁxed discretization ROM methods.
Revised: For instance, we can employ a coarse mesh ( P = 8 vertices) to economize computation during the rigid falling phase (Figure 7c), and a high-resolution mesh ( P = 2 , 065 ) to capture details during contact (Figure 7c), yielding 35 × –48 × wall-clock speedups over fixed discretization ROM methods.

Original: To ensure generalizability of the comparison, the timing is done strictly for the PDE time-stepping stage (Step 2).
Revised: To ensure generalizability of the comparison, the timing is done strictly [Figure] [Figure]

Original:  Prior ROM architectures (i.e, both POD and autoencoder approaches) share the same PDE time-stepping routine.
Revised: [Figure] for the PDE time-stepping stage (Step 2). Prior ROM architectures (i e, both POD and autoencoder approaches) share the same PDE time-stepping routine.

Original: Memory footprint.
Revised: Memory footprint.

Original: Most prior ROM approaches are discretization-dependent.
Revised: Most prior ROM approaches are discretization-dependent.

Original: As discretization resolution ( P ) increases, so does memory consumption for g .By contrast, our discretization- [Figure] [Figure]
Revised: As discretization resolution ( P ) increases, so does memory consumption for g .

Original: [Figure]

Original: [Figure]

Original: y (

Original: N

Original: [Figure]

Original: o f

Original: [Figure]

Original: r

Original: [Figure]

Original: e t e r s ) [Figure] [Figure] independent architecture does not “see” the parameter P and in practice memory footprint does not scale with P .
Revised: By contrast, our discretizationindependent architecture does not “see” the parameter P and in practice memory footprint does not scale with P .

Original: Rather, as with all neural ﬁelds (Xie et al., 2021), memory grows with the intrinsic complexity of the presented training data.
Revised: Rather, as with all neural fields (Xie et al., 2021), memory grows with the intrinsic complexity of the presented training data.

Original: Figure 9 compares memory footprint and reconstruction error for CROM, POD (Barbiˇc, 2012), and neural-network-based autoencoder approaches (Fulton et al., 2019; Shen et al., 2021), as a function of latent space dimension; Figure 8 depicts the corresponding large deformation simulations.
Revised: Figure 9 compares memory footprint and reconstruction error for CROM, POD (Barbi ˇ c, 2012), and neural-network-based autoencoder approaches (Fulton et al., 2019; Shen et al., 2021), as a function of latent space dimension; Figure 8 depicts the corresponding large deformation simulations.

Original: Figurecompares CROM’s memory and accuracy on another large deformation example.
Revised: Figurecompares CROM’s memory and accuracy on another large deformation example.

Original: For these examples,
Revised: For these examples,

Original: CROM offers simultaneously lower memory consumption and reconstruction error.
Revised: CROM offers simultaneously lower memory consumption and reconstruction error.

Original: Furthermore, due to hyper-reduction(from P = 66 , 608 to |M| = 40 ), CROM also obtains a signiﬁcant wall-clock speedup on CPUs, consumer GPUs, and data center GPUs over the unreduced model (see Figure 8). 
Revised: Furthermore, due to hyper-reduction(from P = 66 , 608 to |M| = 40 ), CROM also obtains a significant wall-clock speedup on CPUs, consumer GPUs, and data center GPUs over the unreduced model (see Figure 8). R e c on s t r u c t i on E rr o r ( m ean s qua r ed ) Latent Space Dimension r CROM (Ours)

Original: Different discretizations, same architecture.
Revised: POD [Fulton et al. 2019] [Shen et al. 2021] 1.00E+ 1.00E+ 1.00E+ 1.00E+ 1.00E+ 1.00E+ M e m o r y N u m be r o f P a r a m e t e r s )

Original:  Because CROM is not constructed for a ﬁxed discretization, we can adopt an identical low-dimensional manifold architecture to model-reduce simulations of both tetrahedral meshes and point clouds (see Figure 8g).
Revised: [Figure] [Figure] Different discretizations, same architecture. Because CROM is not constructed for a fixed discretization, we can adopt an identical manifold-parameterization function architecture to modelreduce simulations of both tetrahedral meshes and point clouds (see Figure 8g).

Original: Moreover, even though these examples (Figures 7 and 8) leverage completely different meshes, they use the same network architecture.
Revised: Moreover, even though these examples (Figures 7 and 8) leverage completely different meshes, they use the same network architecture.

Original: This opens the door for future work on transfer learning (Weiss et al., 2016) and weight sharing among various discretizations and PDEs.
Revised: This opens the door for future work on transfer learning (Weiss et al., 2016) and weight sharing among various discretizations and PDEs.

Original: Additional comparisons between CROM, previous ROM, and the full-order models are discussed in
Revised: Additional comparisons between CROM, previous ROM, and the full-order models are discussed in

Original: Appendices M and N.
Revised: Appendices M and N.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: CROM is a model reduction framework for PDEs featuring a discretization-independent architecture.
Revised: CROM is a model reduction framework for PDEs featuring a discretization-independent architecture.

Original: CROM disentangles the low-dimensional manifold from the discretization by reformulating the reduced manifold as a map accepting not only q but also x as an input.
Revised: CROM disentangles the low-dimensional embedding from the discretization by reformulating the reduced manifold as a map accepting not only q but also x as an input.

Original: CROM outperforms discretization-dependent ROM approaches, such as POD and autoencoders, in terms of accuracy and memory consumption for equal latent space dimension, and in the ability to dynamically adapt the discretization during time integration.
Revised: CROM outperforms discretization-dependent ROM approaches, such as POD and autoencoders, in terms of accuracy and memory consumption for equal latent space dimension, and in the ability to dynamically adapt the discretization during time integration.

Original: While offering key advantages over prior ROM methods, CROM also inherits a limitation of ROM.
Revised: While offering key advantages over prior ROM methods, CROM also inherits a limitation of ROM.

Original: It can only treat PDE solutions in the space spanned by the low-dimensional manifold (determined by the training data) and does not generalize to arbitrary unseen scenarios.
Revised: It can only treat PDE solutions in the space spanned by the manifold-parameterization function (determined by the training data) and does not generalize to arbitrary unseen scenarios.

Original: Such a limitation is also commonly found among other implicit neural representation works (Xie et al., 2021).
Revised: Such a limitation is also commonly found among other implicit neural representation works (Xie et al., 2021).

Original: Future research may consider improving generalizability and data efﬁciency via meta-learning and integration of stronger priors (Sitzmann et al., 2020a).
Revised: Future research may consider improving generalizability and data efficiency via meta-learning and integration of stronger priors (Sitzmann et al., 2020a).

Original: Compared to end-to-end ML solutions to PDEs (Sanchez-Gonzalez et al., 2020), CROM employs the neural network strictly as a spatial representation (see Sections 4.1 and 4.3) for the kinematics and solves the PDE using classical PDE-integration numerical methods (see Section 4.2).
Revised: Compared to end-to-end ML solutions to PDEs (Sanchez-Gonzalez et al., 2020), CROM employs the neural network strictly as a spatial representation (see Sections 4.1 and 4.3) for the kinematics and solves the PDE using classical PDE-integration numerical methods (see Section 4.2).

Original: As such, we believe CROM will open doors for more forthcoming hybrid ML-PDE solutions.
Revised: As such, we believe CROM will open doors for more forthcoming hybrid ML-PDE solutions.

Original: As shown in our work, these solutions can retain the PDE’s physical invariants (see Figure 4), allow for easy integration with existing PDE solvers (see Section 4.2), and obtain practical computational savings that can be directly employed in production (see Figure 8).
Revised: As shown in our work, these solutions can retain the PDE’s physical invariants (see Figure 4), allow for easy integration with existing PDE solvers (see Section 4.2), and obtain practical computational savings that can be directly employed in production (see Figure 8).

Original: Appendix
Revised: Appendix

Original: We highly encourage viewing the supplementary video where our results are best illustrated.
Revised: We highly encourage viewing the supplementary video where our results are best illustrated.

Original: A A DDITIONAL L ITERATURE R EVIEW A. R EDUCED -O RDER M ODELING
Revised: A A DDITIONAL L ITERATURE R EVIEW A. R EDUCED -O RDER M ODELING

Original: Accelerating PDE solutions via reduced-order modeling (ROM) has a rich history.
Revised: Accelerating PDE solutions via reduced-order modeling (ROM) has a rich history.

Original: Two common building blocks of ROM are (1) identifying a low-dimensional latent space of the original complex system and (2) solving PDEs via latent space dynamics, i.e., evolving the latent space vector over time.
Revised: Two common building blocks of ROM are (1) identifying a low-dimensional latent space of the original complex system and (2) solving PDEs via latent space dynamics, i.e., evolving the latent space vector over time.

Original: Early works on latent space construction started with linear methods where the the latent space and the full space has a linear relationship.
Revised: Early works on latent space construction started with linear methods where the the latent space and the full space has a linear relationship.

Original: Notable works include proper orthogonal decomposition (Berkooz et al., 1993;
Revised: Notable works include proper orthogonal decomposition (Berkooz et al., 1993;

Original: Holmes et al., 2012; Barbi ˇ c & James, 2005; Barbi ˇ c, 2012), the reduced-basis technique (Prud’homme et al., 2002; Rozza et al., 2007), balanced truncation (Moore, 1981), rational interpolation (Baur et al., 2011; Gugercin et al., 2008), and Craig–Bampton model reduction (Craig Jr & Bampton, 1968).
Revised: Holmes et al., 2012; Barbi ˇ c & James, 2005; Barbi ˇ c, 2012), the reduced-basis technique (Prud’homme et al., 2002; Rozza et al., 2007), balanced truncation (Moore, 1981), rational interpolation (Baur et al., 2011; Gugercin et al., 2008), and Craig–Bampton model reduction (Craig Jr & Bampton, 1968).

Original: Recently, nonlinear methods (Hartman & Mestha, 2017; Kashima, 2016; Gu, 2011; Erichson et al., 2019; Maulik et al., 2020; Regazzoni et al., 2019; Fulton et al., 2019; Maulik et al., 2021; Romero et al., 2021;
Revised: Recently, nonlinear methods (Hartman & Mestha, 2017; Kashima, 2016; Gu, 2011; Erichson et al., 2019; Maulik et al., 2020; Regazzoni et al., 2019; Fulton et al., 2019; Maulik et al., 2021; Romero et al., 2021;

Original: Shen et al., 2021), often constructed via neural networks, have gained signiﬁcant attentions due to their ability to more accurately construct the low-dimensional manifold.
Revised: Shen et al., 2021), often constructed via neural networks, have gained significant attentions due to their ability to more accurately construct the manifold-parameterization function.

Original: In particular, nonlinear manifolds signiﬁcantly outperform their linear counterparts on slowly decaying Kolmogorov n-width problems (e.g., advection) (Peherstorfer, 2022; Ohlberger &
Revised: In particular, nonlinear manifolds significantly outperform their linear counterparts on slowly decaying Kolmogorov n-width problems (e.g., advection) (Peherstorfer, 2022; Ohlberger &

Original: Rave, 2013; Peherstorfer & Willcox, 2015; Taddei et al., 2015; Peherstorfer, 2020; Ehrlacher et al., 2020; Lee & Carlberg, 2021).
Revised: Rave, 2013; Peherstorfer & Willcox, 2015; Taddei et al., 2015; Peherstorfer, 2020; Ehrlacher et al., 2020; Lee & Carlberg, 2021).

Original: Most of these prior works exclusively focus on building a latent space for the already-dicretized vector ﬁelds of interest.
Revised: Most of these prior works exclusively focus on building a latent space for the already-dicretized vector fields of interest.

Original: Chen et al.
Revised: Chen et al.

Original: (2021); Pan et al.
Revised: (2021); Pan et al.

Original: and our work are the ﬁrst attempts at constructing the latent space for the continuous vectors themselves, thereby allowing the training data to come from any discretizations.
Revised: and our work are the first attempts at constructing the latent space for the continuous vectors themselves, thereby allowing the training data to come from any discretizations.

Original: After the latent space is identiﬁed, solving PDEs with parameters unseen during training requires evolving the latent space vector over time.
Revised: After the latent space is identified, solving PDEs with parameters unseen during training requires evolving the latent space vector over time.

Original: Two families of approaches have been proposed for latent space dynamics.
Revised: Two families of approaches have been proposed for latent space dynamics.

Original: The ﬁrst family of approaches (Kim et al., 2019) computes latent space dynamics via a data-driven operator that is learned from the training data.
Revised: The first family of approaches (Kim et al., 2019) computes latent space dynamics via a data-driven operator that is learned from the training data.

Original: A key beneﬁt from learning the operator from data is that the user requires no knowledge of the system’s governing equations and does not need access to the training solver’s source code.
Revised: A key benefit from learning the operator from data is that the user requires no knowledge of the system’s governing equations and does not need access to the training solver’s source code.

Original: However, the downside of this approach is that additional assumptions (e.g., the availability of a Koopman invariant subspace as in Lusch et al.
Revised: However, the downside of this approach is that additional assumptions (e.g., the availability of a Koopman invariant subspace as in Lusch et al.

Original: (2018) and the existence of a ﬁnite-dimensional quadratic representation as it is in the lift and learn approach by Qian et al.
Revised: (2018) and the existence of a finite-dimensional quadratic representation as it is in the lift and learn approach by Qian et al.

Original: (2020)) are made.
Revised: (2020)) are made.

Original: Without these assumptions, the learned operator is prone to violation of physical laws.
Revised: Without these assumptions, the learned operator is prone to violation of physical laws.

Original: The second family of approaches (Lee & Carlberg, 2020;
Revised: The second family of approaches (Lee & Carlberg, 2020;

Original: Fulton et al., 2019), to which our method belongs, computes latent space dynamics according to the
Revised: Fulton et al., 2019), to which our method belongs, computes latent space dynamics according to the

Original: PDE used for training data generation.
Revised: PDE used for training data generation.

Original: The advantage of this approach is that it does not require learning an additional time-evolution operator.
Revised: The advantage of this approach is that it does not require learning an additional time-evolution operator.

Original: It works with arbitrary dynamical systems with known
Revised: It works with arbitrary dynamical systems with known

Original: PDE governing equations and conserves physical laws by construction since explicit PDE model is employed.
Revised: PDE governing equations and conserves physical laws by construction since explicit PDE model is employed.

Original: However, the disadvantage is that the PDE model must be known.
Revised: However, the disadvantage is that the PDE model must be known.

Original: B R OBUST S AMPLING
Revised: B R OBUST S AMPLING

Original: To select the integration samples robustly, we aim to balance the computation speed and accuracy.
Revised: To select the integration samples robustly, we aim to balance the computation speed and accuracy.

Original: CROM employs |M| spatial samples, and the computation speed roughly scales linearly with the number of samples.
Revised: CROM employs |M| spatial samples, and the computation speed roughly scales linearly with the number of samples.

Original: To maximize computation speed, we aspire to select the fewest number of samples possible.
Revised: To maximize computation speed, we aspire to select the fewest number of samples possible.

Original: Given a target accuracy, our greedy algorithm selects the fewest number of integration samples from the discrete samples { x i } Pi =1 of the full-order PDE solution in order to achieve the target accuracy.
Revised: Given a target accuracy, our greedy algorithm selects the fewest number of integration samples from the discrete samples { x i } Pi =1 of the full-order PDE solution in order to achieve the target accuracy.

Original: P  do
Revised: P } do

Original: _ vec
Revised: _ vec

Original: return M else 
Revised: return M else indices =

Revised: Max _ Indices res vec

Original: / ﬁnd Q the residuals res _ list = 0 ∈ R Q for i in indices do res  list 
Revised: Q // find Q the individual res _ list = 0 ∈ R Q for i in indices do res _ list i ] = Metric

Revised: Calculate

Original: = ) end 
Revised: ) end j =

Original: Min 
Revised: Min _ Indices ( list

Original: / ﬁnd
Revised: 1) // find the index with the smallest global residual

Original: M 
Revised: M = M

Original:  end 15 end  do
Revised: } end 15 end 16 Function Calculate : res ∈ R P for µ ∈ D train do

Original: Obtain ﬁnal latent space vector q  t T  µ )  the integration samples 
Revised: Obtain the final latent space vector q ( t T ; µ ) with the integration samples ( M ) for i = 1 , . .

Original: M do 
Revised: , P do res

Original: end end return 
Revised: end end return res _

Original:  In every iteration, the greedy algorithm adds one spatial sample to the sample set M and lowers the error of latent space dynamics.
Revised: 25 Function Metric In every iteration, the greedy algorithm adds one spatial sample to the sample set M and lowers the error of latent space dynamics.

Original: Speciﬁcally, the algorithm loops over Q spatial samples with the largest individual residuals.
Revised: Specifically, the algorithm loops over Q spatial samples with the largest individual residuals.

Original: We test adding each sample to the integration sample set and compute a new global residual.
Revised: We test adding each sample to the integration sample set and compute a new global residual.

Original: From these Q samples, we select the one leading to the smallest new global residual and add it to the actual sample set.
Revised: From these Q samples, we select the one leading to the smallest new global residual and add it to the actual sample set.

Original: The algorithm repeats until the target accuracy is met.
Revised: The algorithm repeats until the target accuracy is met.

Original: In practice, we ﬁnd Q = 10 gives sufﬁcient results.
Revised: In practice, we find Q = 10 gives sufficient results.

Original: Algorithm 1 describes the details.
Revised: Algorithm 1 describes the details.

Original: Compared to the naive uniform sampling approach, our robust sampling scheme yields signiﬁcantly more accurate results (see Figure 4).
Revised: Compared to the naive uniform sampling approach, our robust sampling scheme yields significantly more accurate results (see Figure 4).

Original: We also highlight that Algorithm 1 happens before any simulation occurs (latent space dynamics).
Revised: We also highlight that Algorithm 1 happens before any simulation occurs (latent space dynamics).

Original: It is strictly a pre-computation.
Revised: It is strictly a pre-computation.

Original: C N ETWORK I NVERSION
Revised: C N ETWORK I NVERSION

Original: In order to generate latent space dynamics, we need to invert the neural network employed for the low-dimensional manifold.
Revised: In order to generate latent space dynamics, we need to invert the neural network employed for the manifold-parameterization function.

Original: We do so by solving the minimization problem:
Revised: We do so by solving the minimization problem:

Original: [Equation]
Revised: [Equation]

Original: Since this is a nonlinear least-squares problem, we solve it with the classic iterative Gauss-Newton algorithm (Nocedal & Wright, 2006).
Revised: Since this is a nonlinear least-squares problem, we solve it with the classic iterative Gauss-Newton algorithm (Nocedal & Wright, 2006).

Original: Unlike Newton’s method, Gauss-Newton does not require the
Revised: Unlike Newton’s method, Gauss-Newton does not require the

Original: Hessian of the objective function.
Revised: Hessian of the objective function.

Original: Consequently, in addition to the neural network itself, we only need to evaluate the gradient of the neural network with respect to the latent space vector ∂ g θg ∂ q ( y , q n +1 ) .
Revised: Consequently, in addition to the neural network itself, we only need to evaluate the gradient of the neural network with respect to the latent space vector ∂ g θg ∂ q ( y , q n +1 ) .

Original: We also equip our Gauss-Newton implementation with a standard backtracking line-search scheme to improve the convergence rate.
Revised: We also equip our Gauss-Newton implementation with a standard backtracking line-search scheme to improve the convergence rate.

Original: C. I NITIAL G UESS FOR THE N ONLINEAR S OLVER
Revised: C. I NITIAL G UESS FOR THE N ONLINEAR S OLVER

Original: An initial guess close to the minimum is necessary for the convergence of Gauss-Newton.
Revised: An initial guess close to the minimum is necessary for the convergence of Gauss-Newton.

Original: The previous time step latent space vector q n serves as an ideal initial guess since f does not vary much over a time step.
Revised: The previous time step latent space vector q n serves as an ideal initial guess since f does not vary much over a time step.

Original: However, in the extreme cases where f deviates signiﬁcantly over one time step, we ﬁnd it helpful to compute the initial guess through an encoder network.
Revised: However, in the extreme cases where f deviates significantly over one time step, we find it helpful to compute the initial guess through an encoder network.

Original: For convenience, we employ the encoder network from training, which requires computing the dynamics for P samples.
Revised: For convenience, we employ the encoder network from training, which requires computing the dynamics for P samples.

Original: Since these extreme cases are rare (e.g., the ﬁrst two time steps of an operator-splitting-based ﬂuid solver where a large change in pressure is necessary for enforcing incompressibility), the computation overhead over a fully |M| treatment is minimal.
Revised: Since these extreme cases are rare (e.g., the first two time steps of an operator-splitting-based fluid solver where a large change in pressure is necessary for enforcing incompressibility), the computation overhead over a fully |M| treatment is minimal.

Original: Future work should consider building this encoder network strictly for the degrees of freedom in M .
Revised: Future work should consider building this encoder network strictly for the degrees of freedom in M .

Original: We can further facilitate nonlinear solver convergence by improving the latent space quality, such as employing a Jacobian penalty term (Chen et al., 2021) / local isometry term (Du et al., 2021) and enforcing Lipschitz constraints (Liu et al., 2022).
Revised: We can further facilitate nonlinear solver convergence by encouraging smoothness of the latent space, such as employing a Jacobian penalty term (Chen et al., 2021) / local isometry term (Du et al., 2021) and enforcing Lipschitz constraints (Liu et al., 2022).

Revised: These smoothness regularizations would be particularly important for challenging PDE cases involving bifurcations and buckling (Brush et al., 1975).

Original: C. L INEARIZATION
Revised: C. L INEARIZATION

Original: We can further obtain computation-save by bypassing the iterative nonlinear solver and linearizing the least-squares problem.
Revised: We can further obtain computation-save by bypassing the iterative nonlinear solver and linearizing the least-squares problem.

Original: We deﬁne ∆ f as,∆ f ( y , t n +1 ) = f ( y , t n [Equation] [Equation]
Revised: We define ∆ f as,

Original: − f ( y , t n ) =

Original: f ( y

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation]
Revised: [Equation]

Original: − g

Original: [Equation]

Original: g ( y , q n ) .

Original: Employing Taylor expansion of g θ g at q n , we have g θ g
Revised: Employing Taylor expansion of g θ g at q n , we have g θ g

Original: ( y , q n +1 )
Revised: ( y , q n +1 )

Original: ≈ g θ g
Revised: ≈ g [Equation]

Original:  ( y , q n ) + ∂ g θg ∂ q ( y , q n )∆ q n +1 , where q n +1 = q n + ∆ q n +1 .
Revised: g ( y , q n ) + ∂ g θg ∂ q ( y , q n )∆ q n +1 , where q n +1 = q n + ∆ q n +1 .

Original: Therefore, the original objective function can be approximated as,
Revised: Therefore, the original objective function can be approximated as,

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can obtain the latent space dynamics by solving the linear least-squares problem,
Revised: Therefore, we can obtain the latent space dynamics by solving the linear least-squares problem,

Original: [Equation]
Revised: [Equation]

Original: This is the normal equation and can be solved in the closed form:
Revised: This is the normal equation and can be solved in the closed form:

Original: [Equation]
Revised: [Equation]

Original: where J is the ( |M| · d ) by r Jacobian matrix that contains ∂ g θg ∂ q ( y , q n ) , ∀ y ∈ M ; b is the ( |M| · d ) by 1 residual vector containing ∆ f ( y , t n +1 ) , ∀ y ∈ M .
Revised: where J is the ( |M| · d ) by r Jacobian matrix that contains ∂ g θg ∂ q ( y , q n ) , ∀ y ∈ M ; b is the ( |M| · d ) by 1 residual vector containing ∆ f ( y , t n +1 ) , ∀ y ∈ M .

Original: Since our work features both small |M| and small r , the closed-form network inversion costs just a few small dense matrix multiplications and thereby introduces minimal overhead.
Revised: Since our work features both small |M| and small r , the closed-form network inversion costs just a few small dense matrix multiplications and thereby introduces minimal overhead.

Original: In fact, the solver itself is extremely efﬁcient, and the majority of the computation cost is forming the Jacobian matrix
Revised: In fact, the solver itself is extremely efficient, and the majority of the computation cost is forming the Jacobian matrix

Original: J itself, whose optimization will be discussed in Appendix E.1.1.
Revised: J itself, whose optimization will be discussed in Appendix E.1.1.

Original: Additional performance details are listed in Appendix N.
Revised: Additional performance details are listed in Appendix N.

Original: Although our network inversion step follows the standard optimal-projection-based ROM literature (Carlberg et al., 2013), future work may consider directly learning an inverse during training time.
Revised: Although our network inversion step follows the standard optimal-projection-based ROM literature (Carlberg et al., 2013), future work may consider directly learning an inverse during training time.

Original: The encoder introduced in the manifold construction process may be a step towards learning an inverse.
Revised: The encoder introduced in the manifold construction process may be a step towards learning an inverse.

Original: However, the encoder in its current form does not have optimal accuracy and is bounded to a particular discretization.
Revised: However, the encoder in its current form does not have optimal accuracy and is bounded to a particular discretization.

Original: Future work should consider lifting these limitations.
Revised: Future work should consider lifting these limitations.

Original: D N ETWORK D ETAILS
Revised: D N ETWORK D ETAILS

Original: Consistent with the implicit neural representation literature, we parameterize the low-dimensional manifold with an MLP network g θ g .
Revised: Consistent with the implicit neural representation literature, we parameterize the manifoldparameterization function manifold with an MLP network g θ g .

Original: The input dimension of the network is m + r while the output dimension of the network is d , where m is the dimension of the input spatial vector while d is the output dimension of the vector ﬁeld of interest.
Revised: The input dimension of the network is m + r while the output dimension of the network is d , where m is the dimension of the input spatial vector while d is the output dimension of the vector field of interest.

Original: Our MLP network contains 5 hidden layers, each of which has a width of ( β · d ) , where β is the hyperparameter that deﬁnes the learning capacity of the network.
Revised: Our MLP network containshidden layers, each of which has a width of ( β · d ) , where β is the hyperparameter that defines the learning capacity of the network.

Original: Essentially, our network has two tunable hyperparameters, r and β .
Revised: Essentially, our network has two tunable hyperparameters, r and β .

Original: A detailed study on these hyperparameters will be discussed in Appendix F.
Revised: A detailed study on these hyperparameters will be discussed in Appendix F.

Original: Since we require the network to be continuously differentiable with respect to both the spatial coordinates x and the latent space vector q , we adopt continuously differentiable activation functions.
Revised: Since we require the network to be continuously differentiable with respect to both the spatial coordinates x and the latent space vector q , we adopt continuously differentiable activation functions.

Original: In practice, either ELU (Clevert et al., 2015) or SIREN (Sitzmann et al., 2020b) serves this purpose.
Revised: In practice, either ELU (Clevert et al., 2015) or SIREN (Sitzmann et al., 2020b) serves this purpose.

Original: For the encoder network e θ e , the input is a 1D vector of length P with d channels.
Revised: For the encoder network e θ e , the input is a 1D vector of length P with d channels.

Original: The output is a vector of dimension r .
Revised: The output is a vector of dimension r .

Original: Using MLPs would lead to enormous network size when P is large.
Revised: Using MLPs would lead to enormous network size when P is large.

Original: We therefore design an encoder network ﬁrst to apply multiple 1D convolution layers of kernel size 6, stride size 4, and output channel size d until the output 1D vector’s length is the closest to 32 /d but no smaller.
Revised: We therefore design an encoder network first to apply multiple 1D convolution layers of kernel size 6, stride size 4, and output channel size d until the output 1D vector’s length is the closest to 32 /d but no smaller.

Original: Afterward, the encoder network reshapes the vector into 1 channel and applies an MLP layer to reduce the dimension of the vector to 32.
Revised: Afterward, the encoder network reshapes the vector into 1 channel and applies an MLP layer to reduce the dimension of the vector to 32.

Original: The last MLP layer then transforms the previous 32-dimensional vector into dimension r .
Revised: The last MLP layer then transforms the previous 32-dimensional vector into dimension r .

Original: Discretization-speciﬁc encoder networks can also be used, e.g., PointNet for point clouds and convolutional neural network for grid data.
Revised: Discretization-specific encoder networks can also be used, e.g., PointNet for point clouds and convolutional neural network for grid data.

Original: D. T RAINING D ETAILS
Revised: D. T RAINING D ETAILS

Original: We use the Adam optimizer (Kingma & Ba, 2014) for stochastic gradient descent.
Revised: We use the Adam optimizer (Kingma & Ba, 2014) for stochastic gradient descent.

Original: We use the Xavier initialization for ELU layers and the default initialization ( ω 0 = 30 ) for SIREN layers.
Revised: We use the Xavier initialization for ELU layers and the default initialization ( ω 0 = 30 ) for SIREN layers.

Original: Unless otherwise noted, we train with a base learning rate of lr = 1 e − 4 and adopt a learning rate decay strategy ( 10 · lr → 5 · lr → 2 · lr → 1 · lr → 0 . 

Revised: Unless otherwise noted, we train with a base learning rate of lr = 1 e − 4 and adopt a learning rate decay strategy ( 10 · lr → 5 · lr → 2 · lr → 1 · lr → 0 .

Original: · lr → 0 . 2 · lr ).
Revised: · lr → 0 . 2 · lr ).

Original: For each aforementioned learning rate, we train for 30 , 000 epochs.
Revised: For each aforementioned learning rate, we train for 30 , 000 epochs.

Original: We adopt a batch size 16 (i.e., 16 simulation snapshots) for the encoder and therefore a batch size of 16 · P for the implicit-neural-representation-based low-dimensional manifold.
Revised: We adopt a batch size 16 (i.e., 16 simulation snapshots) for the encoder and therefore a batch size of 16 · P for the implicit-neural-representation-based manifoldparameterization function.

Original: We implement the entire training pipeline in PyTorch Lightning(Falcon et al., 2019) which facilitates distributed training across multiple GPUs.
Revised: We implement the entire training pipeline in PyTorch Lightning(Falcon et al., 2019) which facilitates distributed training across multiple GPUs.

Original: The training vector ﬁelds are standardized to have zero mean and unit variance.
Revised: The training vector fields are standardized to have zero mean and unit variance.

Original: The training spatial coordinates are also standardized for the ELU implementation, but they are preprocessed to be between [ − 1 , 1] for the SIREN implementation.
Revised: The training spatial coordinates are also standardized for the ELU implementation, but they are preprocessed to be between [ − 1 , 1] for the SIREN implementation.

Original: E N ETWORK G RADIENTS
Revised: E N ETWORK G RADIENTS

Original: Gradients of the implicit neural representation ( ∇ x g θ g and ∂ g θg ∂ q ) are crucial for our reducedorder pipeline.
Revised: Gradients of the implicit neural representation ( ∇ x g θ g and ∂ g θg ∂ q ) are crucial for our reducedorder pipeline.

Original: We use them to compute the spatial and temporal gradients of the vector ﬁelds, ∇ f ( x , t ) =
Revised: We use them to compute the spatial and temporal gradients of the vector fields, ∇ f ( x , t ) =

Original: ∇ x g θ g ( x , q ) and ˙ f
Revised: ∇ x g θ g ( x , q ) and ˙ f

Original: ( x , t ) =
Revised: ( x , t ) =

Original: ∂
Revised: ∂

Original: g θg ∂ q ( x , q ) ˙ q .
Revised: g θg ∂ q ( x , q ) ˙ q .

Original: Furthermore, the gradient with respect to the latent space ∂ g θg ∂ q is also a key ingredient for the network inversion (Appendix C).
Revised: Furthermore, the gradient with respect to the latent space ∂ g θg ∂ q is also a key ingredient for the network inversion (Appendix C).

Original: E. N ETWORK G RADIENTS VIA D IRECT D IFFERENTIATION
Revised: E. N ETWORK G RADIENTS VIA D IRECT D IFFERENTIATION

Original: One way to compute the gradients is through directly differentiating the continuously differentiable network.
Revised: One way to compute the gradients is through directly differentiating the continuously differentiable network.

Original: [Figure]
Revised: [Figure]

Original: E
Revised: E

Revised: [Figure]

Original: E FFICIENT I MPLEMENTATION
Revised: E FFICIENT I MPLEMENTATION

Original: Input: x
Revised: Input: x

Original:  1 Let g θ g = g n ◦ · · · ◦ g 1 be a n -layer MLP network of interest.2 Initialization: y ∂ x = I = x 3 for i = 1 , . . . , n do
Revised: Output: ∂ g θg ∂ x ( x ) , g θ g ( x ) 1 Let g θ g = g n ◦ · · · ◦ g 1 be a n -layer MLP network of interest. n do

Original: Unfortunately, computing these differentiations via auto-diff (computational graph tracking) is too slow for the high-performance application explored in this work.
Revised: Unfortunately, computing these differentiations via auto-diff (computational graph tracking) is too slow for the high-performance application explored in this work.

Original: To ensure maximum efﬁciency, we implement the gradients analytically through the chain rules of each network layer, similar to the grad net approach by (Lindell et al., 2021).
Revised: To ensure maximum efficiency, we implement the gradients analytically through the chain rules of each network layer, similar to the grad net approach by (Lindell et al., 2021).

Original: Since we only use fully-connected layers, the implementation is straightforward.
Revised: Since we only use fully-connected layers, the implementation is straightforward.

Original: In addition, along the way of computing the gradient via chain rules, we also obtain the function value of the neural network itself.
Revised: In addition, along the way of computing the gradient via chain rules, we also obtain the function value of the neural network itself.

Original: Therefore, we obtain both the gradient and the function value in a single forward pass.
Revised: Therefore, we obtain both the gradient and the function value in a single forward pass.

Original: Algorithm 2 describes the details.
Revised: Algorithm 2 describes the details.

Original: E.1. F AILURE M ODES
Revised: E.1. F AILURE M ODES

Original: While direct differentiation is generally accurate, we observe failure modes when the training samples are sparse.
Revised: While direct differentiation is generally accurate, we observe failure modes when the training samples are sparse.

Original: Consider the case where g θ g ( x , q ) = x , e.g., the undeformed, reference conﬁguration of elastodynamics.
Revised: Consider the case where g θ g ( x , q ) = x , e.g., the undeformed, reference configuration of elastodynamics.

Original: In this case, the ground truth spatial gradient is ∇ x g θ g
Revised: In this case, the ground truth spatial gradient is ∇ x g θ g

Original: = I .
Revised: = I .

Original: We train the neural network with spatial samples discussed in Appendix L.4.1.
Revised: We train the neural network with spatial samples discussed in Appendix L.4.1.

Original: After training, we evaluate the spatial gradients at the center of each tetrahedron and compute its error (from the identity matrix) in the L2 norm.
Revised: After training, we evaluate the spatial gradients at the center of each tetrahedron and compute its error (from the identity matrix) in the L2 norm.

Original: As shown in Figure 11, errors are observed in both ELU and
Revised: As shown in Figure 11, errors are observed in both ELU and

Original: SIREN networks while SIREN’s error is signiﬁcantly larger, c.f.,(Yang et al., 2021).
Revised: SIREN networks while SIREN’s error is significantly larger, c.f.,(Yang et al., 2021).

Original: This discrepancy can be understood as the high-frequency prior by SIREN being more suitable for approximating
Revised: This discrepancy can be understood as the high-frequency prior by SIREN being more suitable for approximating

Original: R e c on s t r u c t i on
Revised: R e c on s t r u c t i on

Original: E rr o r
Revised: E rr o r

Original: ( m ean s qua r ed )
Revised: ( m ean s qua r ed )

Original: [Figure]
Revised: [Figure]

Original: high-frequency functions.
Revised: high-frequency functions.

Original: By contrast, the low-frequency prior by ELU is better at approximating low-frequency functions (Hertz et al., 2021).
Revised: By contrast, the low-frequency prior by ELU is better at approximating low-frequency functions (Hertz et al., 2021).

Original: We ﬁnd training with more samples and/or gradient supervision (Chen et al., 2021) reduces the errors above at the cost of additional training resources but does not resolve the issue completely.
Revised: We find training with more samples and/or gradient supervision (Chen et al., 2021) reduces the errors above at the cost of additional training resources but does not resolve the issue completely.

Original: Future work may consider more advanced gradient regularization methods (Liu et al., 2022).
Revised: Future work may consider more advanced gradient regularization methods (Liu et al., 2022).

Original: E. N ETWORK G RADIENTS VIA N UMERICAL D IFFERENTIATION
Revised: E. N ETWORK G RADIENTS VIA N UMERICAL D IFFERENTIATION

Original: Another approach to computing neural network gradients is numerical approximation, such as the ﬁnite difference method and the ﬁnite element method.
Revised: Another approach to computing neural network gradients is numerical approximation, such as the finite difference method and the finite element method.

Original: Figure 11 demonstrates the drastically improved gradient accuracy with the ﬁnite element method (linear basis function) on networks trained with SIREN.
Revised: Figure 11 demonstrates the drastically improved gradient accuracy with the finite element method (linear basis function) on networks trained with SIREN.

Original: We ﬁnd such a hybrid approach to be an ideal middle ground, i.e., we represent the function itself with the discretization-independent neural network and compute the gradient via discretization-dependent numerical differentiation.
Revised: We find such a hybrid approach to be an ideal middle ground, i.e., we represent the function itself with the discretization-independent neural network and compute the gradient via discretization-dependent numerical differentiation.

Original: With numerical gradients, users still beneﬁt from our continuous representation that allows them to use any discretization for gradient computation (e.g., ground impact elasticity deformation with remeshing).
Revised: With numerical gradients, users still benefit from our continuous representation that allows them to use any discretization for gradient computation (e.g., ground impact elasticity deformation with remeshing).

Original: Prior ROM approaches require these numerical gradients to be computed by a ﬁxed mesh.
Revised: Prior ROM approaches require these numerical gradients to be computed by a fixed mesh.

Original: F H YPERPARAMETERS
Revised: F H YPERPARAMETERS

Original: During the ofﬂine training stage, we can modify two network architecture hyperparameters: the size of the latent space ( r ) and the width of the MLP ( β ).
Revised: During the offline training stage, we can modify two network architecture hyperparameters: the size of the latent space ( r ) and the width of the MLP ( β ).

Original: After training and during the online deployment stage of the model, the key hyperparameter is the number of integration samples ( |M| ).
Revised: After training and during the online deployment stage of the model, the key hyperparameter is the number of integration samples ( |M| ).

Original: In this section, we run a sensitivity analysis on these hyperparameters.
Revised: In this section, we run a sensitivity analysis on these hyperparameters.

Original: Figure 12 shows that increasing the width of each MLP layer improves the reconstruction accuracy.
Revised: Figure 12 shows that increasing the width of each MLP layer improves the reconstruction accuracy.

Original: Unless otherwise noticed, all reconstruction errors reported are mean squared errors (MSE).
Revised: Unless otherwise noticed, all reconstruction errors reported are mean squared errors (MSE).

Original: While
Revised: While

Original: Figure 12 demonstrates an empirical convergence, future work should consider a theoretical analysis (Kunisch & Volkwein, 2002).
Revised: Figure 12 demonstrates an empirical convergence, future work should consider a theoretical analysis (Kunisch & Volkwein, 2002).

Original: [Figure]
Revised: [Figure]

Original: However, large layer widths also entail higher computational cost (see Figure 13).
Revised: However, large layer widths also entail higher computational cost (see Figure 13).

Original: Compared to the parallelized GPUs, CPUs suffer from a more severe performance drop due to the larger network size.
Revised: Compared to the parallelized GPUs, CPUs suffer from a more severe performance drop due to the larger network size.

Original: Unlike traditional PCA-based model reduction approaches (e.g., POD), increasing the dimension of the latent space vector does not yield higher accuracy for our approach (see Figure 9).
Revised: Unlike traditional PCA-based model reduction approaches (e.g., POD), increasing the dimension of the latent space vector does not yield higher accuracy for our approach (see Figure 9).

Original: In prior ROM approaches, augmenting the dimension of the latent space vector directly increases the number of entries in the low-dimensional manifold matrix, which, in turn, improves the approximation capacity of the low-dimensional manifold.
Revised: In prior ROM approaches, augmenting the dimension of the latent space vector directly increases the number of entries in the manifold-parameterization function matrix, which, in turn, improves the approximation capacity of the manifold-parameterization function.

Original: However, as shown in Figure 12, in the implicit neural representation framework, we can control the learning capacity of the network architecture by explicitly modifying the width of the MLP layer.
Revised: However, as shown in Figure 12, in the implicit neural representation framework, we can control the learning capacity of the network architecture by explicitly modifying the width of the MLP layer.

Original: Therefore, the importance of the latent space dimension diminishes.
Revised: Therefore, the importance of the latent space dimension diminishes.

Original: This experimental observation is consistent with prior work’s theoretical analysis (Remark 2.1 by Lee & Carlberg (2020)) which says the lower bound of the latent space dimension is the dimension of the problem parameters (e.g., material properties) plus one.
Revised: This experimental observation is consistent with prior work’s theoretical analysis (Remark 2.1 by Lee & Carlberg (2020)) which says the lower bound of the latent space dimension is the dimension of the problem parameters (e.g., material properties) plus one.

Original: Consequently, one should consider increasing the learning capacity of the low-dimensional manifold instead of increasing the latent space dimension.
Revised: Consequently, one should consider increasing the learning capacity of the manifold-parameterization function instead of increasing the latent space dimension.

Original: During the online stage of the reduced-order model, we select integration samples via the greedy sampling scheme (Appendix B).
Revised: During the online stage of the reduced-order model, we select integration samples via the greedy sampling scheme (Appendix B).

Original: Increasing the number of integration samples leads to higher accuracy (Figure 14).
Revised: Increasing the number of integration samples leads to higher accuracy (Figure 14).

Original: G T HERMODYNAMICS G 
Revised: G T HERMODYNAMICS G.

Original: [Equation]
Revised: [Equation]

Original: In thermodynamics, we study the 1D heat equation of the spatiotemporal dependent temperature u .
Revised: In thermodynamics, we study the 1D heat equation of the spatiotemporal dependent temperature u .

Original: We assume a zero-Dirichlet boundary condition, though other boundary conditions can also be incorporated.
Revised: We assume a zero-Dirichlet boundary condition, though other boundary conditions can also be incorporated.

Original: G. F ULL - ORDER M ODEL
Revised: G. F ULL - ORDER M ODEL

Original: The full-order model discretizes the spatial vector ﬁeld with a regular Eulerian grid using 501 equally spaced samples ( P = 501 ).
Revised: The full-order model discretizes the spatial vector field with a regular Eulerian grid using 501 equally spaced samples ( P = 501 ).

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: Testing data

Original: We then approximate the spatial gradient using the ﬁnite difference method,
Revised: We then approximate the spatial gradient using the finite difference method,

Original: [Equation]
Revised: [Equation]

Original: where ∆ x is the grid spacing.
Revised: where ∆ x is the grid spacing.

Revised: We use this to compute the next-time step velocity,

Revised: [Equation]

Original: We assume a ﬁrst-order explicit time-stepping scheme,
Revised: We assume a first-order explicit time-stepping scheme,

Original: [Equation]
Revised: [Equation]

Original: G. R EDUCED - ORDER M ODEL
Revised: G. R EDUCED - ORDER M ODEL

Original: Instead of a ﬁnite-difference treatment of the spatial gradient, we compute the gradient via direct differentiation of the network.
Revised: Instead of a finite-difference treatment of the spatial gradient, we compute the gradient via direct differentiation of the network.

Original: The latent space dimension is r = 16 and the width of the MLP is β = 
Revised: The latent space dimension is r = 16 and the width of the MLP is β = 128 .

Original: . G. T RAINING AND T ESTING D ATA
Revised: G. T RAINING AND T ESTING D ATA

Original: We generate training data by setting ν to different piecewise constant functions of three regions, i.e., the parameter vector µ ∈ D
Revised: We generate training data by setting ν to different piecewise constant functions of three regions, i.e., the parameter vector µ ∈ D

Original: =
Revised: =

Original: [0 . 2 , 1 . 0] 3 ⊂ R 3 .
Revised: [0 . 2 , 1 . 0] 3 ⊂ R 3 .

Original: Figure 15 displays the initial and the ﬁnal states of training data.
Revised: Figure 15 displays the initial and the final states of training data.

Original: Notice how different regions diffuse with different speeds due to the spatially varying ν .
Revised: Notice how different regions diffuse with different speeds due to the spatially varying ν .

Original: In total, 8 PDE temporal sequences (of 100 time steps) with different ν ’s are generated for training.
Revised: In total, 8 PDE temporal sequences (of 100 time steps) with different ν ’s are generated for training.

Original: We sample another 4 ν ’s for testing purposes.
Revised: We sample another 4 ν ’s for testing purposes.

Original: All training and testing data adopt the same initial condition (Figure 15a).[Figure] 1.0 x 0. u
Revised: All training and testing data adopt the same initial condition (Figure 15a).

Original: G. R ESULTS
Revised: G. R ESULTS

Original: Figure 16 demonstrates the performance of the reduced-order model on the testing dataset.
Revised: Figure 16 demonstrates the performance of the reduced-order model on the testing dataset.

Original: With the robust sampling scheme, we can achieve 0 .
Revised: With the robust sampling scheme, we can achieve 0 .

Original: 43% error using just 22 spatial samples (see Figure 4).
Revised: 43% error using just 22 spatial samples (see Figure 4).

Original: In addition, by incorporating different initial conditions (e.g., Gaussian proﬁles centered at different horizontal locations) in the training data, our approach can capture diffusion with a wide range of initial conditions (Figure 17). 
Revised: In addition, by incorporating different initial conditions (e.g., Gaussian profiles centered at different horizontal locations) in the training data, our approach can capture diffusion with a wide range of initial conditions (Figure 17). [Figure] [Figure] [Figure]

Original: To gauge our approach’s ability to conserve physical laws, we measure the energy evolution over time in the region between x 1 = 0 .
Revised: To gauge our approach’s ability to conserve physical laws, we measure the energy evolution over time in the region between x 1 = 0 .

Original: 396 and x 2 = 0 .
Revised: 396 and x 2 = 0 .

Original: 526 (Figure 16).
Revised: 526 (Figure 16).

Original: As shown by Cannon (1984), the escaped energy (EE) is measured by the ﬂux difference ( ∂u∂x ( x 1 , t ) − ∂u∂x ( x 2 , t ) ) at the left and right boundaries integrated over time; the stored energy (SE) is measured by the heat energy density, proportional to the temperature ( u ), integrated over space; the total energy (TE) is the sum of the prior two energies, i.e., TE=EE+SE.
Revised: As shown by Cannon (1984), the escaped energy (EE) is measured by the flux difference ( ∂u∂x ( x 1 , t ) − ∂u∂x ( x 2 , t ) ) at the left and right boundaries integrated over time; the stored energy (SE) is measured by the heat energy density, proportional to the temperature ( u ), integrated over space; the total energy (TE) is the sum of the prior two energies, i.e., TE=EE+SE.

Original: Figure 4 demonstrates that the reduced simulation preserves the total energy over time.
Revised: Figure 4 demonstrates that the reduced simulation preserves the total energy over time.

Original: H I MAGE P ROCESSING H. C ONTINUOUSPDE
Revised: H I MAGE P ROCESSING H. C ONTINUOUS

Original: [Equation]
Revised: [Equation]

Original: In image processing, we study the Perona–Malik diffusion equation of the gray-scale image u (Perona & Malik, 1990).
Revised: In image processing, we study the Perona–Malik diffusion equation of the gray-scale image u (Perona & Malik, 1990).

Original: H. F ULL - ORDER M ODEL
Revised: H. F ULL - ORDER M ODEL

Original: The full-order model discretizes the spatial 2D pixel ﬁeld with a regular Eulerian grid using 256 ×equally spaced samples ( P = 65 , 536 ).
Revised: The full-order model discretizes the spatial 2D pixel field with a regular Eulerian grid using 256 ×equally spaced samples ( P = 65 , 536 ).

Original: We then approximate the spatial gradient using the ﬁnite difference method,
Revised: We then approximate the spatial gradient using the finite difference method,

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: We use these to compute the next-time step velocity,

Original: [Equation]
Revised: [Equation]

Original: We assume a ﬁrst-order explicit time-stepping scheme,
Revised: We assume a first-order explicit time-stepping scheme,

Original: [Equation]
Revised: [Equation]

Original: where ∆ x = ∆ y is the grid spacing.
Revised: where ∆ x = ∆ y is the grid spacing.

Original: H. R EDUCED - ORDER M ODEL
Revised: H. R EDUCED - ORDER M ODEL

Original: The reduced-order model follows the same treatments as Appendix G.3.
Revised: The reduced-order model follows the same treatments as Appendix G.3.

Original: H. T RAINING AND T ESTING D ATA
Revised: H. T RAINING AND T ESTING D ATA

Original: Similar to Appendix G.4, we generate training data by setting ν to different piecewise constant functions of four equally-spaced regions, i.e., the parameter vector µ ∈ D
Revised: Similar to Appendix G.4, we generate training data by setting ν to different piecewise constant functions of four equally-spaced regions, i.e., the parameter vector µ ∈ D

Original: = { 0 , 0 . 2 } 4 ⊂ R 4 .
Revised: = { 0 , 0 . 2 } 4 ⊂ R 4 .

Original: D train contains 11 parameter vectors (Figure 18), where 0 , 2 or 3 regions have zero diffusion coefﬁcients, i.e., no blurring.
Revised: D train contains 11 parameter vectors (Figure 18), where 0 , 2 or 3 regions have zero diffusion coefficients, i.e., no blurring.

Original: D test contains 4 parameter vectors, where only 1 region is unblurred.
Revised: D test contains 4 parameter vectors, where only 1 region is unblurred.

Original: For each training and testing parameter vector, we run a simulation of 20 time steps.
Revised: For each training and testing parameter vector, we run a simulation of 20 time steps.

Original: All training and testing data adopt the same initial condition (Figure 18a).
Revised: All training and testing data adopt the same initial condition (Figure 18a).

Original: H. R ESULTS
Revised: H. R ESULTS

Original: Figure 19 demonstrates the reduced simulation’s performance on the testing dataset.
Revised: Figure 19 demonstrates the reduced simulation’s performance on the testing dataset.

Original: Furthermore, with the robust sampling scheme, we can achieve 36 .
Revised: Furthermore, with the robust sampling scheme, we can achieve 36 .

Original: 1 PSNR using just 63 spatial samples (see
Revised: 1 PSNR using just 63 spatial samples (see

Original: Figure 20).
Revised: Figure 20).

Original: The dimension reduction is 99 .
Revised: The dimension reduction is 99 .

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: I A DVECTION E QUATION
Revised: I A DVECTION E QUATION

Original: Nonlinear ROM techniques have been shown to signiﬁcantly outperform linear ROM techniques on problem with slowly decaying Kolmogorov n-widths (Peherstorfer, 2022; Lee & Carlberg, 2020).
Revised: Nonlinear ROM techniques have been shown to significantly outperform linear ROM techniques on problem with slowly decaying Kolmogorov n-widths (Peherstorfer, 2022; Lee & Carlberg, 2020).

Original: In this section, we test CROM, also a nonlinear ROM method, on a transport-dominated problem, given by the advection equation.
Revised: In this section, we test CROM, also a nonlinear ROM method, on a transport-dominated problem, given by the advection equation.

Original: ContinuousPDE
Revised: Continuous

Original: [Equation]
Revised: [Equation]

Original: where u is the advected quantity and a is the advection velocity.
Revised: where u is the advected quantity and a is the advection velocity.

Original: I. F ULL - ORDER M ODEL
Revised: [Equation]

Original: The full-order model discretizes the 1D spatial ﬁeld with a regular grid using 100 equally spaced samples.
Revised: The full-order model discretizes the 1D spatial field with a regular grid using 100 equally spaced samples.

Original: The PDE is temporally discretized via forward Euler.
Revised: The PDE is temporally discretized via forward Euler.

Original: I. R EDUCED - ORDER M ODEL
Revised: I. R EDUCED - ORDER M ODEL

Original: We vary the latent space dimension r and use a ﬁxed width of the MLP β = 20 .
Revised: We vary the latent space dimension r and use a fixed width of the MLP β = 20 .

Original: I. T RAINING AND T ESTING D ATA
Revised: I. T RAINING AND T ESTING D ATA

Original: In this example, we consider a reproductive case where the training and the testing data are the same.
Revised: In this example, we consider a reproductive case where the training and the testing data are the same.

Original: In particular, the initial condition is given by a Gaussian proﬁle and is then advected under a constant velocity.
Revised: In particular, the initial condition is given by a Gaussian profile and is then advected under a constant velocity.

Original: I. R ESULTS
Revised: I. R ESULTS

Original: Since this example is reproductive and the only parameter of the system is time t , the intrinsic solution-manifold dimension (Lee & Carlberg, 2020) is 1 .
Revised: Since this example is reproductive and the only parameter of the system is time t , the intrinsic solution-manifold dimension (Lee & Carlberg, 2020) is 1 .

Original: Figure 21 demonstrates that CROM indeed accurately captures the transport behavior of the Gaussian proﬁle even at the the intrinsic solution-manifold dimension ( r = 1 ).
Revised: Figure 21 demonstrates that CROM indeed accurately captures the transport behavior of the Gaussian profile even at the the intrinsic solution-manifold dimension ( r = 1 ).

Original: By contrast, POD suffers from serious artifacts with lower dimensional latent space and requires a signiﬁcantly higher dimensional latent space ( r = 16 ) in order to model the advection behavior.
Revised: By contrast, POD suffers from serious artifacts with lower dimensional latent space and requires a significantly higher dimensional latent space ( r = 16 ) in order to model the advection behavior.

Original: Furthermore, Figure 6 shows that our approach is also more accurate than the convolutional autoencoder approach by Lee & Carlberg (2020) when r = 1 .
Revised: Furthermore, Figure 6 shows that our approach is also more accurate than the convolutional autoencoder approach by Lee & Carlberg (2020) when r = 1 .

Original: J B URGERS ’ E QUATION
Revised: J B URGERS ’ E QUATION

Original: In this section, we continue to investigate problems with slowly decaying Kolmogorov n-widths.
Revised: In this section, we continue to investigate problems with slowly decaying Kolmogorov n-widths.

Original: In particular, we obtain the setup by Lee & Carlberg (2020) (see Section 7.1 of their paper) and compare with their (discretization-dependent) convolutional autoencoder manifold.[Figure]
Revised: In particular, we obtain the setup by Lee & Carlberg (2020) (see Section 7.1 of their paper) and compare with their (discretization-dependent) convolutional autoencoder manifold.

Original: Continuous PDE
Revised: Continuous PDE

Original: [Equation]
Revised: [Equation]

Original: where w is the variable of interest.
Revised: where w is the variable of interest.

Original: The initial condition is as follows.
Revised: The initial condition is as follows.

Original: [Equation]
Revised: [Equation]

Original: We further assume a zero Neumann boundary condition.
Revised: We further assume a zero Neumann boundary condition.

Original: J. F ULL - ORDER M ODEL
Revised: J. F ULL - ORDER M ODEL

Original: The full-order model discretizes the 1D spatial ﬁeld with a regular grid using 256 equally spaced samples.
Revised: The full-order model discretizes the 1D spatial field with a regular grid using 256 equally spaced samples.

Original: The PDE is temporally discretized via forward Euler. 
Revised: The PDE is temporally discretized via forward Euler. [Figure] [Figure]

Original: J. R EDUCED - ORDER M ODEL
Revised: J. R EDUCED - ORDER M ODEL

Original: We vary the latent space dimension r and use a ﬁxed width of the MLP β = 64 .
Revised: We vary the latent space dimension r and use a fixed width of the MLP β = 64 .

Original: J. T RAINING AND T ESTING D ATA
Revised: J. T RAINING AND T ESTING D ATA

Original: We generate training and testing data by varying the parameter µ D .
Revised: We generate training and testing data by varying the parameter µ D .

Original: We set the training parameter to µ D train = { 0 .
Revised: We set the training parameter to µ D train = { 0 .

Original: + (0 . 015 / 7) j } j =0 , 1 ,... 7 , resulting in 8 training cases.
Revised: + (0 . 015 / 7) j } j =0 , 1 ,... 7 , resulting in 8 training cases.

Original: For online testing, we set the parameter to µ D test = 0 . 021 .
Revised: For online testing, we set the parameter to µ D test = 0 . 021 .

Original: J. R ESULTS
Revised: J. R ESULTS

Original: Since we can uniquely parameterize the family of training and testing data with two variables t and µ D , the intrinsic solution-manifold dimension (Lee & Carlberg, 2020) is p ∗ = 2 , which is the lower bound for the latent space dimension r .
Revised: Since we can uniquely parameterize the family of training and testing data with two variables t and µ D , the intrinsic solution-manifold dimension (Lee & Carlberg, 2020) is p ∗ = 2 , which is the lower bound for the latent space dimension r .

Original: Figure 6b reports the result for r = p ∗ = 2 .
Revised: Figure 6b reports the result for r = p ∗ = 2 .

Original: While both POD and convolutional autoencoder (CAE) approach from Lee & Carlberg (2020) display clear artifacts, CROM agrees well with the ground truth.
Revised: While both POD and convolutional autoencoder (CAE) approach from Lee & Carlberg (2020) display clear artifacts, CROM agrees well with the ground truth.

Original: Such an agreement at the latent space dimension lower bound (i.e., r = p ∗ ) shows that CROM is very close to achieving the optimal performance of any nonlinear trial manifold.
Revised: Such an agreement at the latent space dimension lower bound (i.e., r = p ∗ ) shows that CROM is very close to achieving the optimal performance of any nonlinear trial manifold.

Original: Furthermore, Figure 22 demonstrates the accuracy advantage of our approach over the CAE approach across different latent space dimensions.
Revised: Furthermore, Figure 22 demonstrates the accuracy advantage of our approach over the CAE approach across different latent space dimensions.

Revised: K F LUID D YNAMICS K. C ONTINUOUS PDE

Original: [Equation]
Revised: [Equation]

Original: We study the incompressible Navier-Stokes equation, where u is the velocity ﬁeld, p is the pressure ﬁeld, and ν is the viscosity.
Revised: We study the incompressible Navier-Stokes equation, where u is the velocity field, p is the pressure field, and ν is the viscosity.

Original: We assume there is no external force, i.e., f ext = 0 .
Revised: We assume there is no external force, i.e., f ext = 0 .

Original: We assume a no-penetration boundary condition for the velocity and a zero-Dirichlet boundary condition for the pressure.
Revised: We assume a no-penetration boundary condition for the velocity and a zero-Dirichlet boundary condition for the pressure.

Original: K. F ULL - ORDER M ODEL
Revised: K. F ULL - ORDER M ODEL

Original: The full-order model assumes an operator-splitting scheme following the classic Chorin’s projection method (Chorin, 1968).
Revised: The full-order model assumes an operator-splitting scheme following the classic Chorin’s projection method (Chorin, 1968).

Original: In particular, we follow the implementation by Stam (1999).
Revised: In particular, we follow the implementation by Stam (1999).

Original: We summarize the major ingredients and refer to his paper for details.
Revised: We summarize the major ingredients and refer to his paper for details.

Original: We sequentially apply 3 linear operators to the velocity ﬁeld: diffusion, advection, and projection.
Revised: We sequentially apply 3 linear operators to the velocity field: diffusion, advection, and projection.

Original: From the previous time step velocity ﬁeld u n , we apply the diffusion operator according to the[Figure] viscosity and obtain the diffused velocity u visc n .
Revised: From the previous time step velocity field u n , we apply the diffusion operator according to the viscosity and obtain the diffused velocity u visc n .

Original: Next, we employ semi-Lagrangian to obtain the advected velocity ﬁeld u adv n .
Revised: Next, we employ semi-Lagrangian to obtain the advected velocity field u adv n .

Original: For the last projection step, we ﬁrst compute the pressure p n +1 by solving the poisson equation, ∇ 2 p n +1 = ∇ · u adv n .
Revised: For the last projection step, we first compute the pressure p n +1 by solving the poisson equation, ∇ 2 p n +1 = ∇ · u adv n .

Original: Afterwards, we apply the pressure gradient to obtain the divergence-free velocity u n +1 .
Revised: Afterwards, we apply the pressure gradient to obtain the divergence-free velocity u n +1 .

Original: All operations are done on a 2D 16 by 16 Eulerian grid.
Revised: All operations are done on a 2D 16 by 16 Eulerian grid.

Original: K. R EDUCED - ORDER M ODEL
Revised: K. R EDUCED - ORDER M ODEL

Original: We compute latent space dynamics ( q n +1 ) of the velocity ﬁeld by projecting u n +1 onto the lowdimensional manifold g θ g .
Revised: We compute latent space dynamics ( q n +1 ) of the velocity field by projecting u n +1 onto the manifoldparameterization function g θ g .

Original: Furthermore, we train another implicit neural representation based low-dimensional manifold for the pressure ﬁeld, h θ h ≈ p .
Revised: Furthermore, we train another implicit neural representation based manifold-parameterization function for the pressure field, h θ h ≈ p .

Original: Low-dimensional manifold h θ h evolves a separate latent space vector z .
Revised: manifold-parameterization function h θ h evolves a separate latent space vector z .

Original: One way to compute the latent space dynamics of h θ h is inverting the network from p n +1 :
Revised: One way to compute the latent space dynamics of h θ h is inverting the network from p n +1 :

Original: [Equation]
Revised: [Equation]

Original: This approach requires obtaining the Poisson-solved pressure p n +1 ( ∀ y ∈ M ).
Revised: This approach requires obtaining the Poisson-solved pressure p n +1 ( ∀ y ∈ M ).

Original: However, the Poisson solve itself is a global solve and therefore requires all the degrees of freedom ( P ).
Revised: However, the Poisson solve itself is a global solve and therefore requires all the degrees of freedom ( P ).

Original: To alleviate this computation burden, we bypass the grid-based Poisson solve and directly invert the network with the divergence-free loss function:
Revised: To alleviate this computation burden, we bypass the grid-based Poisson solve and directly invert the network with the divergence-free loss function:

Original: [Equation]
Revised: [Equation]

Original: As such, the latent space dynamics of the pressure ﬁeld now only require spatial samples in M .
Revised: As such, the latent space dynamics of the pressure field now only require spatial samples in M .

Original: In practice, for the ﬁrst two time steps of the solver where extreme pressure change happens, we use the ﬁrst approach (Equation (8)) so that we can beneﬁt from the encoder-based initial guess discussed in Appendix C.1.
Revised: In practice, for the first two time steps of the solver where extreme pressure change happens, we use the first approach (Equation (29)) so that we can benefit from the encoder-based initial guess discussed in Appendix C.1.

Original: For all remaining time steps, we solve the more efﬁcient Equation (9).
Revised: For all remaining time steps, we solve the more efficient Equation (30).

Original: For network hyperparameters, we adopt a latent space dimension r = 6 and a MLP width β
Revised: For network hyperparameters, we adopt a latent space dimension r = 6 and a MLP width β

Original: = 10 .
Revised: = 10 .

Original: [Figure]

Original: K. T RAINING AND T ESTING D ATA
Revised: K. T RAINING AND T ESTING D ATA

Original: We generate training data by varying the spatially-constant viscosity: µ = ν ∈ D =
Revised: We generate training data by varying the spatially-constant viscosity: µ = ν ∈ D =

Original: 02] ⊂ R .
Revised: 02] ⊂ R .

Original: We sample 3 different viscosities from D .
Revised: We sample 3 different viscosities from D .

Original: For each viscosity ν , we run a simulation of 50 time steps.
Revised: For each viscosity ν , we run a simulation of 50 time steps.

Original: In total, 150 time steps are used for training.
Revised: In total, 150 time steps are used for training.

Original: For testing, we ﬁrst sample 3 viscosity values from D = [0 , 0 . 02] .
Revised: For testing, we first sample 3 viscosity values from D = [0 , 0 . 02] .

Original: Furthermore, we also sample 3 values from the extrapolated region D extrapolate =
Revised: Furthermore, we also sample 3 values from the extrapolated region D extrapolate =

Original: Notice how the extrapolated region has a larger range ( 0 . 08 ) than the interpolated region ( 0 . 02 ).
Revised: Notice how the extrapolated region has a larger range ( 0 . 08 ) than the interpolated region ( 0 . 02 ).

Original: All training and testing data adopt the same initial condition (see Figure 23).
Revised: All training and testing data adopt the same initial condition (see Figure 23).

Original: K. R ESULTS
Revised: K. R ESULTS

Original: Figure 23demonstrates our method’s ability to qualitatively capture ﬂuid dynamics with different viscosities on both interpolated data as well as extrapolated data.
Revised: Figure 23 demonstrates our method’s ability to qualitatively capture fluid dynamics with different viscosities on both interpolated data as well as extrapolated data.

Original: Surrogate models that predict velocity ﬁelds directly from viscosities (e.g., using a neural network) typically cannot handle extrapolation.
Revised: Surrogate models that predict velocity fields directly from viscosities (e.g., using a neural network) typically cannot handle extrapolation.

Original: By contrast, our approach handles extrapolated scenarios because we only approximate the spatial representation (kinematics) with a neural network, not the PDE itself.
Revised: By contrast, our approach handles extrapolated scenarios because we only approximate the spatial representation (kinematics) with a neural network, not the PDE itself.

Original: Therefore, as long as the extrapolated cases involve vector ﬁelds that can be represented by the low-dimensional implicit  neural representation, our reduced-order solver is able to evolve the latent space vector to capture them according to the PDE.
Revised: Therefore, as long as the extrapolated cases involve vector fields that can be represented by the low-dimensional implicit [Figure] neural representation, our reduced-order solver is able to evolve the latent space vector to capture them according to the PDE.

Original: However, if the extrapolated quantity lies outside of the space spanned by the low-dimensional manifold, our approach would not be able to capture the dynamics.
Revised: However, if the extrapolated quantity lies outside of the space spanned by the manifold-parameterization function, our approach would not be able to capture the dynamics.

Original: CROM, like most ROM approaches, does not handle arbitrary extrapolation scenarios.
Revised: CROM, like most ROM approaches, does not handle arbitrary extrapolation scenarios.

Original: See more discussion on generalizability at the end of the main text.
Revised: See more discussion on generalizability at the end of the main text.

Original: We also emphasize that the example shown here is a proof-of-concept example highlighting the kinematics-approximation-only nature of CROM.
Revised: We also emphasize that the example shown here is a proof-of-concept example highlighting the kinematics-approximation-only nature of CROM.

Revised: To further test the proposed method, we set up a Karman vortex street experiment.

Revised: Karman vortex street is particularly challenging as the flow bifurcates from a steady state to the periodic vortex shedding regime.

Original: Future work should test CROM on more challenging boundary conditions (Copeland et al., 2022) and higher reynolds number (Holmes et al., 2012).
Revised: Mathematically speaking, the fixed point solution undergoes a Hopf bifurcation and becomes a limit cycle (Huerre & Monkewitz, 1990). To capture such an effect, the latent space dynamics needs to be robust under extreme nonlinearity. Many neural-network-based methods, such as physics informed neural network (Raissi et al., 2019), are well-known to fail in this case (Chuang & Barba, 2022). We follow the geometry and material parameters outlined by Nabizadeh et al.

Revised: No hyperreduction is considered.

Revised: Figure 24 shows that our latent space dynamics method faithfully represents the ground truth flow: from the initial steady state to the bifurcated vortex shedding regime.

Revised: The ground truth solver utilizes a 100 by 200 grid, with a total 40 , 000 degrees of freedom.

Revised: By contrast, our method captures the same level of details while employing a latent space dimension of only r = 4 .

Revised: Due to CROM’s kinematics-approximation-only nature, our approach also generalize beyond the training temporal range.

Revised: Future work should test CROM on more challenging boundary conditions (Copeland et al., 2022) and chaotic systems involving non-Markovian effects (Pan & Duraisamy, 2018).

Original: L S OLID M ECHANICS L. C ONTINUOUS PDE
Revised: L S OLID M ECHANICS L. C ONTINUOUS PDE

Original: We solve the elastodynamics equation arisen from solid mechanics for deformable soft body modeling:
Revised: We solve the elastodynamics equation arisen from solid mechanics for deformable soft body modeling:

Original: [Equation]
Revised: [Equation]

Original:  where φ ( X , t ) maps the undeformed (reference) position ( X ) of an arbitrary material point from the reference conﬁguration Ω to its deformed (current) position ( x ) at time t .
Revised: [Figure] where ϕ ( X , t ) maps the undeformed (reference) position ( X ) of an arbitrary material point from the reference configuration Ω to its deformed (current) position ( x ) at time t .

Original: Proper Dirichlet and
Revised: Proper Dirichlet and

Original: Neumann boundary conditions are applied.
Revised: Neumann boundary conditions are applied.

Original: We assume the ﬁrst Piola–Kirchhoff stress P is strictly a function of the deformation gradient
Revised: We assume the first Piola–Kirchhoff stress P is strictly a function of the deformation gradient

Original: F = ∇ φ .
Revised: F = ∇ ϕ .

Original: In particular, we adopt the corotated linear elasticity constitutive law,
Revised: In particular, we adopt the corotated linear elasticity constitutive law,

Original: [Equation]
Revised: [Equation]

Original: where R is the rotation tensor from the polar decomposition of the deformation gradient F = RS , λ and µ are the ﬁrst lame parameter (Pa) and the second lame parameter / shear modulus (Pa).
Revised: where R is the rotation tensor from the polar decomposition of the deformation gradient F = RS , λ and µ are the first lame parameter (Pa) and the second lame parameter / shear modulus (Pa).

Original: The proposed framework also works with other elasticity constitutive laws and can be easily extended to capture hysteresis and inelasticity.
Revised: The proposed framework also works with other elasticity constitutive laws and can be easily extended to capture hysteresis and inelasticity.

Original: L. F ULL - ORDER M ODEL
Revised: L. F ULL - ORDER M ODEL

Original: We adopt the tetrahedral-mesh-based linear ﬁnite element method (FEM) for the full-order model.
Revised: We adopt the tetrahedral-mesh-based linear finite element method (FEM) for the full-order model.

Original: We closely follow the course by Sifakis & Barbic (2012).
Revised: We closely follow the course by Sifakis & Barbic (2012).

Original: Here we review only the salient features and refer to their work for theoretical and practical details.
Revised: Here we review only the salient features and refer to their work for theoretical and practical details.

Original: We spatially discretize the domain of interest using linear tetrahedra.
Revised: We spatially discretize the domain of interest using linear tetrahedra.

Original: For each tetrahedron, we have four undeformed vertex positions ( X 1 , · · · , X 4 ) and four deformed vertex positions ( x 1 , · · · , x 4 ).
Revised: For each tetrahedron, we have four undeformed vertex positions ( X 1 , · · · , X 4 ) and four deformed vertex positions ( x 1 , · · · , x 4 ).

Original: For each vertex, we have x i =
Revised: For each vertex, we have x i =

Original: F X i + b , where b is the translation.
Revised: F X i + b , where b is the translation.

Original: Consequently, we can obtain the deformation gradient F of each tetrahedron by solving D s = F D m , where D s =
Revised: Consequently, we can obtain the deformation gradient F of each tetrahedron by solving D s = F D m , where D s =

Original: [ x 1 − xx 2 − xx 3 − x 4 ] and D m =
Revised: [ x 1 − xx 2 − xx 3 − x 4 ] and D m =

Original: [ X 1 − X
Revised: [ X 1 − X

Original: X 2 − X
Revised: X 2 − X

Original: X 3 − X 4 ] .
Revised: X 3 − X 4 ] .

Original: Furthermore, the internal elastic forces on the vertices ( f 1 , · · · , f 4 ) can be computed though [ [Equation] − T and f 4 = − f 1 − f 2 − f 3 .
Revised: Furthermore, the internal elastic forces on the vertices ( f 1 , · · · , f 4 ) can be computed though [ fff 3 ] = − 16 | det D m | P ( F ) D m − T and f 4 = − f 1 − f 2 − f 3 .

Original: L. R EDUCED - ORDER M ODEL
Revised: L. R EDUCED - ORDER M ODEL

Original: Since the initial condition is known for a given problem, we opt to construct a low-dimensional manifold for the displacement ﬁeld u ( X , t ) =
Revised: Since the initial condition is known for a given problem, we opt to construct a manifoldparameterization function for the displacement field u ( X , t ) =

Original: φ − X .
Revised: ϕ − X .

Original: The deformation map can then be computed via φ = u + X .
Revised: The deformation map can then be computed via ϕ = u + X .

Original: The hyperparameters of the low-dimensional manifold are r = 2 and β = 20 .
Revised: The hyperparameters of the manifold-parameterization function are r = 2 and β = 20 .

Original: Unlike previously discussed PDEs, the elastodynamics equation is second-order in time.
Revised: Unlike previously discussed PDEs, the elastodynamics equation is second-order in time.

Original: Consequently, in addition to the function value of the manifold, we also need to infer the temporal derivative during latent space dynamics.
Revised: Consequently, in addition to the function value of the manifold, we also need to infer the temporal derivative during latent space dynamics.

Original: We do so by employing the tangent space of the manifold,
Revised: We do so by employing the tangent space of the manifold,

Original: [Equation]
Revised: [Equation]

Revised: ) , where ˙ q n = ( q n − q n − 1 ) / ∆ t and ˙ q 0 = 0 .

Original: To compute the acceleration (and increment the velocity) of a vertex belonging to the integration samples set M , we need to evaluate the internal force at this particular vertex.
Revised: To compute the acceleration (and increment the velocity) of a vertex belonging to the integration samples set M , we need to evaluate the internal force at this particular vertex.

Original: We can calculate the vertex force by accumulating the internal forces of tetrahedra incident on this particular vertex.
Revised: We can calculate the vertex force by accumulating the internal forces of tetrahedra incident on this particular vertex.

Original: Tetrahedron forces are computed using the formula described in the previous section.
Revised: Tetrahedron forces are computed using the formula described in the previous section.

Original: To attain optimal gradient accuracy (Appendix E.2), we opt to use ﬁnite element’s linear basis function to compute the deformation gradient F of each tetrahedron.
Revised: To attain optimal gradient accuracy (Appendix E.2), we opt to use finite element’s linear basis function to compute the deformation gradient F of each tetrahedron.

Original: To compute these internal forces and deformation gradients, we also need to obtain the position information (via the low-dimensional manifold) of all vertices in the one-ring neighborhoods of the integration samples.
Revised: To compute these internal forces and deformation gradients, we also need to obtain the position information (via the manifold-parameterization function) of all vertices in the one-ring neighborhoods of the integration samples.

Original: We refer to the set of these neighboring vertices as N .
Revised: We refer to the set of these neighboring vertices as N .

Original: Therefore, the total number of spatial samples in the reduced-order model now becomes |M| + |N| .
Revised: Therefore, the total number of spatial samples in the reduced-order model now becomes |M| + |N| .

Original: Since one-ring neighbors only entail a small subset of the original degrees of freedom, we still offer signiﬁcant spatial sample reduction from the full order model ( P ).
Revised: Since one-ring neighbors only entail a small subset of the original degrees of freedom, we still offer significant spatial sample reduction from the full order model ( P ).

Original: L. T RAINING AND T ESTING D ATA L.4. G RAVITY - INDUCED I MPACT
Revised: L. T RAINING AND T ESTING D ATA L.4. G RAVITY - INDUCED I MPACT

Original: An initially-static rectangular-shaped deformable object ( P = 2 , 065 vertices and 9 , 346 tetrahedra) accelerates due to downward gravity.
Revised: An initially-static rectangular-shaped deformable object ( P = 2 , 065 vertices and 9 , 346 tetrahedra) accelerates due to downward gravity.

Original: After impacting the collision objects underneath, the material undergoes intense deformation.
Revised: After impacting the collision objects underneath, the material undergoes intense deformation.

Original: 𝜇 = 6200,𝜆 = 46667 ; 1.28% error 𝜇 = 10800,𝜆 = 10000 ; 0.91% error 𝜇

Original: =

Original: 15400,𝜆

Original: = 65000 ; 2.24% error 𝜇 = 20000,𝜆

Original: = 28333 ; 2.79% error

Original: We generate training and testing data by sampling the shear modulus µ = µ ∈ D =
Revised: We generate training and testing data by sampling the shear modulus µ = µ ∈ D =

Original: [60 , 000 , 70 , 000] ⊂ R .
Revised: [60 , 000 , 70 , 000] ⊂ R .

Original: The ﬁrst lame parameter λ is ﬁxed to be zero, in which case the corotated linear elasticity constitutive law simpliﬁes to the As-Rigid-As-Possible energy (Sorkine &
Revised: The first lame parameter λ is fixed to be zero, in which case the corotated linear elasticity constitutive law simplifies to the As-Rigid-As-Possible energy (Sorkine &

Original: Alexa, 2007).
Revised: Alexa, 2007).

Original: Speciﬁcally, we uniformly generate 4 samples of the shear modulus for training data.
Revised: Specifically, we uniformly generate 4 samples of the shear modulus for training data.

Original: We simulate 200 time steps for each shear modulus.
Revised: We simulate 200 time steps for each shear modulus.

Original: Therefore, a total of 800 time steps are used for training.
Revised: Therefore, a total of 800 time steps are used for training.

Original: We then randomly generate 3 samples of the shear modulus from D .
Revised: We then randomly generate 3 samples of the shear modulus from D .

Original: We also simulate 200 time steps for these shear moduli and use them for testing.
Revised: We also simulate 200 time steps for these shear moduli and use them for testing.

Original: We use the same initial condition for training and testing.
Revised: We use the same initial condition for training and testing.

Original: L.4. T ORSION AND T ENSION
Revised: L.4. T ORSION AND T ENSION

Original: We study another common solid mechanics application where the material undergoes torsion and tension.
Revised: We study another common solid mechanics application where the material undergoes torsion and tension.

Original: The left and the right boundaries kinematically move at constant translational and angular velocities of the same magnitude but in opposite directions.
Revised: The left and the right boundaries kinematically move at constant translational and angular velocities of the same magnitude but in opposite directions.

Original: We solve the parameterized family of problem where µ = ( µ, λ ) ∈ D
Revised: We solve the parameterized family of problem where µ = ( µ, λ ) ∈ D

Original: = [6 , 200 , 20 , 000] × [10 , 000 , 65 , 000] ⊂ R 2 .
Revised: = [6 , 200 , 20 , 000] × [10 , 000 , 65 , 000] ⊂ R 2 .

Original: We obtain 9 training samples via uniform full-factorial sampling of the parameter space.
Revised: We obtain 9 training samples via uniform full-factorial sampling of the parameter space.

Original: We simulate 100 time steps for each training sample.
Revised: We simulate 100 time steps for each training sample.

Original: We then use the Latin hypercube method to randomly generate another 4 testing samples.
Revised: We then use the Latin hypercube method to randomly generate another 4 testing samples.

Original: We also simulate 100 time steps for these testing samples.
Revised: We also simulate 100 time steps for these testing samples.

Original: Training and testing data adopt the same initial condition.
Revised: Training and testing data adopt the same initial condition.

Original: L. R ESULTS
Revised: L. R ESULTS

Original: Figure 7 demonstrates a testing case ( µ = 62500 , 2 .
Revised: Figure 7 demonstrates a testing case ( µ = 62500 , 2 .

Original: 68% error from the full-order ground truth) for the experiment from Appendix L.4.1.
Revised: 68% error from the full-order ground truth) for the experiment from Appendix L.4.1.

Original: Since our method has a discretization-agnostic architecture, we can adapt the simulation resolution to improve efﬁciency.
Revised: Since our method has a discretization-agnostic architecture, we can adapt the simulation resolution to improve efficiency.

Original: The adaptive simulation adopts a contact-based oracle that switches from the low-resolution mesh to the high-resolution mesh once a contact event is detected.
Revised: The adaptive simulation adopts a contact-based oracle that switches from the low-resolution mesh to the high-resolution mesh once a contact event is detected.

Original: Before the start of the simulation, we generated both the low-resolution and the high-resolution meshes using TetWild (Hu et al., 2018) with small and large ideal edge lengths.
Revised: Before the start of the simulation, we generated both the low-resolution and the high-resolution meshes using TetWild (Hu et al., 2018) with small and large ideal edge lengths.

Original: After training on the experiment from Appendix L.4.2, our method captures a wide range of shearing and volume-preserving behaviors (see Figure 24).
Revised: After training on the experiment from Appendix L.4.2, our method captures a wide range of shearing and volume-preserving behaviors (see Figure 25).

Revised: 𝜇 = 6200,𝜆 = 46667 ; 1.28% error 𝜇 = 10800,𝜆 = 10000 ; 0.91% error 𝜇

Revised: =

Revised: 15400,𝜆

Revised: = 65000 ; 2.24% error 𝜇 =

Revised: [Figure]

Original:  Furthermore, to showcase CROM’s ability to handle large deformations, we reproduce the dinosaur example by Shen et al.
Revised: = [Figure] [Figure] Furthermore, to showcase CROM’s ability to handle large deformations, we reproduce the dinosaur example by Shen et al.

Original: Figure 10 shows that CROM accurately captures the large deformations of the dinosaur.
Revised: Figure 10 shows that CROM accurately captures the large deformations of the dinosaur.

Original: By contrast, under the same latent space dimension, POD has a limited expressivity and suffers from visual artifacts: the head and the tail’s motions are constrained.
Revised: By contrast, under the same latent space dimension, POD has a limited expressivity and suffers from visual artifacts: the head and the tail’s motions are constrained.

Original: This observation is consistent with the results reported by Shen et al.
Revised: This observation is consistent with the results reported by Shen et al.

Original: (2021) where nonlinear approaches outperform linear approaches.
Revised: (2021) where nonlinear approaches outperform linear approaches.

Original: In terms of memory consumptions, CROM is orders-of-magnitude more efﬁcient than these prior discretization-dependent ROM approaches (Barbiˇc, 2012; Fulton et al., 2019;
Revised: In terms of memory consumptions, CROM is orders-of-magnitude more efficient than these prior discretization-dependent ROM approaches (Barbiˇc, 2012; Fulton et al., 2019;

Original: Shen et al., 2021).
Revised: Shen et al., 2021).

Original: M C OMPARISON WITH P RIOR A PPROACHES
Revised: M C OMPARISON WITH P RIOR A PPROACHES

Original: We compare our approach with prior reduced-order approaches that generate fast PDE solutions via latent space dynamics.
Revised: We compare our approach with prior reduced-order approaches that generate fast PDE solutions via latent space dynamics.

Original: In total, we compare with four baselines: (1) the classic widely-adopted proper orthogonal decomposition (POD) method (Berkooz et al., 1993; Holmes et al., 2012).
Revised: In total, we compare with four baselines: (1) the classic widely-adopted proper orthogonal decomposition (POD) method (Berkooz et al., 1993; Holmes et al., 2012).

Original: Our
Revised: Our

Original: POD implementation follows the PCA approach by Barbiˇc & James (2005);
Revised: POD implementation follows the PCA approach by Barbi ˇ c & James (2005);

Original: Barbiˇc (2012); (2) the neural-network-based autoencoder approach by Fulton et al.
Revised: Barbi ˇ c (2012); (2) the neural-network-based autoencoder approach by Fulton et al.

Original: (3) the neural-network-based autoencoder approach by Shen et al.
Revised: (3) the neural-network-based autoencoder approach by Shen et al.

Original: (2021); (4) the convolutional autoencoder (CAE) approach by
Revised: (2021); (4) the convolutional autoencoder (CAE) approach by

Original: Lee & Carlberg (2020).
Revised: Lee & Carlberg (2020).

Original: All these four baselines are discretization-dependent .
Revised: All these four baselines are discretization-dependent .

Original: SIREN, PINN, or GNN are related to our work but are very different in nature.
Revised: SIREN, PINN, or GNN are related to our work but are very different in nature.

Original: While their goal is to directly solve PDEs with neural network architectures, our goal is to leverage neural networks for dimension reduction.
Revised: While their goal is to directly solve PDEs with neural network architectures, our goal is to leverage neural networks for dimension reduction.

Original: Therefore, we do not compare with them and focus on comparison with other dimension reduction methods.
Revised: Therefore, we do not compare with them and focus on comparison with other dimension reduction methods.

Original: POD approximates the discretized vector ﬁeld with a linear basis U ,
Revised: POD approximates the discretized vector field with a linear basis U ,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where U is a matrix of size Pd by r .
Revised: where U is a matrix of size Pd by r .

Original: U is most easily constructed via a singular value decomposition of the discretized training data (Sifakis & Barbic, 2012).
Revised: U is most easily constructed via a singular value decomposition of the discretized training data (Sifakis & Barbic, 2012).

Original: Autoencoders have the same input and output as POD.
Revised: Autoencoders have the same input and output as POD.

Original: However, the linear basis is replaced with nonlinear neural networks, 
Revised: However, the linear basis is replaced with nonlinear neural networks, −→

Original: [Equation]
Revised: [Equation]

Original: where h is the decoder portion of the autoencoder structure.
Revised: where h is the decoder portion of the autoencoder structure.

Original: Fulton et al.
Revised: Fulton et al.

Original: ; Shen et al.
Revised: ; Shen et al.

Original: Lee & Carlberg (2020) all follow this formulation but differ in their particular architecture choices.
Revised: Lee & Carlberg (2020) all follow this formulation but differ in their particular architecture choices.

Original: We implement both POD and the autoencoder approaches in the same PyTorch framework as CROM.

Original: These approaches share exactly the same PDE time-stepping algorithm (step 2) during latent space dynamics.

Original: For POD and the autoencoder approach, the network inference step (step 1) and the network inversion step (step 3) employ the linear basis U and the nonlinear decoder function h , respectively.

Original: M. S PEED

Original: We measure the wall-clock computation time during the falling stage (before contact) of the gravityinduced impact experiment (see Appendix L.4.1 and Figure 7).

Original: In particular, since all model reduction methods (including our approach, POD, and the autoencoder approaches) share the same PDE timestepping algorithm (step 2), we measure its wall-clock computation cost to ensure the generalizability of the comparison.

Original: Since POD and the autoencoder approach are baked into a particular discretization, it does not allow for adaptive discretization.

Original: Therefore, the high-resolution mesh ( 2 , 065 vertices, 9 , 346 tetrahedra) has to be used throughout the falling stage even though there is minimal deformation.

Original: Consequently, its computation cost is signiﬁcantly higher than our approach, which employs a low-resolution mesh ( 8 vertices, 5 tetrahedra) during the falling stage.

Original: Figure 25 shows that our approach is faster than the prior ROM approach with different CPU thread counts.

Original: Our method’s advantage is most obvious when there is extremely limited computational resource (1 CPU thread).

Original: Since this speedup comes from the discretization independent nature of CROM, we expect it to generalize to other prior discretization-dependent ROM approaches as well.

Original: However, for application where remeshing is unnecessary, we do not expect this kind of speedup.

Original: Hyper-reduction techniques (An et al., 2008), such as the integration samples introduced in our approach, can also be employed to speed up the high-resolution mesh solution by reducing spatial

Original: 0 135 1 CPU thread 0.
Revised: 0 135 1 CPU thread 0.

Original: W a ll c l o ck ( s )
Revised: W a ll c l o ck ( s )

Original: SIMULATION EFFICIENCY: OURS VS.
Revised: SIMULATION EFFICIENCY: OURS VS.

Original: PRIOR
Revised: PRIOR

Original: CROM (Ours)
Revised: CROM (Ours)

Original: Prior ROM 024 4 CPU threads 0.
Revised: Prior ROM 024 4 CPU threads 0.

Original: W a ll c l o ck ( s )
Revised: W a ll c l o ck ( s )

Original: SIMULATION EFFICIENCY: OURS VS.
Revised: SIMULATION EFFICIENCY: OURS VS.

Original: PRIOR
Revised: PRIOR

Original: CROM (Ours)
Revised: CROM (Ours)

Original: Prior ROM 024 32 CPU threads 0.
Revised: Prior ROM 024 32 CPU threads 0.

Original: W a ll c l o ck ( s )
Revised: W a ll c l o ck ( s )

Original: SIMULATION EFFICIENCY: OURS VS.
Revised: SIMULATION EFFICIENCY: OURS VS.

Original: PRIOR
Revised: PRIOR

Original: CROM (Ours)
Revised: CROM (Ours)

Original: Prior ROM samples.
Revised: Prior ROM We implement both POD and the autoencoder approaches in the same PyTorch framework as CROM. These approaches share exactly the same PDE time-stepping algorithm (step 2) during latent space dynamics. For POD and the autoencoder approach, the network inference step (step 1) and the network inversion step (step 3) employ the linear basis U and the nonlinear decoder function h , respectively.

Revised: M. S PEED

Revised: We measure the wall-clock computation time during the falling stage (before contact) of the gravityinduced impact experiment (see Appendix L.4.1 and Figure 7).

Original: However, this is orthogonal to our adaptive discretization contribution, and the hyperreduction techniques can be employed to further accelerate the  low-resolution mesh solution as well.
Revised: In particular, since all model reduction methods (including our approach, POD, and the autoencoder approaches) share the same PDE timestepping algorithm (step 2), we measure its wall-clock computation cost to ensure the generalizability of the comparison. Since POD and the autoencoder approach are baked into a particular discretization, it does not allow for adaptive discretization. Therefore, the high-resolution mesh ( 2 , 065 vertices, 9 , 346 tetrahedra) has to be used throughout the falling stage even though there is minimal deformation. Consequently, its computation cost is significantly higher than our approach, which employs a low-resolution mesh ( 8 vertices, 5 tetrahedra) during the falling stage.

Revised: Figure 26 shows that our approach is faster than the prior ROM approach with different CPU thread counts.

Revised: Our method’s advantage is most obvious when there is extremely limited computational resource (1 CPU thread).

Revised: Since this speedup comes from the discretization independent nature of CROM, we expect it to generalize to other prior discretization-dependent ROM approaches as well.

Revised: However, for application where remeshing is unnecessary, we do not expect this kind of speedup.

Revised: Hyper-reduction techniques (An et al., 2008), such as the integration samples introduced in our approach, can also be employed to speed up the high-resolution mesh solution by reducing spatial samples.

Revised: However, this is orthogonal to our adaptive discretization contribution, and the hyperreduction techniques can be employed to further accelerate the low-resolution mesh solution as well.

Original: M. M EMORY
Revised: M. M EMORY

Original: We compare the memory consumptions of our method, POD, and the autoencoder approaches with the same latent space dimension ( r ), both trained on data from the torsion and tension experiment (see
Revised: We compare the memory consumptions of our method, POD, and the autoencoder approaches with the same latent space dimension ( r ), both trained on data from the torsion and tension experiment (see

Original: Appendix L.4.2).
Revised: Appendix L.4.2).

Original: For all implicit neural representation networks, we set the MLP width to be β = 20 .
Revised: For all implicit neural representation networks, we set the MLP width to be β = 20 .

Original: Independent of latent space dimension ( r ), our network has a near-constant memory consumption (see Figure 9).
Revised: Independent of latent space dimension ( r ), our network has a near-constant memory consumption (see Figure 9).

Original: By contrast, the memory consumptions of POD and the autoencoder approaches scale linearly with r .
Revised: By contrast, the memory consumptions of POD and the autoencoder approaches scale linearly with r .

Original: More importantly, since the output dimension of POD and the autoencoder is Pd , their memory consumptions scale linearly with the number of discretized positions P .
Revised: More importantly, since the output dimension of POD and the autoencoder is Pd , their memory consumptions scale linearly with the number of discretized positions P .

Original: By contrast, our approach’s memory consumption is independent of the number of spatial samples.
Revised: By contrast, our approach’s memory consumption is independent of the number of spatial samples.

Original: Consequently, in this large-scale example that features P = 66 , 608 vertices, we observe more than ten-fold advantages with our method across all latent space dimensions (see Figure 9).
Revised: Consequently, in this large-scale example that features P = 66 , 608 vertices, we observe more than ten-fold advantages with our method across all latent space dimensions (see Figure 9).

Original: In addition, we also compare our method with POD on the image processing experiment (see
Revised: In addition, we also compare our method with POD on the image processing experiment (see

Original: Appendix H).
Revised: Appendix H).

Original: CROM uses an order-of-magnitude less memory than POD (see Figure 5c).
Revised: CROM uses an order-of-magnitude less memory than POD (see Figure 5c).

Original: In
Revised: In

Original: Figure 26, we also demonstrate CROM’s memory advantage over the convolutional autoencoder approach by Lee & Carlberg (2020). 
Revised: Figure 27, we also demonstrate CROM’s memory advantage over the convolutional autoencoder approach by Lee & Carlberg (2020). 1.00E+ 1.00E+ 1.00E+ M e m o r y ( N u m be r

Revised: o

Original: Even though the overall memory consumption of our approach is signiﬁcantly lower, our approach’s per-sample evaluation cost is higher.
Revised: f P a r a m e t e r s ) Latent Space Dimension r CROM (Ours)

Revised: [Lee and Carlberg 2020]

Revised: Even though the overall memory consumption of our approach is significantly lower, our approach’s per-sample evaluation cost is higher.

Original: For any given discretized spatial sample, POD only requires a vector-matrix multiplication of a d by r matrix.
Revised: For any given discretized spatial sample, POD only requires a vector-matrix multiplication of a d by r matrix.

Original: By contrast, our approach needs to go through the entire MLP that employs dβ by dβ matrices.
Revised: By contrast, our approach needs to go through the entire MLP that employs dβ by dβ matrices.

Original: The autoencoder approaches share similar computation overheads.
Revised: The autoencoder approaches share similar computation overheads.

Original: M. A CCURACY
Revised: M. A CCURACY

Original: Figure 9 demonstrates the accuracy of the networks studied in the previous memory section.
Revised: Figure 9 demonstrates the accuracy of the networks studied in the previous memory section.

Original: Our method consistently offers orders-of-magnitude smaller training accuracies.
Revised: Our method consistently offers orders-of-magnitude smaller training accuracies.

Original: This accuracy advantage is especially noticeable with small latent space dimensions (e.g., r = 2 ).
Revised: This accuracy advantage is especially noticeable with small latent space dimensions (e.g., r = 2 ).

Original: Further increasing the latent space dimension leads to higher accuracies with POD and the autoencoder approaches but not our approach.
Revised: Further increasing the latent space dimension leads to higher accuracies with POD and the autoencoder approaches but not our approach.

Original: As discussed in Appendix F, our approach’s accuracy depends more saliently on the MLP width as opposed to the latent space dimension.
Revised: As discussed in Appendix F, our approach’s accuracy depends more saliently on the MLP width as opposed to the latent space dimension.

Original: Furthermore, unlike these discretization-dependent approaches, our continuous low-dimensional manifold also facilitates accurate handling of adaptive spatiotemporal data (Pan et al., 2022).
Revised: Furthermore, unlike these discretization-dependent approaches, our continuous manifold-parameterization function also facilitates accurate handling of adaptive spatiotemporal data (Pan et al., 2022).

Original: Furthermore, Figure 5 shows that our method is more accurate than POD on the image processing experiment, both visually and quantitatively.
Revised: Furthermore, Figure 5 shows that our method is more accurate than POD on the image processing experiment, both visually and quantitatively.

Original: Figure 22 displays our accuracy advantage over CAE on
Revised: Figure 22 displays our accuracy advantage over CAE on

Original: Burgers’ equations.
Revised: Burgers’ equations.

Original: N C OMPARISON WITH THE F ULL - ORDER M ODEL
Revised: M. T RAINING TIME

Revised: Figure 28 reports the offline training time for different ROM approaches. All neural-network-based methods, including ours, take longer time to train than the linear POD approach. We emphasize that the timing data is purely informative and calculated post-hod.

Original: To gauge the practical performance of our approach over the (unreduced) full-order model, we measure the sheer wall-clock performances of our reduced-order model and the full-order model.
Revised: The main target of our work is not to optimize training time but to achieve efficient latent space dynamics. Nevertheless, specifically for our implicit neural representation approach, future work may consider expediting training time via more advanced data structures and optimization (Müller et al., 2022; Liu et al., 2020; Martel et al., 2021; Takikawa et al., 2021).

Original: [Figure]
Revised: [Figure]

Original: M e m o r y

Original: ( N u m be r o f
Revised: [Fulton et al. 2019]

Original: P a r a m e t e r s )
Revised: [Shen et al. 2021]

Original: [Figure]
Revised: [Figure]

Original: r CROM
Revised: N C OMPARISON WITH THE F ULL - ORDER M ODEL

Original: [Figure] [Lee and Carlberg 2020] [Figure]
Revised: To gauge the practical performance of our approach over the (unreduced) full-order model, we measure the sheer wall-clock performances of our reduced-order model and the full-order model.

Original: To facilitate fair comparison
Revised: To facilitate fair comparison

Original: [Figure]
Revised: [Figure]

Original: we implement, optimize
Revised: we implement, optimize

Original: [Figure]
Revised: [Figure]

Original: and (fully) parallelize the full-order PDE solution and the reduced-order approach within the PyTorch framework (Paszke et al., 2019) without any other dependency.
Revised: and (fully) parallelize the full-order PDE solution and the reduced-order approach within the PyTorch framework (Paszke et al., 2019) without any other dependency.

Original: The full-order model and the reduced model share the same (parallelized)
Revised: The full-order model and the reduced model share the same (parallelized)

Original: PDE solver code (step 2) [Figure] and the only difference between them is the neural network evaluation in the reduced model.
Revised: PDE solver code (step 2), and the only difference between them is the neural network evaluation in the reduced model.

Original: To encourage fair comparison, for operations involving neural networks (step 1 and step 3), we do not use customized SIMD vectorizations, CUDA kernels, or highly optimized inference libraries such as TensorRT (Vanholder, 2016), though these optimizations should lead to a further performance gain of our algorithm.
Revised: To encourage fair comparison, for operations involving neural networks (step 1 and step 3), we do not use customized SIMD vectorizations, CUDA kernels, or highly optimized inference libraries such as TensorRT (Vanholder, 2016), though these optimizations should lead to a further performance gain of our algorithm.

Original: In addition, having both models implemented in the same framework allows us to compare them under the same computing environments, avoiding biased comparison where the full-order model runs on a signiﬁcantly limited computing resource (e.g., 8-core CPU) while the reduced-order model runs on a high-performance computing device (e.g., 5120-core GPU).
Revised: In addition, having both models implemented in the same framework allows us to compare them under the same computing environments, avoiding biased comparison where the full-order model runs on a significantly limited computing resource (e.g., 8-core CPU) while the reduced-order model runs on a high-performance computing device (e.g., 5120-core GPU).

Original: Figure 27 reports the wall clock time of our reduced-order method and the full-order method across different computing platforms.
Revised: Figure 29 reports the wall clock time of our reduced-order method and the full-order method across different computing platforms.

Original: While our approach is signiﬁcantly faster on all computing platforms, its strongest speedup is obtained on the CPUs.
Revised: While our approach is significantly faster on all computing platforms, its strongest speedup is obtained on the CPUs.

Original: This is unsurprising since the computationally expensive full-order model beneﬁts more from high-end data-center GPUs (e.g., V100) than the reduced-order model.
Revised: This is unsurprising since the computationally expensive full-order model benefits more from high-end data-center GPUs (e.g., V100) than the reduced-order model.

Original: In addition, because the neural network in the reduced-order model only[Figure] [Figure] employs a few 60 by 60 matrices ( d = 3 , β = 20 ), inference on CPU is also efﬁcient.
Revised: In addition, because the neural network in the reduced-order model only employs a few 60 by 60 matrices ( d = 3 , β = 20 ), inference on CPU is also efficient.

Original: This opens doors for employing our models on limited computing platforms such as mobile and VR devices.
Revised: This opens doors for employing our models on limited computing platforms such as mobile and VR devices.

Original:  Figure 28 further breaks down the computation time spent on each component of the reduced-order model.
Revised: [Figure] [Figure] Figure 30 further breaks down the computation time spent on each component of the reduced-order model.

Original: While the overhead of the neural network operations (step 1 and step 3) is non-negligible, the bottleneck of the algorithm remains to be PDE time-stepping (step 2).
Revised: While the overhead of the neural network operations (step 1 and step 3) is non-negligible, the bottleneck of the algorithm remains to be PDE time-stepping (step 2).

Original: In particular, the network inversion cost is very low, thanks to the linearized inversion solver discussed in Appendix C.
Revised: In particular, the network inversion cost is very low, thanks to the linearized inversion solver discussed in Appendix C.

Revised: Figure 31 reports the accuracy and the speed of our approach using different MLP layer widths ( β ), neural network inversion methods (nonlinear least squares (NLS) vs. linear least squares (LLS)),

Original: Overall, we observe a smooth nonlinear trajectory over time.
Revised: Overall, we observe a smooth nonlinear trajectory over time.

Original: In addition, we also linearly interpolate and extrapolate inside the latent space.
Revised: In addition, we also linearly interpolate and extrapolate inside the latent space.

Original: The corresponding images are displayed in Figure 32.
Revised: The corresponding images are displayed in Figure 34.

Original: Figure 29 reports the accuracy and the speed of our approach using different MLP layer widths ( β ), neural network inversion methods (nonlinear least squares (NLS) vs. linear least squares (LLS)), computing platforms (CPU vs. GPU), and integration sample sizes ( |M| ).
Revised: computing platforms (CPU vs. GPU), and integration sample sizes ( |M| ).

Original: In general, we observe similar accuracies between NLS and LLS (e.g., [ β = 20 , LLS, GPUs, |M| = 40 ] vs. [ β = 20 , NLS,
Revised: In general, we observe similar accuracies between NLS and LLS (e.g., [ β = 20 , LLS, GPUs, |M| = 40 ] vs. [ β = 20 , NLS,

Original: GPUs, |M| = 40 ]), while LLS is signiﬁcantly faster.
Revised: GPUs, |M| = 40 ]), while LLS is significantly faster.

Original: Increasing sample size lead to higher accuracies but also require longer computation time (e.g., [ β = 5 , NLS, CPUs, |M| = 10 ] vs. [ β = 5 , NLS,
Revised: Increasing sample size lead to higher accuracies but also require longer computation time (e.g., [ β = 5 , NLS, CPUs, |M| = 10 ] vs. [ β = 5 , NLS,

Original: CPUs, |M| = 40 ]).
Revised: CPUs, |M| = 40 ]).

Original: Increasing the MLP layer width yields higher accuracies but also increases the computation time.
Revised: Increasing the MLP layer width yields higher accuracies but also increases the computation time.

Original: In particular, the computation time is increased more on CPUs (e.g., [ β = 5 , LLS,
Revised: In particular, the computation time is increased more on CPUs (e.g., [ β = 5 , LLS,

Original: CPUs, |M| = 40 ] vs. [ β = 20 , LLS, CPUs, |M| = 40 ]) than on GPUs.
Revised: CPUs, |M| = 40 ] vs. [ β = 20 , LLS, CPUs, |M| = 40 ]) than on GPUs.

Original: O L ATENT - SPACE TRAJECTORY
Revised: O L ATENT - SPACE TRAJECTORY

Original: A key element of CROM is latent space dynamics, in which the latent space vector evolves nonlinearly over time under explicit PDE constraints.
Revised: A key element of CROM is latent space dynamics, in which the latent space vector evolves nonlinearly over time under explicit PDE constraints.

Original: Figure 30 demonstrates a nonlinear latent space trajectory from the image smoothing experiment.
Revised: Figure 32 demonstrates a nonlinear latent space trajectory from the image smoothing experiment.

Original: Guided by the PDE, the latent space vector traverses through the latent space and obtains the desirable smoothing effects in the image (see Figure 31).
Revised: Guided by the PDE, the latent space vector traverses through the latent space and obtains the desirable smoothing effects in the image (see Figure 33).

Original: After the simulation is ﬁnished, as a postprocessing experiment, we linearly interpolate and extrapolate inside the latent space (see Figure 30).
Revised: After the simulation is finished, as a postprocessing experiment, we linearly interpolate and extrapolate inside the latent space (see Figure 32).

Original: The interpolation end points are the latent vector for the unmodiﬁed image and the latent vector for the ﬁnal image determined by the latent space dynamics.
Revised: The interpolation end points are the latent vector for the unmodified image and the latent vector for the final image determined by the latent space dynamics.

Original: Figure 32 demonstrates the corresponding smooth transitions in the image space.
Revised: Figure 34 demonstrates the corresponding smooth transitions in the image space.

Original: P S TABILITY ANALYSIS
Revised: P S TABILITY ANALYSIS

Original: Figure 33 systematically studies the stability of our method.
Revised: Figure 35 systematically studies the stability of our method.

Original: CROM exhibits much larger stable time step sizes than its unreduced counterpart.
Revised: CROM exhibits much larger stable time step sizes than its unreduced counterpart.

Original: This is consistent with the theoretical analysis reported in the literature (Bach et al., 2018).
Revised: This is consistent with the theoretical analysis reported in the literature (Bach et al., 2018).

Original: t =
Revised: t =

Original: t =t = t =t =
Revised: t =t = t =t =

Original: t =t = t =t =
Revised: t =t = t =t =

Original: t =
Revised: t =

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Q L ATENT - SPACE D YNAMICS P SEUDOCODE
Revised: 1D [Table] [Equation]

Original: Algorithm 3: Latent-space Dynamics
Revised: Algorithm 3: Latent-space Dynamics

Original: Input: Latent space vector q n
Revised: Input: Latent space vector q n

Original: Output: Next-time-step latent space vector q n + 1 Network inference: gather full-space information.
Revised: Output: Next-time-step latent space vector q n + 1 Network inference: gather full-space information (Algorithm 4).

Original: 2 PDE time-stepping with boundary conditions: update full-space information via differential equations, e.g., the elastodynamics equation.
Revised: 2 PDE time-stepping with boundary conditions: update full-space information via differential equations, e.g., the elastodynamics equation (more details available from Appendix G to

Revised: Appendix L).

Original: 3 Network inversion: optimally project the updated full-space information onto the low-dimensional manifold.
Revised: 3 Network inversion: optimally project the updated full-space information onto the low-dimensional embedding (Algorithm 5).

Revised: Algorithm 4: Network inference: gather full-space information

Revised: Input: Latent space vector q n

Revised: Output: Full space information f n , ∇ f n , ˙ f n 1 for y ∈ M do

Revised: Compute the vector field value: f ( y , t n ) =

Revised: g θ g ( y , q n ) .

Original: R R EDUCED - ORDER M ODEL ’ S STATISTICS
Revised: Compute the temporal gradient either by differentiating the network ˙ f ( y , t n ) = ∂ g θg ∂ q ˙ q n , where ˙ q n = ( q n − q n − 1 ) / ∆ t , or by numerical approximation (Appendix E).

Revised: Compute the spatial gradient either by differentiating the network, ∇ f ( y , t n ) = ∇ y g θ g , or by numerical approximation (Appendix E).

Revised: 5 end

Revised: [Table]

Revised: space information f n +

Revised: [Table]

Revised: q n +

Revised: [Equation]

Revised: R R EDUCED - ORDER M ODEL ’ S STATISTICS

Original: CROM obtains considerable dimension reductions across different PDEs (Table 1).
Revised: CROM obtains considerable dimension reductions across different PDEs (Table 1).