Revised: W HAT

Revised: TO

Revised: D O

Revised: IF

Original: W HAT TO D O IF S PARSE R EPRESENTATION L EARNING F AILS U NEXPECTEDLY ?
Revised: S PARSE P HYSICAL E QUATION L EARNING F AILS U NEXPECTEDLY ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning physical equations from data is essential for scientiﬁc discovery and engineering modeling.
Revised: Learning physical equations from data is essential for scientific discovery and engineering modeling.

Original: However, most of the existing methods rely on two rules: (1) learn a sparse representation to ﬁt data and (2) check if the loss objective function satisﬁes error thresholds.
Revised: However, most of the existing methods rely on two rules: (1) learn a sparse representation to fit data and (2) check if the loss objective function satisfies error thresholds.

Original: This paper illustrates that such conditions are far from sufﬁcient.
Revised: This paper illustrates that such conditions are far from sufficient.

Original: Speciﬁcally, we show that sparse non-physical approximations exist with excellent ﬁtting accuracy, but fail to adequately model the situation.
Revised: Specifically, we show that sparse non-physical approximations exist with excellent fitting accuracy, but fail to adequately model the situation.

Original: To fundamentally resolve the data-ﬁtting problem, we propose a physical neural network (PNN) utilizing “Range, Inertia, Symmetry, and Extrapolation” (RISE) constraints.
Revised: To fundamentally resolve the data-fitting problem, we propose a physical neural network (PNN) utilizing “Range, Inertia, Symmetry, and Extrapolation” (RISE) constraints.

Original: RISE is based on a complete analysis for the generalizability of data properties for physical systems.
Revised: RISE is based on a complete analysis for the generalizability of data properties for physical systems.

Original: The ﬁrst three techniques focus on the deﬁnition of physics in space and time.
Revised: The first three techniques focus on the definition of physics in space and time.

Original: The last technique of extrapolation is novel based on active learning without an inquiry, using cross-model validation.
Revised: The last technique of extrapolation is novel based on active learning without an inquiry, using cross-model validation.

Original: We validate the proposed PNN-RISE method via a synthetic dataset, power system dataset, and mass-damper system dataset.
Revised: We validate the proposed PNN-RISE method via a synthetic dataset, power system dataset, and mass-damper system dataset.

Original: Numerical results show the universal capability of the PNN-RISE approach to quickly identify the hidden physical models without local optima, opening the door for the fast and highly accurate discovery of the physical laws or systems with external loads.
Revised: Numerical results show the universal capability of the PNN-RISE approach to quickly identify the hidden physical models without local optima, opening the door for the fast and highly accurate discovery of the physical laws or systems with external loads.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Internet of Everything (IoE) connects new edge devices into an intelligent web at an unprecedented speed.
Revised: Internet of Everything (IoE) connects new edge devices into an intelligent web at an unprecedented speed.

Original: If utilized systematically, IoE can create much more efﬁcient productivity via coordinated effort among humans, processes, data, and things (Li et al., 2020).
Revised: If utilized systematically, IoE can create much more efficient productivity via coordinated effort among humans, processes, data, and things (Li et al., 2020).

Original: Among various analytical frameworks, dynamic data-driven applications systems (DDDAS) naturally interweaves diversiﬁed information via data-driven analysis to assimilate the system model for a wide range of applications (Blasch et al., 2018), thus signiﬁcantly increasing the performance of both the simulation and application models.
Revised: Among various analytical frameworks, dynamic data-driven applications systems (DDDAS) naturally interweaves diversified information via data-driven analysis to assimilate the system model for a wide range of applications (Blasch et al., 2018), thus significantly increasing the performance of both the simulation and application models.

Original: The simulation model usually incorporates physical methods (Nasiakou et al., 2018; Chowdhury & Subramani, 2020) which provides a structure for model-based and data-driven coordination.
Revised: The simulation model usually incorporates physical methods (Nasiakou et al., 2018; Chowdhury & Subramani, 2020) which provides a structure for model-based and data-driven coordination.

Original: However, many of the physical laws in the web of IoE can be unknown, which remain to be discovered manually or automatically to enable down-streaming applications.
Revised: However, many of the physical laws in the web of IoE can be unknown, which remain to be discovered manually or automatically to enable down-streaming applications with accurate modeling.

Original: The most economically efﬁcient way of physical law discovery is to use machine learning to analyze the pattern behind the data stream, which is an important topic (Sahoo et al., 2018; Udrescu &
Revised: The most economically efficient way of physical law discovery is to use machine learning to analyze the pattern behind the data stream, which is an important topic (Sahoo et al., 2018; Udrescu &

Original: Tegmark, 2020).
Revised: Tegmark, 2020).

Original: In previous studies, such as symbolic regression (Petersen, 2019), the sparsity representation is used to measure the accountability of physics (Brunton et al., 2016).
Revised: In previous studies, such as symbolic regression (Petersen, 2019), the sparsity representation is used to measure the accountability of physics (Brunton et al., 2016).

Original: The underlying assumption is that sparsity can ensure physical exactness due to parsimonious principle (Li, 2013).
Revised: The underlying assumption is that sparsity can ensure physical exactness due to parsimonious principle (Li, 2013).

Original: While low complexity of learning models is a necessary condition for enhancing the understanding of physical laws (Sch¨olkopf et al., 2002; Blasch et al., 2021), there is not a universal sufﬁcient condition to solve model recovery problem with a guarantee.
Revised: While low complexity of learning models is a necessary condition for enhancing the understanding of physical laws (Sch¨olkopf et al., 2002; Blasch et al., 2021), there is not a universal sufficient condition to solve model recovery problem with a guarantee.

Original: Therefore, one approach is to add additional conditions to ensure a physical recovery, e.g., near-zero ﬁtting errors (Champion et al., 2019) or/and cross validation (Kim et al., 2020).
Revised: Therefore, one approach is to add additional conditions to ensure a physical recovery, e.g., near-zero fitting errors (Champion et al., 2019) or/and cross validation (Kim et al., 2020).

Original: However, it can be observed that different models easily ﬁt the available data with a sparse but wrong model, which can happen even with cross-validation.
Revised: However, it can be observed that different models easily fit the available data with a sparse but wrong model, which can happen even with cross-validation.

Original: The overﬁtting should challenge the directions of many researchers in physical system discovery, who believe that only the true physical laws can achieve sparse recovery with a perfect ﬁt.
Revised: The overfitting should challenge the directions of many researchers in physical system discovery, who believe that only the true physical laws can achieve sparse recovery with a perfect fit.

Original: To resolve the problem, we ﬁrst eliminate non-physical local optima of the neural network (NN) by discarding all the non-physical parameters, i.e., model parameters indirectly related to the physical representation.
Revised: To resolve the problem, we first eliminate non-physical local optima of the neural network (NN) by discarding all the non-physical parameters, i.e., model parameters indirectly related to the physical representation.

Original: The process is novel as existing NN-based methods require certain non-physical parameters to control the representation sparsity (Liu & Laili, 2018; Petersen, 2019).
Revised: The process is novel as existing NN-based methods require certain non-physical parameters to control the representation sparsity (Liu & Laili, 2018; Petersen, 2019).

Original: We show how to design a NN with pure physical parameters, entitled a physical neural network (PNN).
Revised: We show how to design a NN with pure physical parameters, entitled a physical neural network (PNN).

Original: The key of PNN is to enforce activation to physical weights for sparse selection, completely removing the non-physical parameters and local optima.
Revised: The key of PNN is to enforce activation to physical weights for sparse selection, completely removing the non-physical parameters and local optima.

Original: In training the PNN, there can still be local optimum solutions, bus actual knowledge  can enhance model ﬁdelity.
Revised: In training the PNN, there can still be local optimum solutions, the actual knowledge of physical system can enhance model fidelity.

Original: For obtaining the ground truth, physical systems’ examination can be used to extract common and universal properties.
Revised: To obtain the ground truth, physical system examination can be used to extract common and universal properties.

Original: Subsequently, the PNN is restricted to systematically obey the physicaland system properties.
Revised: Subsequently, the PNN is restricted to systematically obey the physical system properties.

Original: Speciﬁcally, constraints based on the physical distance andphysical symmetry are embedded in the training function to quantify the spatial relationship enforced by physical law.
Revised: Specifically, constraints based on the physical distance and symmetry are embedded in the training function to quantify the spatial relationship enforced by physical law.

Original: Furthermore, as physical systems evolve over time, inertial property for physical systems are extracted and updated.
Revised: Furthermore, as physical systems evolve over time, inertial property for physical systems are extracted and updated.

Original: In general, the regularization from physical laws focuses the learning/optimization towards a more meaningful ﬁtness understanding of the given data.
Revised: In general, the regularization from physical laws focuses the learning/optimization towards a more meaningful fitness understanding of the given data.

Original: Although the two designs of PNN and physics constraints aim to obtain the physical laws via ﬁtting existing data, the incorporation of physical laws’ biggest impact is its capability to achieve zero ﬁtting errors even beyond the training data range.
Revised: Although the two designs of PNN and physics constraints aim to obtain the physical laws via fitting existing data, the incorporation of physical laws’ biggest impact is its capability to achieve zero fitting errors even beyond the training data range.

Original: Unfortunately, it will be costly if not impossible to conduct active learning for every Internet of Things (IoT) system due to its scale.
Revised: Unfortunately, it will be costly if not impossible to conduct active learning for every Internet of Things (IoT) system due to its scale.

Original: To provide the “free lunch” of adding the extrapolation capability of PNN, we propose an new learning method for cross-model validation based on probabilistic estimates with physical kernels.
Revised: To provide the “free lunch” of adding the extrapolation capability of PNN, we propose a new learning method for cross-model validation based on probabilistic estimates with physical kernels.

Original: We use a physically meaningful forecast to identify out-of-sample points with a conﬁdence level.
Revised: We use a physically meaningful forecast to identify out-of-sample points with a confidence level.

Original: Then, the results are integrated as a weak supervision for learning the exact physics for extrapolation, in addition to the constrained PNN.
Revised: Then, the results are integrated as a weak supervision for learning the exact physics for extrapolation, in addition to the constrained PNN.

Original: Together, PNN-RISE uses the PNN container to include the RISE principles, enforcing the recovered function to satisfy constraints on range (R), inertia (I), symmetry (S), and extrapolation (E).
Revised: Together, PNN-RISE uses the PNN container to include the RISE principles, enforcing the recovered function to satisfy constraints on range (R), inertia (I), symmetry (S), and extrapolation (E).

Original: Empirically, extensive validations show that the PNN-RISE outperforms the state-of-the-art methods.
Revised: Empirically, extensive validations show that the PNN-RISE outperforms the state-of-the-art methods.

Original: Consistently, scenarios with or without each constraint in RISE validate the approach.
Revised: Consistently, scenarios with or without each constraint in RISE validate the approach.

Original: Furthermore, for each constraint, we quantify its strength with different levels, highlighting that loose constraints are enough to obtain high accuracy in the proposed PNN.
Revised: Furthermore, for each constraint, we quantify its strength with different levels, highlighting that loose constraints are enough to obtain high accuracy in the proposed PNN.

Original: In summary, the contributions in Fig.
Revised: In summary, the contributions in Fig.

Original: 1 are 1) showing that the sparse recovery with low training and testing errors does not ensure physical law recovery, 2) reducing the possibilities of local optima signiﬁcantly by eliminating non-physical parameters in learning model selection and design, 3) proposing four physical principles to completely characterize physical systems via constrained learning, and 4) creating a novel learning method for physical systems extrapolation as probabilistic forecasting out-of-sample cross-model validation (FOCV).
Revised: 1 are 1) showing that the sparse recovery with low training and testing errors does not ensure physical law recovery, 2) reducing the possibilities of local optima significantly by eliminating non-physical parameters in learning model selection and design, 3) proposing four physical principles to completely characterize physical systems via constrained learning, and 4) creating a novel learning method for physical systems extrapolation as probabilistic forecasting out-of-sample cross-model validation (FOCV).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Physical System Identiﬁcation.
Revised: Physical System Identification.

Original: To recover system information, supervised learning (SL) can approximate input-output relationships for inferences (Huang et al., 2006; Hornik et al., 1990).
Revised: To recover system information, supervised learning (SL) can approximate input-output relationships for inferences (Huang et al., 2006; Hornik et al., 1990).

Original: They show good numerical performance in non-physical and physical applications, including computer vision tasks and natural language processing (Lu & Weng, 2007;
Revised: They show good numerical performance in non-physical and physical applications, including computer vision tasks and natural language processing (Lu & Weng, 2007;

Original: Wang et al., 2017a; Karpathy et al., 2014; Nadkarni et al., 2011; Hirschberg & Manning, 2015).
Revised: Wang et al., 2017a; Karpathy et al., 2014; Nadkarni et al., 2011; Hirschberg & Manning, 2015).

Original: Deep neural networks (DNNs) boost classiﬁcation accuracy (Hinton et al., 2006; Krizhevsky et al., 2012) by using latent layers to extract nonlinear features hidden in images.
Revised: Deep neural networks (DNNs) boost classification accuracy (Hinton et al., 2006; Krizhevsky et al., 2012) by using latent layers to extract nonlinear features hidden in images.

Original: While it makes SL useful for non-physical systems, physical systems require a much higher accuracy for operation or control, e.g., near 100% for Six
Revised: While it makes SL useful for non-physical systems, physical systems require a much higher accuracy for operation or control, e.g., near 100% for Six

Original: Sigma performance (Walshe et al., 2010), so that operators can safely operate the system to avoid outages or cascading failures (Chung et al., 2018; Liu et al., 2017).
Revised: Sigma performance (Walshe et al., 2010), so that operators can safely operate the system to avoid outages or cascading failures (Chung et al., 2018; Liu et al., 2017).

Original: Even worse, many SL models are black boxes that lack physical interpretability for reliable operations, as well as certiﬁability for performance standards (Mestav et al., 2018;
Revised: Even worse, many SL models are black boxes that lack physical interpretability for reliable operations, as well as certifiability for performance standards (Mestav et al., 2018;

Original: Du & Li, 2019; Hu et al., 2020; Baghaee et al., 2017).
Revised: Du & Li, 2019; Hu et al., 2020; Baghaee et al., 2017).

Original: Therefore, some works ﬁnd utilizing contextual information improves physical-system identiﬁcation where they assume a good knowledge of physical basis and sufﬁcient measurements across the system (Yu et al., 2017; Chen et al., 2020).
Revised: Therefore, some works find utilizing contextual information improves physical-system identification where they assume a good knowledge of physical basis and sufficient measurements across the system (Yu et al., 2017; Chen et al., 2020).

Original: Similarly, (Brunton et al., 2016; Champion et al., 2019;
Revised: Similarly, (Brunton et al., 2016; Champion et al., 2019;

Original: Li & Weng, 2021) assume prior symbols are known in a dictionary and introduce sparse regression to select symbols.
Revised: Li & Weng, 2021) assume prior symbols are known in a dictionary and introduce sparse regression to select symbols.

Original: However, the assumptions are too limiting due to the difﬁculty of knowing all the physical basis for arbitrary complex systems (Wang et al., 2017b; Tarca et al., 2007).
Revised: However, the assumptions are too limiting due to the difficulty of knowing all the physical basis for arbitrary complex systems (Wang et al., 2017b; Tarca et al., 2007).

Original: One ad-hoc approach (Schmidt & Lipson, 2009; Petersen, 2019) uses a genetic algorithm to search for the symbolic basis but the problem is NP-hard without any guarantee on recovering the governing physical laws from empirical data.
Revised: One ad-hoc approach (Schmidt & Lipson, 2009; Petersen, 2019) uses a genetic algorithm to search for the symbolic basis but the problem is NP-hard without any guarantee on recovering the governing physical laws from empirical data.

Original: Thus, we investigate whether the sparsity technique and a small modeling error are sufﬁcient to claim the success of learning the physical model.
Revised: Thus, we investigate whether the sparsity technique and a small modeling error are sufficient to claim the success of learning the physical model.

Original: Moreover, how can analytical techniques ensure exact physics learning for domain-speciﬁc systems?
Revised: Moreover, how can analytical techniques ensure exact physics learning for domain-specific systems?

Original: Constrained Machine Learning.
Revised: Constrained Machine Learning.

Original: There are many applications of constrained machine learning (CML) in different systems.
Revised: There are many applications of constrained machine learning (CML) in different systems.

Original: (Small et al., 2011) restricts the weight vectors with domain knowledge in support vector machine and (Li et al., 2021) proposes constrained adversarial learning to generate adversarial examples under physical constraints.
Revised: (Small et al., 2011) restricts the weight vectors with domain knowledge in support vector machine and (Li et al., 2021) proposes constrained adversarial learning to generate adversarial examples under physical constraints.

Original: In deep learning applications, (Zhao et al., 2019) employs constraints from conservation law to restrict the output variables of DNN.
Revised: In deep learning applications, (Zhao et al., 2019) employs constraints from conservation law to restrict the output variables of DNN.

Original: (Velloso & Van Hentenryck, 2020; Fioretto et al., 2020a) solve optimal power ﬂow problems using DNN with physical constraints, where the output variables are biased to guarantee feasible solution.
Revised: (Velloso & Van Hentenryck, 2020; Fioretto et al., 2020a) solve optimal power flow problems using DNN with physical constraints, where the output variables are biased to guarantee feasible solution.

Original: (Tran et al., 2020) restricts the input to DNN into different groups to guarantee privacy.
Revised: (Tran et al., 2020) restricts the input to DNN into different groups to guarantee privacy.

Original: Among these studies, (Fioretto et al., 2020b) optimizes DNN with output constraints using Lagrangian duality.
Revised: Among these studies, (Fioretto et al., 2020b) optimizes DNN with output constraints using Lagrangian duality.

Original: These CML methods can solve their own problems effectively; however, there is a need for general principles to embed the physical constraints systematically with efﬁcient solutions.
Revised: These CML methods can solve their own problems effectively; however, there is a need for general principles to embed the physical constraints systematically with efficient solutions.

Original: Data Forecasting.
Revised: Data Forecasting.

Original: For previous data forecasting methods, autoregressive moving average (ARMA) and autoregressive integrated moving average (ARIMA) predict the evolving variables based on their lagged values (Gilbert, 2005).
Revised: For previous data forecasting methods, autoregressive moving average (ARMA) and autoregressive integrated moving average (ARIMA) predict the evolving variables based on their lagged values (Gilbert, 2005).

Original: To capture nonlinear correlations in the short-term and long-term, there are many deep learning-based methods, e.g., recurrent Neural Network (RNN), long shortterm memory (LSTM), bidirectional LSTM (BiLSTM), gated recurrent units (GRUs), and variational autoencoder (VAE) (Zeroual et al., 2020).
Revised: To capture nonlinear correlations in the short-term and long-term, there are many deep learning-based methods, e.g., recurrent Neural Network (RNN), long shortterm memory (LSTM), bidirectional LSTM (BiLSTM), gated recurrent units (GRUs), and variational autoencoder (VAE) (Zeroual et al., 2020).

Original: To forecast the result with conﬁdence, methods like Gaussian process (GP) (Quinonero-Candela & Rasmussen, 2005), Quantile Regression (Maciejowska et al., 2016) and deep learning-based models (Salinas et al., 2020) afford an efﬁcient way to predict distributions for the future data.
Revised: To forecast the result with confidence, methods like Gaussian process (GP) (Quinonero-Candela & Rasmussen, 2005), Quantile Regression (Maciejowska et al., 2016) and deep learning-based models (Salinas et al., 2020) afford an efficient way to predict distributions for the future data.

Original: Since the uncertainty evaluation is important to integrate the forecast data into the physical learning process, probabilistic methods are presented to estimate probability density functions in the proposed FOCV.
Revised: Since the uncertainty evaluation is important to integrate the forecast data into the physical learning process, probabilistic methods are presented to estimate probability density functions in the proposed FOCV.

Original:  M ETHODS
Revised:  M ETHODS

Original: The PNN-RISE approach is to understand the reason why a “good” sparse ﬁtting is not physical veriﬁable, to design physical principles to guide the ﬁtting internally based on a ﬂexible PNN structure, and to create a cost-free active learning method for extrapolation at unseen operating points.
Revised: The PNN-RISE approach is to understand the reason why a “good” sparse fitting is not physical verifiable, to design physical principles to guide the fitting internally based on a flexible PNN structure, and to create a cost-free active learning method for extrapolation at unseen operating points.

Original: Symbolic regression (SR) is a common approach to ﬁnd equation operators ( { + , − , × , / } ) and expressions like ( x 2 , cos( x )) , etc.
Revised: Symbolic regression (SR) is a common approach to find equation operators ( { + , − , × , / } ) and expressions like ( x 2 , cos( x )) , etc.

Original: Existing papers (Brunton et al., 2016; Sahoo et al., 2018; Champion et al., 2019; Petersen, 2019;
Revised: Existing papers (Brunton et al., 2016; Sahoo et al., 2018; Champion et al., 2019; Petersen, 2019;

Original: Kim et al., 2020) show that if the following three conditions are satisﬁed, SR is capable to ﬁnd the true equation: 1) apply the sparse regularization, 2) achieve nearly zero training error, and 3) utilize cross validation to avoid overﬁtting.
Revised: Kim et al., 2020) show that if the following three conditions are satisfied, SR is capable to find the true equation: 1) apply the sparse regularization, 2) achieve nearly zero training error, and 3) utilize cross validation to avoid overfitting.

Original: However, we observe that SR methods are prone to non-physical solutions even when all three assumptions are satisﬁed.
Revised: However, we observe that SR methods are prone to non-physical solutions even when all three assumptions are satisfied.

Original: For example, to learn the function y =
Revised: For example, to learn the function y =

Original: x 1 x 2 + x 2 x 3 from data, different SR methods can easily get trapped in local optima.
Revised: x 1 x 2 + x 2 x 3 from data, different SR methods can easily get trapped in local optima.

Original: To see if these observation is special, we test different cases, where Fig.
Revised: To see if these observations are special, we test different cases, where Fig.

Original: 2 shows a table summary of four different cases, with integer coefﬁcients or not.
Revised: 2 shows a table summary of four different cases, with integer coefficients or not.

Original: We also test different numbers for multiplication and summation.
Revised: We also test different numbers for multiplication and summation.

Original: From all examples, one can see that the recovered objective has sparse coefﬁcients and negligible error (loss objective L ) at non-desirable local optimums.
Revised: From all examples, one can see that the recovered objective has sparse coefficients and negligible error (loss objective L ) at non-desirable local optimums.

Original: Other than the toy examples, we conduct a realistic power system case study and visualize the objective surface in Appendix A. Similarly, for a small parameter range, there are points achieving the global optimum or local optima that are very close to the global optimum.
Revised: Other than the toy examples, we conduct a realistic power system case study and visualize the objective surface in Appendix A. Similarly, for a small parameter range, there are points achieving the global optimum or local optima that are very close to the global optimum.

Original: However, only one of them is the physical ground truth.
Revised: However, only one of them is the physical ground truth.

Original: For example, one can go from the red point to a non-physical solution with an error close to zero, even when the sparsity regularization is enforced.
Revised: For example, one can go from the red point to a non-physical solution with an error close to zero, even when the sparsity regularization is enforced.

Original: Therefore, we can see that “good” sparse ﬁtting is insufﬁcient to avoid overﬁtting.
Revised: Therefore, we can see that “good” sparse fitting is insufficient to avoid overfitting.

Original: Thus, we need to avoid overﬁtting for physical systems not only by choosing sparse learning models , but also new ways to constrain the learning process and avoid non-physical solutions.
Revised: Thus, we need to avoid overfitting for physical systems not only by choosing sparse learning models , but also new ways to constrain the learning process and avoid non-physical solutions.

Revised: 3. P ROPOSED METHOD : PHYSICAL NEURAL NETWORK WITH WEIGHT ACTIVATION

Original: For complex physical systems, the governing functions are formed by basic operations such as additive and multiplicative accumulations of variables.
Revised: For complex physical systems, the governing functions are formed by basic operations such as additive and multiplicative accumulations of variables.

Original: For example, the topology of power system can be represented as a graph, where the interaction between different nodes along the lines is represented by polynomials of the voltages for power ﬂow (Yu et al., 2017).
Revised: For example, the topology of power system can be represented as a graph, where the interaction between different nodes along the lines is represented by polynomials of the voltages for power flow (Yu et al., 2017).

Original: To combine the beneﬁt of universal approximation and the freedom to choose activation function, we propose to use a NN as a container to learn the representation of physical equations, and thus call it a physical neural network (PNN), denoted as f .
Revised: To combine the benefit of universal approximation and the freedom to choose activation function, we propose to use a NN as a container to learn the representation of physical equations, and thus call it a physical neural network (PNN), denoted as f .

Original: To train f , the training data set is { x i , y i } Ni =1 , where x i and y
Revised: To train f , the training data set is { x i , y i } Ni =1 , where x i and y

Original: i are the i th input and output of f , and N is the number of training samples.
Revised: i are the i th input and output of f , and N is the number of training samples.

Original: To construct the physical equation in PNN, the key is to develop a sparse selection mechanism to automatically select part of the variables to join summation/abstraction and multiplication/division.
Revised: To construct the physical equation in PNN, the key is to develop a sparse selection mechanism to automatically select part of the variables to join summation/subtraction and multiplication/division.

Original: For sparse selection, existing work utilizes sparse regularization like LASSO for weight updates (Kim et al., 2020), introduces extra neural network modules like gates (Liu & Laili, 2018), or conducts symbolic generators (Petersen, 2019).
Revised: For sparse selection, existing work utilizes sparse regularization like LASSO for weight updates (Kim et al., 2020), introduces extra neural network modules like gates (Liu & Laili, 2018), or conducts symbolic generators (Petersen, 2019).

Original: In general, they demand the quantiﬁcation of extra hyper-parameters (i.e., the LASSO penalty) or parameters (i.e., parameters in gates or symbolic generators), which signiﬁcantly increases the possibility of local optima.
Revised: In general, they demand the quantification of extra hyper-parameters (i.e., the LASSO penalty) or parameters (i.e., parameters in gates or symbolic generators), which significantly increases the possibility of local optima.

Original: In order to address the problem towards limiting local optima, we advocate the design philosophy: enforce activation functions on weights for the sparse selection of features.
Revised: In order to address the problem towards limiting local optima, we advocate the design philosophy: enforce activation functions on weights for the sparse selection of features.

Original: Thus, no extra parameters are added, and the searching is only for physical parameters.
Revised: Thus, no extra parameters are added, and the searching is only for physical parameters.

Original: To elaborate on our design, we need to answer (1) which parameters need to be activated and (2) how to enforce the activation.
Revised: To elaborate on our design, we need to answer (1) which parameters need to be activated and (2) how to enforce the activation.

Original: For parameter activation, we start from the basic symbolic operations in neural networks, e.g., multiplications and divisions of the form z j = Π Ni x ip ij , where z j is the j th element of the layer next to the input.
Revised: For parameter activation, we start from the basic symbolic operations in neural networks, e.g., multiplications and divisions of the form z j = Π Ni x ip ij , where z j is the j th element of the layer next to the input.

Original: x i is the i th element of the input vector variable x and p ij is the index number  of x i .
Revised: x i is the i th element of the input vector variable x and p ij is the polynomial order number (power) of x i .

Original: Note p ij ∈ {− K, · · · , − 1 , 0 , 1 , · · · , K } , where the positive integer represents the multiplication, the possibility of 0 corresponds to the sparsity, and the negative integer represents the division.
Revised: Note p ij ∈ {− K, · · · , − 1 , 0 , 1 , · · · , K } , where the positive integer represents the multiplication, the possibility of 0 corresponds to the sparsity, and the negative integer represents the division.

Original: Then, each z j is a more complex symbol for the next operation and p ij is the optimization variable.
Revised: Then, each z j is a more complex symbol for the next operation and p ij is the optimization variable.

Original: To accommodate p ij into a NN, we adopt logarithm and exponential operators to achieve a linear summation format z j = exp (cid:0) (cid:80) Ni =1 p ij ln
Revised: To accommodate p ij into a NN, we adopt logarithm and exponential operators to achieve a linear summation format z j = exp (cid:0) (cid:80) Ni =1 p ij ln

Original: ( x i ) (cid:1) (Trask et al., 2018).
Revised: ( x i ) (cid:1) (Trask et al., 2018).

Original: Thus, the linear layer with weights p ij needs to be activated for arithmetic operations and sparsity.
Revised: Thus, the linear layer with weights p ij needs to be activated for arithmetic operations and sparsity.

Original: Considering the target integer range of p ij , it’s vital to know how to control the values for parameterenforcement activation.
Revised: Considering the target integer range of p ij , it’s vital to know how to control the values for parameter activation.

Original: We ﬁrst show how to enforce p ij ∈ {− 1 , 0 , 1 } .
Revised: We first show how to enforce p ij ∈ {− 1 , 0 , 1 } .

Original: Since tanh function can restrict the output value to be in [ − 1 , 1] , we can utilize a positive scaling factor c 1 and tanh to activate some free and optimizable parameter w ij such that tanh ( c 1 w ij ) ∈ {− 1 , 1 } , where the scaling is to make sure the output is very close to the target − 1 or 1 .
Revised: Since tanh function can restrict the output value to be in [ − 1 , 1] , we can utilize a positive scaling factor c 1 and tanh to activate some free and optimizable parameter w ij such that tanh ( c 1 w ij ) ∈ {− 1 , 1 } , where the scaling is to make sure the output is very close to the target − 1 or 1 .

Original: Subsequently, the network needs to enforce 0 for sparsity.
Revised: Subsequently, the network needs to enforce 0 for sparsity.

Original: We similarly create another sigmoid function to achieve σ ( c 2 w (cid:48) ij ) ∈ { 0 , 1 } , where σ is the sigmoid function, c 2 is the positive scaling factor, and w (cid:48) ij is another optimizable parameter.
Revised: We similarly create another sigmoid function to achieve σ ( c 2 w ′ ij ) ∈ { 0 , 1 } , where σ is the sigmoid function, c 2 is the positive scaling factor, and w ′ ij is another optimizable parameter.

Original: To summarize, we have p ij = tanh ( c 1 w ij ) · σ ( c 2 w (cid:48) ij ) , where p ij ∈ ( − 1 , 1) .
Revised: To summarize, we have p ij = tanh ( c 1 w ij ) · σ ( c 2 w ′ ij ) , where p ij ∈ ( − 1 , 1) .

Original: The multiplication of tanh and sigmoid functions not only generates a smooth optimization surface but also naturally results in limited parameters.
Revised: The multiplication of tanh and sigmoid functions not only generates a smooth optimization surface but also naturally results in limited parameters.

Original: After that, a third linear layer is used to obtain y k
Revised: After that, a third linear layer is used to obtain y k

Original: = (cid:80) j m jk z j , where m jk denote the layer weights for linear summation.
Revised: = (cid:80) j m jk z j , where m jk denote the layer weights for linear summation.

Original: Since the summation and abstraction are incorporated into the linear layer, we complete the function y = f
Revised: Since the summation and abstraction are incorporated into the linear layer, we completely represent y = f

Original: ( x ) .
Revised: ( x ) .

Original: Using such a design as a basic unit, we can extend from {− 1 , 0 , 1 } to {− K, · · · , − 1 , 0 , 1 , · · · , K } for different physical systems.
Revised: Using such a design as a basic unit, we can extend from {− 1 , 0 , 1 } to {− K, · · · , − 1 , 0 , 1 , · · · , K } for different physical systems.

Original: Such a design converts the non-convex multiplication to a convex form of linear summation for NN training.
Revised: Such a design converts the non-convex multiplication to a convex form of linear summation for NN training.

Original: Based on this design, we show the following property of our PNN to guarantee the ﬁnding of stationary points in the optimization.
Revised: Based on this design, we show the following property of our PNN to guarantee the finding of stationary points in the optimization.

Original: The following proposition states the ﬁnding of stationary points using PNN with the design and the proof is inlcuded in Appendix.
Revised: The following proposition states the finding of stationary points using PNN with the design and the proof is included in Appendix.

Original: Proposition 1 (Biconvexity of PNN) .
Revised: Proposition 1 (Biconvexity of PNN) .

Original: PNN is biconvex on the parameters of the logarithmlayer and the linear summation layer.
Revised: PNN is biconvex on the parameters of the logarithm and the linear summation layers.

Original: Then, the searching algorithm can ﬁnd the stationary points of PNN.
Revised: The stationary points for physical function can be found in optimization.

Original: Proposition 1 indicates that PNN can ﬁnd the stationary points (i.e., saddle points, local optima, and global optima).
Revised: Proposition 1 indicates that PNN can find the stationary points (i.e., saddle points, local optima, and global optima).

Original: Subsequently, can we ﬁnd the global optima and can it represent the true physics?
Revised: Subsequently, can we find the global optima and can it represent the true physics?

Original: 3. E MBED PHYSICAL PRINCIPLES VIA CONSTRAINING PNN
Revised: 3. E MBED PHYSICAL PRINCIPLES VIA CONSTRAINING PNN

Original: After providing a ﬂexible design to limit possibilities of local optima, we need comprehensive principles to avoid non-physical local optima.
Revised: After providing a flexible design to limit possibilities of local optima, we need comprehensive principles to avoid non-physical local optima.

Original: If the derived constraints can be convex, they will provide guarantees in solving or reducing the issue of non-physical local optima.
Revised: If the derived constraints can be convex, they will provide guarantees in solving or reducing the issue of non-physical local optima.

Original: For physical systems, two important perspectives are physical existence and systematic behaviors.
Revised: For physical systems, two important perspectives are physical existence and systematic behaviors.

Original: As the components are physically located, there are geographical or electrical distances, etc.
Revised: As the components are physically located, there are geographical or electrical distances, etc.

Original: Therefore, one common constraint through physical systems is the symmetric (S) property, e.g., a branch connecting two components represents a common coupling.
Revised: Therefore, one common constraint through physical systems is the symmetric (S) property, e.g., a branch connecting two components represents a common coupling.

Original: In addition, the links between different components also have their physical limitations, leading to a range (R) constraint.
Revised: In addition, the links between different components also have their physical limitations, leading to a range (R) constraint.

Original: At the system level, the physical system will continue making an impact to the environment, leading to an inertia (I) concept, which is the momentum from a physical object because physical systems typically have natural inertia, whether it is strong or weak.
Revised: At the system level, the physical system will continue making an impact to the environment, leading to an inertia (I) concept, which is the momentum from a physical object because physical systems typically have natural inertia, whether it is strong or weak.

Original: For man-made systems, there could be additional artiﬁcial inertia due to factors such as guaranteeing stability for sustainable operation, e.g., in power systems.
Revised: For man-made systems, there could be additional artificial inertia due to factors such as guaranteeing stability for sustainable operation, e.g., in power systems.

Original: Finally, such inertia does not only live in the temporal space but also in the input/output data space.
Revised: Finally, such inertia does not only live in the temporal space but also in the input/output data space.

Original: For example, the physical equations are typically continuous functions.
Revised: For example, the physical equations are typically continuous functions.

Original: These concepts open the door to investigate the data range beyond the training and testing data.
Revised: These concepts open the door to investigate the data range beyond the training and testing data.

Original: Such generalized inertia leads  an important extrapolation (E) property across different physical systems to forecast information yet to be observed but representative of plausible scenarios.
Revised: Such generalized inertia leads to an important extrapolation (E) property across different physical systems to forecast information yet to be observed but representative of plausible scenarios.

Original: The constraints based on different properties of physical systems are called RISE concept by combing their initials, which is demonstrated via a ﬁgure in Appendix.
Revised: The constraints based on different properties of physical systems are called RISE concept by combining their initials, which is demonstrated via a figure in Appendix.

Original: This subsection introduces the ﬁrst three to constrain PNN in
Revised: This subsection introduces the first three to constrain PNN in

Original: Subsection 3.1 while Subsection 3.3 introduces the FOCV method for physical system extrapolation.
Revised: Subsection 3.1 while Subsection 3.3 introduces the FOCV method for physical system extrapolation.

Original: Range (R): Shape Constraint on Physical Parameters: While local optima generate many possible parameters to minimize the prediction error, the true physical parameters of corresponding features usually yield speciﬁc shapes (Cotter et al., 2019).
Revised: Range (R): Shape Constraint on Physical Parameters: While local optima generate many possible parameters to minimize the prediction error, the true physical parameters of corresponding features usually yield specific shapes (Cotter et al., 2019).

Original: For example, the range of power line admittance (reciprocal of impedance) of power distribution systems can be speciﬁed with voltage level and line materials.
Revised: For example, the range of power line admittance (reciprocal of impedance) of power distribution systems can be specified with voltage level and line materials.

Original: In addition to the bound on value range of independent parameters, there can also be shape constraints on the conditional relationships among parameters, e.g., longer lines have larger resistance than shorter lines.
Revised: In addition to the bound on value range of independent parameters, there can also be shape constraints on the conditional relationships among parameters, e.g., longer lines have larger resistance than shorter lines.

Original: Such a rule speciﬁed by the physical shape constraint function can ﬁlter out many local optima even with low errors.
Revised: Such a rule specified by the physical shape constraint function can filter out many local optima even with low errors.

Original: In order to enforce the learned physical parameters W p within the constraint, a customized gate function is proposed toweights g ( · ) is a function of physical weights W p to represent the type of shape constraint.
Revised: In order to enforce the learned physical parameters W p within the constraint, a customized gate function is proposed to is a function of physical weights W p to represent the type of shape constraint.

Original: As an example, g ( · ) is linear for speciﬁc value constraint of independent parameters, where (cid:98) g p (cid:99) and (cid:100) g p (cid:101) are the lower and upper bounds, and g (cid:48) ( W p ) are the constrained weights.
Revised: As an example, g ( · ) is linear for specific value constraint of independent parameters, where ⌊ g p ⌋ and ⌈ g p ⌉ are the lower and upper bounds, and g ′ ( W p ) are the constrained weights.

Original: Symmetry (S): Constraints on Physical Couplings: For physical systems with multiple outputs, there are variable couplings among different outputs, e.g., one term is contained in two outputs.
Revised: Symmetry (S): Constraints on Physical Couplings: For physical systems with multiple outputs, there are variable couplings among different outputs, e.g., one term is contained in two outputs.

Original: Let’s use the power system as an example.
Revised: Let’s use the power system as an example.

Original: A certain amount of power ﬂowing from one place to another means that the same amount of power will be subtracted and added simultaneously at the two places.
Revised: A certain amount of power flowing from one place to another means that the same amount of power will be subtracted and added simultaneously at the two places.

Original: Based on our observations, most of the local optima of NN do not satisfy such a rule.
Revised: Based on our observations, most of the local optima of NN do not satisfy such a rule.

Original: Thus, the property is embedded by minimizing L Mutual = [-(cid:107) W p − W Tp (cid:107) 
-]
Revised: Thus, the property is embedded by minimizing L Mutual = [-(cid:107) W p − W Tp (cid:107) 
-]∥

Original:  + (cid:107) rank ( W pi ) − N mutual (cid:107) 0 .
Revised: W p − W Tp ∥ 2 + ∥ rank ( W pi ) − N mutual ∥ 1 .

Original: The ﬁrst term is an l 2 norm penalty on the symmetry of recovered physical parameters and the second term is to further ensure that the number of mutual terms N mutual satisﬁes the system property, e.g., power system equation has N mutual = 2 , because two nodes are connected by one power line.
Revised: The first term is an l 2 norm penalty on the symmetry of recovered physical parameters and the second term is to further ensure that the number of mutual terms N mutual satisfies the system property, e.g., power system equation has N mutual = 2 , because two nodes are connected by one power line.

Revised: For practical implementation, the l 0 norm is relaxed to l 1 norm in optimization.

Original: Inertia (I): Constraints on Physical Tendency: For physical systems, the states are typically continuous and cannot change suddenly, leading to physical inertia.
Revised: Inertia (I): Constraints on Physical Tendency: For physical systems, the states are typically continuous and cannot change suddenly, leading to physical inertia.

Original: Such a property can be embedded by bounding the outputs of two adjacent time slots.
Revised: Such a property can be embedded by bounding the outputs of two adjacent time slots.

Original: To ensure the partial convexity, the Euclidean norm is modeled as a penalty in loss function L Inertia = (cid:107) f
Revised: To ensure the partial convexity, the Euclidean norm is modeled as a penalty in loss function L Inertia = ∥

Original:  ( x t +1 ) − f ( x t ) (cid:107) 2 .
Revised: f ( x t +1 ) − f ( x t ) ∥ 2 .

Original:  C ROSS -M ODEL V ALIDATION FOR P HYSICAL E XACTNESS AT U NKNOWN D ATA R EGION
Revised: 3. C ROSS -M ODEL V ALIDATION FOR P HYSICAL E XACTNESS AT U NKNOWN D ATA R EGION

Original: Subsections 3.1 and 3.2 show how to avoid overﬁtting by considering the NN-based model design and ﬁtness of data to the physical laws.
Revised: Subsections 3.1 and 3.2 show how to avoid overfitting by considering the NN-based model design and fitness of data to the physical laws.

Original: However, all these designs must be within the training data range.
Revised: However, all these designs must be within the training data range.

Original: What will happen beyond the training data range, e.g., a new operation point of physical systems unseen in the past?
Revised: What will happen beyond the training data range, e.g., a new operation point of physical systems unseen in the past?

Original: Actually, determining the performance region is the most useful part for physical systems modeling after understanding the physical laws, e.g., for universal data ﬁtting at arbitrary points.
Revised: Actually, determining the performance region is the most useful part for physical systems modeling after understanding the physical laws, e.g., for universal data fitting at arbitrary points.

Original: For example, Fig.
Revised: For example, Fig.

Original: 3 shows three globally optimal or near globally optimal lines (red, blue, and green) that can ﬁt data equally well within the training dataset.
Revised: 3 shows three globally optimal or near globally optimal lines (red, blue, and green) that can fit data equally well within the training dataset.

Original: One way to avoid the selection dilemma is to conduct an active inquiry out of the historical data range for a selection.
Revised: One way to avoid the selection dilemma is to conduct an active inquiry out of the historical data range for a selection.

Original: For example, Fig. 3 illustrates that a few out-of-sample data points in grey can regularize the learning to the ground truth (dotted black line).
Revised: For example, Fig. 3 illustrates that a few out-of-sample data points in grey can regularize the learning to the ground truth (dotted black line).

Original: However, the costs of such an inquiry can be quite expensive as one inquiry may not be enough.
Revised: However, the costs of such an inquiry can be quite expensive as one inquiry may not be enough.

Original: Also, given the numerous networks that the internet of everything (IoE) has, it is infeasible and sometimes impossible to conduct an inquiry for out-of-sample estimation.
Revised: Also, given the numerous networks that the internet of everything (IoE) has, it is infeasible and sometimes impossible to conduct an inquiry for out-of-sample estimation.

Original: Therefore, how can a learning approach avoid any cost of active learning but still explore the generalization capability via some alternative active learning?
Revised: Therefore, how can a learning approach avoid any cost of active learning but still explore the generalization capability via some alternative active learning?

Original: Extrapolation (E): Constraints on Physical Validation at Unknown Regions: To conduct active learning without an inquiry cost, we implement physically meaningful forecasts via a probabilistic model for data points not existing in historical measurements.
Revised: Extrapolation (E): Constraints on Physical Validation at Unknown Regions: To conduct active learning without an inquiry cost, we implement physically meaningful forecasts via a probabilistic model for data points not existing in historical measurements.

Original: The learning objective is to provide another level of physical guidance for the constrained PNN in previous subsections.
Revised: The learning objective is to provide another level of physical guidance for the constrained PNN in previous subsections.

Original: Speciﬁcally, an ensemble of forecasts is conducted for both the mean and variance, as the forecast ability decreases when the inquiry point is far away from the data range.
Revised: Specifically, an ensemble of forecasts is conducted for both the mean and variance, as the forecast ability decreases when the inquiry point is far away from the data range.

Original: Another important design is to embed the physical form into the forecast so that the model can cross-validate the learning models.
Revised: Another important design is to embed the physical form into the forecast so that the model can cross-validate the learning models.

Original: We propose to utilize Gaussian Process (GP), which not only has computational efﬁciency in probabilistic forecasts but also provides the ﬂexibility to embed physical forms into its kernel function.
Revised: We propose to utilize Gaussian Process (GP), which not only has computational efficiency in probabilistic forecasts but also provides the flexibility to embed physical forms into its kernel function.

Original: For training the GP, the data in represented as { x i , ˜ x i } Ni =1 , where x i represents the input samples and ˜ x
Revised: For training the GP, the data is represented as { x i , ˜ x i } N i =1 , where x i represents the input samples and ˜ x

Original: i represents a sample point k -step later than x i .
Revised: i represents a sample point k -step later than x i .

Original: Based on such data, GP assumes prior function values in a Bayesian framework, e.g., p ( g | x 1 , · · · , x N ) =
Revised: Based on such data, GP assumes prior function values in a Bayesian framework, e.g., p ( g | x 1 , · · · , x N ) =

Original: N ( 0 , K ) , where g is a vector of latent function values, N represents normal distribution, and K is a kernelized covariance matrix.
Revised: N ( 0 , K ) , where g is a vector of latent function values, N represents normal distribution, and K is a kernelized covariance matrix.

Original: In particular, we can add different kernel mappings to calculate K based on our priors of the physical knowledge.
Revised: In particular, we can add different kernel mappings to calculate K based on our priors of the physical knowledge.

Original: For example, for power systems, the polynomial of order 2 is often utilized to represent the relationship between voltages and powers (Yu et al., 2017).
Revised: For example, for power systems, the polynomial of order 2 is often utilized to represent the relationship between voltages and powers (Yu et al., 2017).

Original: Using data with kernel knowledge, we obtain a predicting function via Bayes rule p ( g (cid:63) )=
Revised: Using data with kernel knowl- edge, we obtain a predicting function via Bayes rule p ( g ⋆ )

Original: [Equation]
Revised: = 1 p ( ˜ x ) (cid:82) p ( ˜ x | g ) p ( g , g ⋆ )

Original:  which has a closed-form solution g (cid:63) based on Gaussianity.
Revised: d g , which has a closed-form solution g ⋆ based on Gaussianity.

Original: The goal is to predict the k -step later points for any input sample.
Revised: The goal is to predict the k -step later points for any input sample.

Original: Speciﬁcally, one can plug-in the input data x and obtain the distribution of the k -step points p ( g (cid:63) (cid:0) x ) (cid:1) .
Revised: Specifically, one can plug-in the input data x and obtain the distribution of the k -step points p ( g ⋆ (cid:0) x ) (cid:1) .

Original: Thus, the GP-based forecasting not only brings the forecast point ˜ x k , e.g., the mean of p ( g (cid:63) (cid:0) x ) (cid:1) , but also the conﬁdence level α k based on the standard deviation.
Revised: Thus, the GP-based forecasting not only brings the forecast point ˜ x k , e.g., the mean of p ( g ⋆ (cid:0) x ) (cid:1) , but also the confidence level α k based on the standard deviation.

Original: Similarly, we can obtain ˜ y k and β k , where ˜ y k is the forecast point for the output data y k and β k is the corresponding conﬁdence level.
Revised: Similarly, we can obtain ˜ y k and β k , where ˜ y k is the forecast point for the output data y k and β k is the corresponding confidence level.

Original: Fig.
Revised: Fig.

Original: 3 illustrates the results of GP, where the forecast points have smaller uncertainty if they are closer to the dataset boundary.
Revised: 3 illustrates the results of GP, where the forecast points have smaller uncertainty if they are closer to the dataset boundary.

Original: With the forecast results from above, we can add ˜ x k ( ˜ y k ) to the training data set, like active learning.
Revised: With the forecast results from above, we can add ˜ x k ( ˜ y k ) to the training data set, like active learning.

Original: However, our method is different as active learning, which is associated with a cost.
Revised: However, our method is different from active learning, which is associated with a cost.

Original: With the cost, active learning can provide a highly accurate label.
Revised: With the cost, active learning can provide a highly accurate label.

Original: But, our new data points are always associated with conﬁdence bounds.
Revised: But, our new data points are always associated with confidence bounds.

Original: The good news is that the FOCV method can provide inﬁnite data points without a cost.
Revised: The good news is that the FOCV method can provide infinite data points without a cost.

Original: Therefore, different than active learning, FOCV uses all the data points with unequal weights, e.g., the inverse of conﬁdence level, to achieve model selection.
Revised: Therefore, different than active learning, FOCV uses all the data points with unequal weights, e.g., the inverse of confidence level, to achieve model selection.

Original: Therefore, we propose an integral-based regularization for extrapolation L Extrapol = 

Revised: Therefore, we propose an integral-based regularization for extrapolation L Extrapol =

Original: α k β k (cid:80) Kk
Revised: α k β k (cid:80) Kk

Original: =1 ( ˜ y k − f ( ˜ x k )) 2 .
Revised: =1 ( ˜ y k − f ( ˜ x k )) 2 .

Original: With extrapolation (E), the proposed methods complete the derivation of RISE.
Revised: With extrapolation (E), the proposed methods complete the derivation of RISE.

Original: It’s important to note that in RISE, the factors are decoupled out of the PNN.
Revised: It’s important to note that in RISE, the factors are decoupled out of the PNN.

Original: Furthermore, the factors can be relaxed into a convex form, which signiﬁcantly increases the probability of ﬁnding the global optima.
Revised: Furthermore, the factors can be relaxed into a convex form, which significantly increases the probability of finding the global optima.

Original: Speciﬁcally,
Revised: Specifically,

Original: Theorem 1 highlights the regularization can lead to strong convexity over a set of closed and convex regions of the parameter space, i.e., the piecewise strong convexity.
Revised: Theorem 1 highlights the regularization can lead to strong convexity over a set of closed and convex regions of the parameter space, i.e., the piecewise strong convexity.

Original: Theorem 1 (Piecewise strong convexity of PNN with regularization) .
Revised: Theorem 1 (Piecewise strong convexity of PNN with regularization) .

Original: With inertia and symmetry regularization, the loss function of the PNN is piecewise strongly convex in the parameter space.
Revised: With inertia and symmetry regularization, the loss function of the PNN is piecewise strongly convex in the parameter space.

Original: Proof.
Revised: Proof.

Original: For the derivation simplicity, we denote W as the parameter set of PNN and denote l ( W ) as the loss function, i.e., mean square error.
Revised: For the derivation simplicity, we denote W as the parameter set of PNN and denote l ( W ) as the loss function, i.e., mean square error.

Original: The range constraint (R) and the extrapolation constraint (E) are included in l ( W ) as they are included in the PNN representation.
Revised: The range constraint (R) and the extrapolation constraint (E) are included in l ( W ) as they are included in the PNN representation.

Original: Subsequently, the inertia (I) and symmetry (S) regularizations help to deﬁne l 1 ( W ) =
Revised: Subsequently, the inertia (I) and symmetry (S) regularizations help to define l 1 ( W ) =

Original: l ( W ) + λ ( L Inertia + L Mutual ) as the regularized loss function for training, where λ is a positive penalty term.
Revised: l ( W ) + λ ( L Inertia + L Mutual ) as the regularized loss function for training, where λ is a positive penalty term.

Original: To determine convexity, we write the second derivative of the regularized loss function d {+2 dt 2 | t =
+}
Revised: To determine convexity, we write the second derivative of the regularized loss function d {+2 dt 2 | t =
+}

Original: [Equation] tX ) 
Revised: l 1 ( W + tX ) =

Original:  +  λh ( W ) , where h ( W ) is the second derivative of L Inertia + L Mutual .
Revised: d 2 dt 2 | t =0 l ( W + tX )+ λh ( W ) , where h ( W ) is the second derivative of L Inertia + L Mutual .

Original: L Inertia has a quadratic form to be convex.
Revised: L Inertia has a quadratic form to be convex.

Original: For symmetry L Mutual , the l0 norm can be relaxed to l 1 normin optimization. Thus, regularization is convex and h ( W ) > 0 .
Revised: For symmetry L Mutual , the l 1 norm regularization is convex and h ( W ) > 0 .

Original: However, due to the non-convexity of neural network, the ﬁrst term d 2 dt 2 | t [-=
-] l ( W + tX ) is generally not positive semi-deﬁnite.
Revised: However, due to the non-convexity of neural network, the first term d 2 dt 2 | t [-=
-]=0 l ( W + tX ) is generally not positive semi-definite.

Original: The following lemma in (Milne, 2018) proves a lower bound for d 2 dt 2 | t =

Revised: The following lemma in (Milne, 2018) proves a lower bound for d 2 dt 2 | t =

Original: l ( W + tX ) with respect to the loss value of l .
Revised: l ( W + tX ) with respect to the loss value of l .

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose ||
Revised: Suppose ||

Original: y i ||
Revised: y i ||

Original: 2 ≤ r, r > 0 , ∀ 1 ≤ i ≤ N .
Revised: 2 ≤ r, r > 0 , ∀ 1 ≤ i ≤ N .

Original: The second derivative of l in direction X satisﬁes
Revised: The second derivative of l in direction X satisfies

Original: d 2 dt 2 | [Equation] 0 l ( W + tX ) > −√ 2 H
Revised: d 2 dt 2 | t =0 l ( W + tX ) > −√ 2 H

Original: ( H − [Equation]
Revised: ( H − 1)

Original: ||
Revised: ||

Original: W ||
Revised: W ||

Original: [Equation]{+H +
+}
Revised: {+H +
+}

Original:  ||
Revised: ⋆ ||

Original: X
Revised: X

Original: || 22 rl ( W ) 1 / 2 , where H is the number of hidden layers in a neural network, || · || (cid:63) is a deﬁned norm such that || W || (cid:63)
Revised: || 22 rl ( W ) 1 / 2 , where H is the number of hidden layers in a neural network, || · || ⋆ is a defined norm such that || W || ⋆ = max 0 ≤ i

Original: = max 0 ≤ i ≤ H
Revised: ≤ H

Original: || W i
Revised: || W i

Original: || 2 .
Revised: || 2 .

Original: According to Lemma 1, the second derivative of l is lower-bounded by a term that depends on l value.
Revised: According to Lemma 1, the second derivative of l is lower-bounded by a term that depends on l value.

Original: Then, for regions with small loss values, the positive value h ( W ) helps to restrict d 2 dt 2 | t =

Revised: Then, for regions with small loss values, the positive value h ( W ) helps to restrict d 2 dt 2 | t =

Original: l 1 ( W + tX )
Revised: l 1 ( W + tX )

Original: to a positive number, which determines the shape of the region to be convex.
Revised: to a positive number, which determines the shape of the region to be convex.

Original: Mathematically, we deﬁne the region U ( λ, θ ) , where λ > θ > 0 , by U ( λ, θ ) =
Revised: Mathematically, we define the region U ( λ, θ ) , where λ > θ > 0 , by U ( λ, θ ) =

Original: { W | l ( W ) 1 / 2 ||
Revised: { W | l ( W ) 1 / 2 ||

Original: W ||
Revised: W ||

Original: H − 1 (cid:63) < ( λ − θ ) h ( W ) √ 2 H ( H +1)
Revised: H − 1 ⋆ < ( λ − θ ) h ( W ) √ 2 H ( H +1)

Original: r
Revised: r

Original: ||
Revised: ||

Original: X
Revised: X

Original: || 22 } .
Revised: || 22 } .

Original: It’s obvious
Revised: It’s obvious

Original: that ∀ W ∈ U ( λ, θ ) , d 2 dt 2 | t =0 l 1 ( W + tX ) > 0 .
Revised: that ∀ W ∈ U ( λ, θ ) , d 2 dt 2 | t =0 l 1 ( W + tX ) > 0 .

Original: Namely, within the region of U ( λ, θ ) , the regularized loss function is strongly convex.
Revised: Namely, within the region of U ( λ, θ ) , the regularized loss function is strongly convex.

Original: In general, U ( λ, θ ) is not a convex set since many local optima points are disconnected and have small l values.
Revised: In general, U ( λ, θ ) is not a convex set since many local optima points are disconnected and have small l values.

Original: However, based on measure theory, we can utilize a set of closed convex sets to completely cover U ( λ, θ ) , leading to piecewise strong convexity of PNN over these closed sets.
Revised: However, based on measure theory, we can utilize a set of closed convex sets to completely cover U ( λ, θ ) , leading to piecewise strong convexity of PNN over these closed sets.

Original: Therefore, the piecewise strong convexity from RISE regularization guarantees that we can ﬁnd the global optimal point for each sub-regions, and the global optima satisﬁes constraints for physical parameters.
Revised: Therefore, the piecewise strong convexity from RISE regularization guarantees that we can find the global optimal point for each sub-region, and the global optima satisfies constraints for physical parameters.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Data Preparation.
Revised: Data Preparation.

Original: In experiments, we ﬁrst use synthetic datasets for illustration and introduce real physical system datasets of power systems and mass-damper systems as the underlying physical systems for model training and comparison.
Revised: In experiments, we first use synthetic datasets for illustration and introduce real physical system datasets of power systems and mass-damper systems as the underlying physical systems for model training and comparison.

Original: The datasets are summarized as follows: 1) Synthetic dataset is generated from quadratic functions with adding random noises.
Revised: The datasets are summarized as follows: 1) Synthetic dataset is generated from quadratic functions with adding random noises.

Original: 2) Power system datasets: The full training datasets of nodal voltage states and power injections are generated from power ﬂow simulations using real feeder models (IEEE 8-bus system and Utility feeder), one-year power consumption proﬁles, and distributed photovoltaics generation proﬁles (Yu et al., 2017).
Revised: 2) Power system datasets: The full training datasets of nodal voltage states and power injections are generated from power flow simulations using real feeder models (IEEE 8-bus system and Utility feeder), one-year power consumption profiles, and distributed photovoltaics generation profiles (Yu et al., 2017).

Original: Motion dynamics dataset: We simulate the dynamic process of the mass-dampers systems withnodes and obtain the measurements of velocity and momentum for the test of physical equation discovery.
Revised: Motion dynamics dataset: We simulate the dynamic process of the mass-dampers systems withnodes and obtain the measurements of velocity and momentum for the test of physical equation discovery.

Original: Aerodynamic dataset: The aircraft sensor data for weight estimation is collected from the aircraft simulation based on the aerodynamic model in (Chen et al., 2018).
Revised: Aerodynamic dataset: The aircraft sensor data for weight estimation is collected from the aircraft simulation based on the aerodynamic model in (Chen et al., 2018).

Original: Performance Evaluation Metrics.
Revised: Performance Evaluation Metrics.

Original: The following metrics based on (Blasch & Sung, 2021) are used to evaluate the performance of learning physical system governing equations.
Revised: The following metrics based on (Blasch & Sung, 2021) are used to evaluate the performance of learning physical system governing equations.

Original: Prediction error is to evaluate the modeling accuracy and extrapolation capacity of learning model via mean square error (MSE) of the output prediction.
Revised: Prediction error is to evaluate the modeling accuracy and extrapolation capacity of learning model via mean square error (MSE) of the output prediction.

Original: Physical exactness (PhyE) is to evaluate how much physics is extracted by comparing the recovered symbolic function and related parameters with true physical equation.
Revised: Physical exactness (PhyE) is to evaluate how much physics is extracted by comparing the recovered symbolic function and related parameters with true physical equation.

Original: Robustness to noise ratio (RNR) is proposed to average all results tested under different signal-to-noise ratios for practical consideration.
Revised: Robustness to noise ratio (RNR) is proposed to average all results tested under different signal-to-noise ratios for practical consideration.

Original: More details are provided in Appendix.
Revised: More details are provided in Appendix.

Original: Benchmark Methods and Testing Scenarios.
Revised: Benchmark Methods and Testing Scenarios.

Original: The following benchmark methods are used for comparison: (1) sparse identiﬁcation of nonlinear dynamics (SINDy)
Revised: The following benchmark methods are used for comparison: (1) sparse identification of nonlinear dynamics (SINDy)

Original: (Brunton et al., 2016), (2) support vector regression (SVR) (Yu et al., 2017), (3) deep residual network (Resnet)
Revised: (Brunton et al., 2016), (2) support vector regression (SVR) (Yu et al., 2017), (3) deep residual network (Resnet)

Original: (He et al., 2016), and (4) equation learner (EQL ÷ ) (Sahoo et al., 2018).
Revised: (He et al., 2016), and (4) equation learner (EQL ÷ ) (Sahoo et al., 2018).

Original: Different scenarios are generated for testing: S 1 denotes the regular setup of dataset size and noise level (independent Gaussian noise with zero mean and 0.01 standard deviation); S 2 denotes a noisy setup with noises of 0.1 standard deviation; S 3 denotes a data-limited setup with 10% samples of original dataset available.
Revised: Different scenarios are generated for testing: S 1 denotes the regular setup of dataset size and noise level (independent Gaussian noise with zero mean and 0.01 standard deviation); S 2 denotes a noisy setup with noises of 0.1 standard deviation; S 3 denotes a data-limited setup with 10% samples of original dataset available.

Original: The results are summarized in Fig.
Revised: The results are summarized in Fig.

Original: Although other methods can have low errors in some testing cases, they can not maintain the good performance as PNN-RISE does for all the datasets.
Revised: Although other methods can have low errors in some testing cases, they can not maintain the good performance as PNN-RISE does for all the datasets.

Original: Especially, EQL ÷ and SINDy’s performance deteriorates when the problem size becomes larger, e.g., power system datasets with data from many nodes.
Revised: Especially, EQL ÷ and SINDy’s performance deteriorates when the problem size becomes larger, e.g., power system datasets with data from many nodes.

Original: While SVR is robust against noises, it can not recover the nonlinear physical function exactly to further decrease the learning error.
Revised: While SVR is robust against noises, it can not recover the nonlinear physical function exactly to further decrease the learning error.

Original: In this case, the proposed method shows low learning error and a stable performance in both noisy and data-limited scenarios.
Revised: In this case, the proposed method shows low learning error and a stable performance in both noisy and data-limited scenarios.

Original: Moreover, the performance of function recovery is compared in Fig.
Revised: Moreover, the performance of function recovery is compared in Fig.

Original: 5 (left).
Revised: 5 (left).

Original: The bar plot shows the recovery rates (percentage of runs that correctly recover the ground truth symbolic expression) and the line plot with shades shows the recovery accuracy of the corresponding physical parameters.
Revised: The bar plot shows the recovery rates (percentage of runs that correctly recover the ground truth symbolic expression) and the line plot with shades shows the recovery accuracy of the corresponding physical parameters.

Original: While EQL ÷ easily fails in large power system cases, PNN-RISE has high success rate in true function recovery, merely trapped into local optima.
Revised: While EQL ÷ easily fails in large power system cases, PNN-RISE has high success rate in true function recovery, merely trapped into local optima.

Original: For physical parameter recovery, PNN-RISE not only has higher average accuracy but also has lower standard deviation than EQL ÷ .
Revised: For physical parameter recovery, PNN-RISE not only has higher average accuracy but also has lower standard deviation than EQL ÷ .

Original: 4. A BLATION STUDY FOR PHYSICAL CONSTRAINTS
Revised: 4. A BLATION STUDY FOR PHYSICAL CONSTRAINTS

Original: In order to show the performance of using different physical constraints, we provide an ablation study of the proposed NN.
Revised: In order to show the performance of using different physical constraints, we provide an ablation study of the proposed NN.

Original: We ﬁrst use the synthetic example to visualize the effect of constraints and show in Appendix.
Revised: We first use the synthetic example to visualize the effect of constraints and show in Appendix.

Original: Fig.
Revised: Fig.

Original: 5 (right) presents the learning results on power system datasets (IEEE 8-bus system).
Revised: 5 (right) presents the learning results on power system datasets (IEEE 8-bus system).

Original: The performance is evaluated in different test scenarios: 1) regular data acquisition, 2) limited input range (voltages within [0 . 98 , 1 . 01] for training and [0 . 90 , 1 . 10] for testing), 3) out- of-scale invariants in a governing function, and 4) limited data amount.
Revised: The performance is evaluated in different test scenarios: 1) regular data acquisition, 2) limited input range (voltages within [0 . 98 , 1 . 01] for training and [0 . 90 , 1 . 10] for testing), 3) out- of-scale invariants in a governing function, and 4) limited data amount.

Original: Generally, adding physical constraints boosts the performance in different testing scenarios.
Revised: Generally, adding physical constraints boosts the performance in different testing scenarios.

Original: The constraint on mutual variables signiﬁcantly reduces the learning error by pushing the features and parameters to be symmetric for multiple-output problems.
Revised: The constraint on mutual variables significantly reduces the learning error by pushing the features and parameters to be symmetric for multiple-output problems.

Original: When the data amount is very limited to train NN efﬁciently, the bound on the inertia of system outputs in time sequence improves the function recovery.
Revised: When the data amount is very limited to train NN efficiently, the bound on the inertia of system outputs in time sequence improves the function recovery.

Original: 4. I LLUSTRATION OF THE EFFECTIVENESS OF FORECASTING - BASED PHYSICS LEARNING
Revised: 4. I LLUSTRATION OF THE EFFECTIVENESS OF FORECASTING - BASED PHYSICS LEARNING

Original: [Figure]
Revised: [Figure]

Original: To illustrate the effectiveness of GP forecasting, we utilize 200 samples for training and propose to forecast { 500 , 1000 , 1500 , 2000 } samples out of the training dataset.
Revised: To illustrate the effectiveness of GP forecasting, we utilize 200 samples for training and propose to forecast { 500 , 1000 , 1500 , 2000 } samples out of the training dataset.

Original: Also, we consider different
Revised: Also, we consider different

Original: SNRs in { 50 , 100 , 150 } .
Revised: SNRs in { 50 , 100 , 150 } .

Original: The forecast samples are integrated into PNN model and other benchmark models for comparisons.
Revised: The forecast samples are integrated into PNN model and other benchmark models for comparisons.

Original: Fig.
Revised: Fig.

Original: 6 demonstrates the results with respect to the forecast points.
Revised: 6 demonstrates the results with respect to the forecast points.

Original: Due to the space limit, we only show results of GP-based PNN and GP-based Resnet for comparisons.
Revised: Due to the space limit, we only show results of GP-based PNN and GP-based Resnet for comparisons.

Original: The patterns of results for other benchmark methods are similar to GP-based Resent.
Revised: The patterns of results for other benchmark methods are similar to GP-based Resent.

Original: We observe that
Revised: We observe that

Original: GP-based PNN performs much better than GP-based Resnet, which is consistent to observations in the previous subsections.
Revised: GP-based PNN performs much better than GP-based Resnet, which is consistent to observations in the previous subsections.

Original: As the number of forecast data points increases, the prediction MSE value generally decreases for both PNN model and Resnet model.
Revised: As the number of forecast data points increases, the prediction MSE value generally decreases for both PNN model and Resnet model.

Original: It implies that our GP forecasting produces accurate forecast points, suitable for all prediction models.
Revised: It implies that our GP forecasting produces accurate forecast points, suitable for all prediction models.

Original: When the number is larger than 500 , the results don’t show a signiﬁcant improvement.
Revised: When the number is larger than 500 , the results don’t show a significant improvement.

Original: The reason is that forecast points far from the training set boundary are with low conﬁdence, which don’t have a big impact on learning results.
Revised: The reason is that forecast points far from the training set boundary are with low confidence, which don’t have a big impact on learning results.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The lack of physical understanding makes it difﬁcult for planning, monitoring, and control of IoE.
Revised: The lack of physical understanding makes it difficult for planning, monitoring, and control of IoE.

Original: To learn governing physical equations from data, the paper ﬁrst presents a fundamental challenge for data-driven methods like symbolic regression and sparse regression.
Revised: To learn governing physical equations from data, the paper first presents a fundamental challenge for data-driven methods like symbolic regression and sparse regression.

Original: Data-driven only methods cannot guarantee to ﬁnd the exact physics because many low-complexity expressions can perfectly ﬁt the data.
Revised: Data-driven only methods cannot guarantee to find the exact physics because many low-complexity expressions can perfectly fit the data.

Original: To resolve the issue, we propose a complete set of constraints based on physical principles to restrict search space.
Revised: To resolve the issue, we propose a complete set of constraints based on physical principles to restrict search space.

Original: Thus, an efﬁcient physical neural network (PNN-RISE) is developed.
Revised: Thus, an efficient physical neural network (PNN-RISE) is developed.

Original: Finally, the FOCV supports the scenario when (partial) physical constraints are unknown.
Revised: Finally, the FOCV supports the scenario when (partial) physical constraints are unknown.

Original: The forecast of data beyond the training dataset enhances model robustness with few extra costs.
Revised: The forecast of data beyond the training dataset enhances model robustness with few extra costs.

Original: Results on physical system datasets show the superior performance of proposed methods.
Revised: Results on physical system datasets show the superior performance of proposed methods.

Original: B P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: First, we deﬁne the biconvexity as follows (Gorski et al., 2007).
Revised: First, we define the biconvexity as follows (Gorski et al., 2007).

Original: Deﬁnition 1 (Biconvexity) .
Revised: Definition 1 (Biconvexity) .

Original: A function φ : A × B → R is called biconvex if f ( a, b ) is convex in a for ﬁxed b ∈ B , and f ( a, b ) is convex in b for ﬁxed a ∈ A .
Revised: A function ϕ : A × B → R is called biconvex if f ( a, b ) is convex in a for fixed b ∈ B , and f ( a, b ) is convex in b for fixed a ∈ A .

Original: Then, we prove PNN is biconvex.
Revised: Then, we prove PNN is biconvex.

Original: Recall for the PNN function y = f
Revised: Recall for the PNN function y = f

Original: ( x ) , the k th element can be written as: y k
Revised: ( x ) , the k th element can be written as: y k

Original: = (cid:80) j m jk z j = (cid:80) j m jk exp (cid:0) (cid:80) Ni =1 p ij ln
Revised: = (cid:80) j m jk z j = (cid:80) j m jk exp (cid:0) (cid:80) Ni =1 p ij ln

Original: ( x i ) (cid:1) .
Revised: ( x i ) (cid:1) .

Original: ∀ j, k , if m jk is known, we need to prove the convexity of f with respect to p ij , ∀ 1 ≤ i ≤ N .
Revised: ∀ j, k , if m jk is known, we need to prove the convexity of f with respect to p ij , ∀ 1 ≤ i ≤ N .

Original: Since the linear combination (cid:80) Ni =1 p ij ln
Revised: Since the linear combination (cid:80) Ni =1 p ij ln

Original: ( x i ) is convex with respect to p ij , and the exponential function is convex, the convex function composition implies
Revised: ( x i ) is convex with respect to p ij , and the exponential function is convex, the convex function composition implies

Original: that exp (cid:0) (cid:80) Ni =1 p ij ln
Revised: that exp (cid:0) (cid:80) Ni =1 p ij ln

Original: ( x i ) (cid:1) is convex with respect to p ij .
Revised: ( x i ) (cid:1) is convex with respect to p ij .

Original: Therefore, ∀ j, k , if m jk is known, the function y k
Revised: Therefore, ∀ j, k , if m jk is known, the function y k

Original: = (cid:80)
Revised: = (cid:80)

Original: j m jk exp (cid:0) (cid:80) Ni =1 p ij ln
Revised: j m jk exp (cid:0) (cid:80) Ni =1 p ij ln

Original: ( x i ) (cid:1) is convex with respect to p ij , ∀ 1 ≤ i ≤ N .
Revised: ( x i ) (cid:1) is convex with respect to p ij , ∀ 1 ≤ i ≤ N .

Original: On the other hand, ∀ i, j , if p ij is known, y k
Revised: On the other hand, ∀ i, j , if p ij is known, y k

Original: = (cid:80)
Revised: = (cid:80)

Original: j m jk exp (cid:0) (cid:80) Ni =1 p ij ln
Revised: j m jk exp (cid:0) (cid:80) Ni =1 p ij ln

Original: ( x i ) (cid:1) is a linear combination of m jk .
Revised: ( x i ) (cid:1) is a linear combination of m jk .

Original: Therefore, y k is convex with respect to m jk given p ij .
Revised: Therefore, y k is convex with respect to m jk given p ij .

Original: Therefore, PNN function y = f
Revised: Therefore, PNN function y = f

Original: ( x ) is biconvex with respect to the logarithm layer and the linear summation layer.
Revised: ( x ) is biconvex with respect to the logarithm layer and the linear summation layer.

Original: The biconvexity can lead to an efﬁcient alternate convex search (ACS)
Revised: The biconvexity can lead to an efficient alternate convex search (ACS)

Original: (Gorski et al., 2007) to seek for optimal solutions.
Revised: (Gorski et al., 2007) to seek for optimal solutions.

Original: Namely, ACS will ﬁx one part of parameters to search for the other part iteratively.
Revised: Namely, ACS will fix one part of parameters to search for the other part iteratively.

Original: Due to biconvexity, in each iteration, ACS will solve a convex problem.
Revised: Due to biconvexity, in each iteration, ACS will solve a convex problem.

Original: Then, Corollary 4.10 in (Gorski et al., 2007) guarantees that the searched points will converge to stationary points of f ( x ) .
Revised: Then, Corollary 4.10 in (Gorski et al., 2007) guarantees that the searched points will converge to stationary points of f ( x ) .

Original: C D ETAILS OF D ATASETS AND E XPERIMENT S ETUPS
Revised: C D ETAILS OF D ATASETS AND E XPERIMENT S ETUPS

Original: Synthetic dataset: generating random samples of x from Gaussian distribution N (0 , 1) , and obtain the outputs by y =
Revised: Synthetic dataset: generating random samples of x from Gaussian distribution N (0 , 1) , and obtain the outputs by y =

Original: c 1 x 1 x 2 + c 2 x 21 .
Revised: c 1 x 1 x 2 + c 2 x 21 .

Original: Power system datasets: IEEE provides standard power system models, including the grid topology, parameters, and generation models, etc., for simulations.
Revised: Power system datasets: IEEE provides standard power system models, including the grid topology, parameters, and generation models, etc., for simulations.

Original: The model ﬁles and the simulation platform, MATPOWER (MATPOWER, 2020), are based on MATLAB.
Revised: The model files and the simulation platform, MATPOWER (MATPOWER, 2020), are based on MATLAB.

Original: For simulations, the load ﬁles are need as the inputs to the systems.
Revised: For simulations, the load files are needed as the inputs to the systems.

Original: Thus, we introduce power consumption data fromPJM
Revised: Thus, we introduce power consumption data from

Original:  Interconnection LLC (PJM Interconnection LLC, 2018).
Revised: PJM Interconnection LLC (PJM Interconnection LLC, 2018).

Original: Such load ﬁles contain hourly power consumption in 2017 for the PJM RTO regions.
Revised: Such load files contain hourly power consumption in 2017 for the PJM RTO regions.

Original: With the above data, MATPOWER produces the system states of voltages and nodal power injections.
Revised: With the above data, MATPOWER produces the system states of voltages and nodal power injections.

Original: Then, we utilize the voltage as input and the nodal power as the output to learn the physical mapping, for which the true physical function is used for evaluations (Yu et al., 2017).
Revised: Then, we utilize the voltage as input and the nodal power as the output to learn the physical mapping, for which the true physical function is used for evaluations (Yu et al., 2017).

Original: To diversify the system types, we utilize the mass-damper system and aerodynamic system to further validate our results.
Revised: To diversify the system types, we utilize the mass-damper system and aerodynamic system to further validate our results.

Original: Motion dynamics dataset: mass dampers of in-plane wind turbine.
Revised: Motion dynamics dataset: mass dampers of in-plane wind turbine.

Original: The physical equation of the mass-damper system is representative of aircraft motion for analysis of system dynamics (van der
Revised: The physical equation of the mass-damper system is representative of aircraft motion for analysis of system dynamics (van der

Original: Schaft, 2017; McRuer et al., 2014).
Revised: Schaft, 2017; McRuer et al., 2014).

Original: Moreover, the active tuned mass dampers are implemented to control in-plane blade vibrations (Fitzgerald et al., 2013).
Revised: Moreover, the active tuned mass dampers are implemented to control in-plane blade vibrations (Fitzgerald et al., 2013).

Original: Using MATLAB, we simulate the dynamic process of the mass-damper system with 10 nodes and obtain the measurements of velocity and momentum for the test of physical equation discovery.
Revised: Using MATLAB, we simulate the dynamic process of the mass- damper system with 10 nodes and obtain the measurements of velocity and momentum for the test of physical equation discovery.

Original: Aerodynamic dataset: aircraft weight estimation: The aircraft sensor data is collected from the aircraft simulation in MATLAB and the corresponding aerodynamic model is illustrated in (Chen et al., 2018).
Revised: Aerodynamic dataset: aircraft weight estimation: The aircraft sensor data is collected from the aircraft simulation in MATLAB and the corresponding aerodynamic model is illustrated in (Chen et al., 2018).

Original: In this case, the goal is to recover the aerodynamic model of gross weight and angle of attack in a cambered airfoil.
Revised: In this case, the goal is to recover the aerodynamic model of gross weight and angle of attack in a cambered airfoil.

Revised: For the implementation of machine learning methods, the prepared data samples are split as 60% for training the model, 15% for validation of model hyperparameters, and 25% for testing.

Revised: Specifically, hyperparameters of the benchmark methods are selected through k -fold cross-validation.

Revised: For SVR, the kernel is chosen from the candidates that include polynomial kernel ( 2 nd -degree and 3 nd -degree), radial basis function (RBF) kernel, and hyperbolic tangent kernel.

Revised: The box parameter is similarly chosen from { 1 × 10 − 3 , 5 × 10 − 3 , 1 × 10 − 2 , 5 × 10 − 2 , 1 × 10 − 1 , 5 × 10 − 1 , 1 × 10 0 , 5 × 10 0 , 1 × 10 1 } .

Original: D R ESULTS OF THE S YNTHETIC D ATASETS
Revised: D R ESULTS OF THE S YNTHETIC D ATASETS

Original: In order to show the performance of using different physical constraints, we provide an ablation study of the proposed NN.
Revised: In order to show the performance of using different physical constraints, we provide an ablation study of the proposed NN.

Original: We use the synthetic example to visualize the effect of constraints.
Revised: We use the synthetic example to visualize the effect of constraints.

Original: The left-hand side of Fig.
Revised: The left-hand side of Fig.

Original: 9 compares the local optimum (transparent gray surface) learned by NN with the true function (solid gray surface).
Revised: 10 compares the local optimum (transparent gray surface) learned by NN with the true function (solid gray surface).

Original: Though there are some overlaps, the large gap between the two surfaces causes large errors for generalization.
Revised: Though there are some overlaps, the large gap between the two surfaces causes large errors for generalization.

Original: By adding constraints, we observe in the middle of Fig.
Revised: By adding constraints, we observe in the middle of Fig.

Original: 9 that the variable coupling constraint rotates and stretches the surface to the correct shape like the maximum and minimum values of the surface.
Revised: 10 that the variable coupling constraint rotates and stretches the surface to the correct shape like the maximum and minimum values of the surface.

Original: On the other hand, the parameter range constraint smoothes the surface of the local optimum.
Revised: On the other hand, the parameter range constraint smoothes the surface of the local optimum.

Original: Together, the constraints push the recovered function (cyan surface) towards the ground truth, where the overlap in Fig.
Revised: Together, the constraints push the recovered function (cyan surface) towards the ground truth, where the overlap in Fig.

Original: 9 (right) shows the physical exactness in not only feature components but also the corresponding parameters.
Revised: 10 (right) shows the physical exactness in not only feature components but also the corresponding parameters.