Original: W HAT TO D O IF S PARSE R EPRESENTATION L EARNING F AILS U NEXPECTEDLY ?
Revised: W HAT TO D O IF S PARSE R EPRESENTATION L EARNING F AILS U NEXPECTEDLY ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning physical equations from data is essential for scientiﬁc discovery and engineering modeling.
Revised: Learning physical equations from data is essential for scientiﬁc discovery and engineering modeling.

Original: However, most of the existing methods rely on two rules: (1) learn a sparse representation to ﬁt data and (2) check if the loss objective function satisﬁes error thresholds.
Revised: However, most of the existing methods rely on two rules: (1) learn a sparse representation to ﬁt data and (2) check if the loss objective function satisﬁes error thresholds.

Original: This paper illustrates that such conditions are far from sufﬁcient.
Revised: This paper illustrates that such conditions are far from sufﬁcient.

Original: Speciﬁcally, we show that sparse non-physical approximations exist with excellent ﬁtting accuracy, but fail to adequately model the situation.
Revised: Speciﬁcally, we show that sparse non-physical approximations exist with excellent ﬁtting accuracy, but fail to adequately model the situation.

Original: To fundamentally resolve the data-ﬁtting problem, we propose a physical neural network (PNN) utilizing “Range, Inertia, Symmetry, and Extrapolation” (RISE) constraints.
Revised: To fundamentally resolve the data-ﬁtting problem, we propose a physical neural network (PNN) utilizing “Range, Inertia, Symmetry, and Extrapolation” (RISE) constraints.

Original: RISE is based on a complete analysis for the generalizability of data properties for physical systems.
Revised: RISE is based on a complete analysis for the generalizability of data properties for physical systems.

Original: The ﬁrst three techniques focus on the deﬁnition of physics in space and time.
Revised: The ﬁrst three techniques focus on the deﬁnition of physics in space and time.

Original: The last technique of extrapolation is novel based on active learning without an inquiry, using cross-model validation.
Revised: The last technique of extrapolation is novel based on active learning without an inquiry, using cross-model validation.

Original: We validate the proposed PNN-RISE method via a synthetic dataset, power system dataset, and mass-damper system dataset.
Revised: We validate the proposed PNN-RISE method via a synthetic dataset, power system dataset, and mass-damper system dataset.

Original: Numerical results show the universal capability of the PNN-RISE approach to quickly identify the hidden physical models without local optima, opening the door for the fast and highly accurate discovery of the physical laws or systems with external loads.
Revised: Numerical results show the universal capability of the PNN-RISE approach to quickly identify the hidden physical models without local optima, opening the door for the fast and highly accurate discovery of the physical laws or systems with external loads.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Internet of Everything (IoE) connects new  devices into an intelligent web at an unprecedented speed.
Revised: Internet of Everything (IoE) connects new edge devices into an intelligent web at an unprecedented speed.

Original: If utilized systematically, IoE can create much more efﬁcient productivity via coordinated effort among humans, processes, data, and things (Li et al., 2020).
Revised: If utilized systematically, IoE can create much more efﬁcient productivity via coordinated effort among humans, processes, data, and things (Li et al., 2020).

Original: Among various analytical frameworks, dynamic data-driven applications systems (DDDAS) naturally interweaves diversiﬁed information viadynamic and data-driven analysis to assimilate the system model for a wide range of applications (Blasch et al., 2018), thus signiﬁcantly increasing the performance of  the simulationmodel andthe application model.
Revised: Among various analytical frameworks, dynamic data-driven applications systems (DDDAS) naturally interweaves diversiﬁed information via data-driven analysis to assimilate the system model for a wide range of applications (Blasch et al., 2018), thus signiﬁcantly increasing the performance of both the simulation and application models.

Original: The simulation modelin DDDAS usually incorporates physical methods (Nasiakou et al., 2018; Chowdhury & Subramani, 2020) which provides a structure for model-based and data-driven coordination.
Revised: The simulation model usually incorporates physical methods (Nasiakou et al., 2018; Chowdhury & Subramani, 2020) which provides a structure for model-based and data-driven coordination.

Original: However, many of the physical laws in the web of IoE can be unknown. The laws of the web of IoE remain to be discovered manually or automatically to beneﬁt society.
Revised: However, many of the physical laws in the web of IoE can be unknown, which remain to be discovered manually or automatically to enable down-streaming applications.

Original: The most economically efﬁcient way to discover the physical law  is to use machine learning to analyze the pattern behind the data stream, which is an important topicof artiﬁcial intelligence (Sahoo et al., 2018; Udrescu & Tegmark, 2020).
Revised: The most economically efﬁcient way of physical law discovery is to use machine learning to analyze the pattern behind the data stream, which is an important topic (Sahoo et al., 2018; Udrescu & Tegmark, 2020).

Original: In previous studies, such as symbolic regression (Petersen, 2019), the sparsity representation is used to measure the accountability of physics (Brunton et al., 2016).
Revised: In previous studies, such as symbolic regression (Petersen, 2019), the sparsity representation is used to measure the accountability of physics (Brunton et al., 2016).

Original: The underlying assumption is that sparsity can ensure physical exactness due to parsimonious principle (Li, 2013).
Revised: The underlying assumption is that sparsity can ensure physical exactness due to parsimonious principle (Li, 2013).

Original: While low complexity of learning models is a necessary condition forfor enhancing the understanding of physical laws (Sch¨olkopf et al., 2002; Blasch et al., 2021), there is not a universal sufﬁcient condition to solve model recovery problem with a guarantee.
Revised: While low complexity of learning models is a necessary condition for enhancing the understanding of physical laws (Sch¨olkopf et al., 2002; Blasch et al., 2021), there is not a universal sufﬁcient condition to solve model recovery problem with a guarantee.

Original: Therefore, one approach is to add additional conditions to ensure a physical recovery, e.g., near-zero ﬁtting errors (Brunton et al., 2016; Champion et al.,2019; Petersen, 2019) or/and cross validation (Sahoo et al., 2018; Kim et al., 2020).
Revised: Therefore, one approach is to add additional conditions to ensure a physical recovery, e.g., near-zero ﬁtting errors (Champion et al., 2019) or/and cross validation (Kim et al., 2020).

Original: However, it can be observed that different machine learners easily ﬁt the available data with a sparse but wrong model. Such a case can happen even with cross-validation.
Revised: However, it can be observed that different models easily ﬁt the available data with a sparse but wrong model, which can happen even with cross-validation.

Original: The overﬁtting should challenge the directions of many researchers in physical system discovery, who believe that only the true physical laws can achieve sparse recovery with a perfect ﬁt.
Revised: The overﬁtting should challenge the directions of many researchers in physical system discovery, who believe that only the true physical laws can achieve sparse recovery with a perfect ﬁt.

Original: To resolve the problem, we ﬁrst eliminate non-physical local optima of the neural network (NN) by discarding all the non-physical parameters, i.e., model parameters indirectly related to the physical representation.
Revised: To resolve the problem, we ﬁrst eliminate non-physical local optima of the neural network (NN) by discarding all the non-physical parameters, i.e., model parameters indirectly related to the physical representation.

Original: The process is novel as existing NN-based methods require certain non-physical parameters to control the representation sparsity (Liu & Laili, 2018; Petersen, 2019).
Revised: The process is novel as existing NN-based methods require certain non-physical parameters to control the representation sparsity (Liu & Laili, 2018; Petersen, 2019).

Original: We show how to design a NN with pure physical parameters, entitled a physical neural network (PNN).
Revised: We show how to design a NN with pure physical parameters, entitled a physical neural network (PNN).

Original: The key of PNN is to enforce activation to physical weights for sparse selection, completely removing the non-physical parameters and local optima.
Revised: The key of PNN is to enforce activation to physical weights for sparse selection, completely removing the non-physical parameters and local optima.

Original: In training the PNN, there can still be local optimum solutions, bus actual knowledge can enhance model ﬁdelity.
Revised: In training the PNN, there can still be local optimum solutions, bus actual knowledge can enhance model ﬁdelity.

Original: For obtaining the ground truth, physical systems’ examination can be used to extract common and universal properties.
Revised: For obtaining the ground truth, physical systems’ examination can be used to extract common and universal properties.

Original: Subsequently, the PNN is restricted to systematically obey the physical and system properties.
Revised: Subsequently, the PNN is restricted to systematically obey the physical and system properties.

Original: Speciﬁcally, constraints based on the physical distance and physical symmetry are embedded in the training function to quantify the spatial relationship enforced by physical law.
Revised: Speciﬁcally, constraints based on the physical distance and physical symmetry are embedded in the training function to quantify the spatial relationship enforced by physical law.

Original: Furthermore, as physical systems evolve over time, inertial property for physical systems are extracted and updated.
Revised: Furthermore, as physical systems evolve over time, inertial property for physical systems are extracted and updated.

Original: In general, the regularization from physical laws focuses the learning/optimization towards a more meaningful ﬁtness understanding of the given data.
Revised: In general, the regularization from physical laws focuses the learning/optimization towards a more meaningful ﬁtness understanding of the given data.

Original: Although the two designs of PNN and physics constraints aim to obtain the physical laws via ﬁtting existing data, the incorporation of physical laws’ biggest impact is its capability to achieve zero ﬁtting errors even beyond the training data range.
Revised: Although the two designs of PNN and physics constraints aim to obtain the physical laws via ﬁtting existing data, the incorporation of physical laws’ biggest impact is its capability to achieve zero ﬁtting errors even beyond the training data range.

Original: Unfortunately, it will be costly if not impossible to conduct active learning for every Internet of Things (IoT) system due to its scale.
Revised: Unfortunately, it will be costly if not impossible to conduct active learning for every Internet of Things (IoT) system due to its scale.

Original: To provide the “free lunch” of adding the extrapolation capability of PNN, we create an new learning method for cross-model validation based on probabilistic estimates with physical kernels.
Revised: To provide the “free lunch” of adding the extrapolation capability of PNN, we propose an new learning method for cross-model validation based on probabilistic estimates with physical kernels.

Original: Speciﬁcally, we propose a physically meaningful forecast to identify out-of-sample points with a conﬁdence level.
Revised: We use a physically meaningful forecast to identify out-of-sample points with a conﬁdence level.

Original: Then, the results are integrated as a weak supervision for learning the exact physics for extrapolation, in addition to thephysical constrained PNN.
Revised: Then, the results are integrated as a weak supervision for learning the exact physics for extrapolation, in addition to the constrained PNN.

Original: Together, PNN-RISE uses the PNN container to include the RISE principles, enforcing the recovered function to satisfy constraints on range (R), inertia (I), symmetry (S), and extrapolation (E).
Revised: Together, PNN-RISE uses the PNN container to include the RISE principles, enforcing the recovered function to satisfy constraints on range (R), inertia (I), symmetry (S), and extrapolation (E).

Original: And, it is worth highlighting that the extrapolation technique is a brand-new technique for active learning without an inquiry.

Original: Empirically, extensive validations show that the PNN-RISE outperforms the state-of-the-art methods.
Revised: Empirically, extensive validations show that the PNN-RISE outperforms the state-of-the-art methods.

Original: Consistently, scenarios with or without each constraint in RISE validate the approach.
Revised: Consistently, scenarios with or without each constraint in RISE validate the approach.

Original: Furthermore, for each constraint, we quantify its strength with different levels, highlighting that loose constraints are enough to obtain high accuracy in the proposed PNN.
Revised: Furthermore, for each constraint, we quantify its strength with different levels, highlighting that loose constraints are enough to obtain high accuracy in the proposed PNN.

Original: In summary, the contributions in Fig.
Revised: In summary, the contributions in Fig.

Original: 1 are (1) showing that the sparse recovery with low training and testing errors does not ensure physical law recovery, (2) reducing the possibilities of local optima signiﬁcantly by eliminating non-physical parameters in learning model selection and design, (3) proposing four physical principles to completely characterize physical systems via constrained learning, and (4) creating a novel learning method for physical systems extrapolation as probabilistic forecasting out-of-sample cross-model validation (FOCV).
Revised: 1 are 1) showing that the sparse recovery with low training and testing errors does not ensure physical law recovery, 2) reducing the possibilities of local optima signiﬁcantly by eliminating non-physical parameters in learning model selection and design, 3) proposing four physical principles to completely characterize physical systems via constrained learning, and 4) creating a novel learning method for physical systems extrapolation as probabilistic forecasting out-of-sample cross-model validation (FOCV).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Physical System Identiﬁcation.
Revised: Physical System Identiﬁcation.

Original: To recover system information, supervised learning  canbe used to approximate input-output relationships for inferences, e.g., regression and classiﬁcation (Huang et al., 2006; Hornik et al., 1990).
Revised: To recover system information, supervised learning (SL) can approximate input-output relationships for inferences (Huang et al., 2006; Hornik et al., 1990).

Original: These regression and classiﬁcation tools show good numerical performance in non-physical and physical applications, including computer vision for images and videos (Lu & Weng, 2007;
Revised: They show good numerical performance in non-physical and physical applications, including computer vision tasks and natural language processing (Lu & Weng, 2007;

Original: Wang et al., 2017a; Karpathy et al., 2014), natural language processing for texts (Nadkarni et al., 2011; Hirschberg & Manning, 2015), and speech processing for audio (Yu & Deng, 2016; Abdel-Hamid et al., 2014).
Revised: Wang et al., 2017a; Karpathy et al., 2014; Nadkarni et al., 2011; Hirschberg & Manning, 2015).

Original: Additionally, deep neural networks (DNNs) boostthe image classiﬁcation accuracy (Hinton et al., 2006; Krizhevsky et al., 2012) by using latent layers to extract nonlinear features hidden in image data.
Revised: Deep neural networks (DNNs) boost classiﬁcation accuracy (Hinton et al., 2006; Krizhevsky et al., 2012) by using latent layers to extract nonlinear features hidden in images.

Original: While boosting makes machine learning useful for non-physical systems, physical system applications require the accuracy to be much higher  for operation or control, e.g., near 100% for SixSigma performance Walshe et al.
Revised: While it makes SL useful for non-physical systems, physical systems require a much higher accuracy for operation or control, e.g., near 100% for Six

Original: (2010), so thatsystem operators can safely operate the system to avoid outages or cascading failures (Chung et al., 2018; Liu et al., 2017).
Revised: Sigma performance (Walshe et al., 2010), so that operators can safely operate the system to avoid outages or cascading failures (Chung et al., 2018; Liu et al., 2017).

Original: Even worse, many supervised learning approaches are black boxes that lack physical interpretability for reliable operations,accountability for control, as well as certiﬁability for performance standards (Manitsas et al., 2012; Zamzam & Sidiropoulos, 2020;
Revised: Even worse, many SL models are black boxes that lack physical interpretability for reliable operations, as well as certiﬁability for performance standards (Mestav et al., 2018;

Original: Shareef et al., 2013; Mestav et al., 2018; Du & Li, 2019; Hu et al., 2020;Karami & Mohammadi, 2008; M¨uller et al., 2010; Baghaee et al., 2017).
Revised: Du & Li, 2019; Hu et al., 2020; Baghaee et al., 2017).

Original: Therefore, some works ﬁnd utilizing contextual information improves physical-system identiﬁcation where they assume a good knowledge of physical basis and sufﬁcient measurements across the system (Yu et al., 2017; Chen et al., 2020).
Revised: Therefore, some works ﬁnd utilizing contextual information improves physical-system identiﬁcation where they assume a good knowledge of physical basis and sufﬁcient measurements across the system (Yu et al., 2017; Chen et al., 2020).

Original: Similarly, (Brunton et al., 2016; Champion et al., 2019) assume prior symbols are known in a dictionary and introduce sparse regression to select symbols.
Revised: Similarly, (Brunton et al., 2016; Champion et al., 2019; Li & Weng, 2021) assume prior symbols are known in a dictionary and introduce sparse regression to select symbols.

Original: However, these assumptions are too limiting in that for most complex systems, it would be difﬁcult to know all the physical basis responses ahead of time. For example, the dictionary assumption can hardly be generalized to complexstructured systems like multi-energy systems (Wang et al., 2017b) and biology systems (Tarca et al., 2007), etc. that are subject to external disturbances resulting from changing environmental condition.
Revised: However, the assumptions are too limiting due to the difﬁculty of knowing all the physical basis for arbitrary complex systems (Wang et al., 2017b; Tarca et al., 2007).

Original: One ad-hoc approach (Schmidt & Lipson, 2009; Petersen, 2019) uses a genetic algorithm  search for the symbolic basiswithout prior knowledge, but the problem is NP-hard without any guarantee on recovering the governing physical laws from empirical data.
Revised: One ad-hoc approach (Schmidt & Lipson, 2009; Petersen, 2019) uses a genetic algorithm to search for the symbolic basis but the problem is NP-hard without any guarantee on recovering the governing physical laws from empirical data.

Original: Therefore, it is important to investigate whether the sparsity technique and a small modeling error are sufﬁcient to claim the success of learning the physical model.
Revised: Thus, we investigate whether the sparsity technique and a small modeling error are sufﬁcient to claim the success of learning the physical model.

Original: Additionally, how can analytical techniques ensure exact physics learning forall systems, or domain-speciﬁc systems?
Revised: Moreover, how can analytical techniques ensure exact physics learning for domain-speciﬁc systems?

Original: Constrained Machine Learning.
Revised: Constrained Machine Learning.

Original: There are many applications of constrained machine learning (CML) in different systems.
Revised: There are many applications of constrained machine learning (CML) in different systems.

Original: For example, (Small et al., 2011) restricts the weight vectors with domain knowledge inthe support vector machine. (Li et al., 2021) proposes constrained adversarialmachine learning to enable the generation of adversarial examples under physical constraints.
Revised: (Small et al., 2011) restricts the weight vectors with domain knowledge in support vector machine and (Li et al., 2021) proposes constrained adversarial learning to generate adversarial examples under physical constraints.

Original: In deep learning applicartions, (Zhao et al., 2019) employs constraints from conservation law to restrict the output variables of artiﬁcial neural networks.
Revised: In deep learning applications, (Zhao et al., 2019) employs constraints from conservation law to restrict the output variables of DNN.

Original: (Velloso & Van Hentenryck, 2020; Fioretto et al., 2020a) solve optimal power ﬂow problems using deep learning with physical constraints, where the output variablesof a DNN are constrained to guaranteea feasible solution.
Revised: (Velloso & Van Hentenryck, 2020; Fioretto et al., 2020a) solve optimal power ﬂow problems using DNN with physical constraints, where the output variables are biased to guarantee feasible solution.

Original: (Tran et al., 2020) restricts the input to DNN into different groups to guarantee privacy.
Revised: (Tran et al., 2020) restricts the input to DNN into different groups to guarantee privacy.

Original: Among these studies, (Fioretto et al., 2020b) proposes a general framework to optimize DNN with  constraints using Lagrangian duality, where the constraints on the output are added.
Revised: Among these studies, (Fioretto et al., 2020b) optimizes DNN with output constraints using Lagrangian duality.

Original: These CML methods can solve their own problems effectively; however, there is a need for general principles to embed the physical constraints systematically with efﬁcient solutions.
Revised: These CML methods can solve their own problems effectively; however, there is a need for general principles to embed the physical constraints systematically with efﬁcient solutions.

Original: Data Forecasting.
Revised: Data Forecasting.

Original: To enhance physical robustness, we propose to conduct cross-model validation based on forecasting out-of-sample data to enrich the training dataset.

Original: The FOCV approach provides cooperative opportunities utilizing and updating physical-modeling knowledge (e.g., laws) as well as real-time data for responsive control.

Original: For previous  forecasting methods, autoregressive moving average (ARMA) and autoregressive integrated moving average (ARIMA) predict the evolving variables based on their lagged values (Gilbert, 2005).
Revised: For previous data forecasting methods, autoregressive moving average (ARMA) and autoregressive integrated moving average (ARIMA) predict the evolving variables based on their lagged values (Gilbert, 2005).

Original: To capture nonlinear correlations in the shortterm and long-term, there are many deep learning-based methods, e.g., recurrent Neural Network (RNN), long short-term memory (LSTM), bidirectional LSTM (BiLSTM), gated recurrent units (GRUs), and variational autoencoder (VAE) (Zeroual et al., 2020).
Revised: To capture nonlinear correlations in the short-term and long-term, there are many deep learning-based methods, e.g., recurrent Neural Network (RNN), long shortterm memory (LSTM), bidirectional LSTM (BiLSTM), gated recurrent units (GRUs), and variational autoencoder (VAE) (Zeroual et al., 2020).

Original: To forecast the result with conﬁdence, methods like Gaussian process (GP) (Quinonero-Candela & Rasmussen, 2005), Quantile Regression (Maciejowska et al., 2016) and deep learning-based models (Salinas et al., 2020) afford an efﬁcient way to predict distributions for the future data.
Revised: To forecast the result with conﬁdence, methods like Gaussian process (GP) (Quinonero-Candela & Rasmussen, 2005), Quantile Regression (Maciejowska et al., 2016) and deep learning-based models (Salinas et al., 2020) afford an efﬁcient way to predict distributions for the future data.

Original: Since the uncertainty evaluation is important to integrate the forecast data into the physical learning process, probabilistic methods are presented to estimate probability density functions in the proposed FOCV.
Revised: Since the uncertainty evaluation is important to integrate the forecast data into the physical learning process, probabilistic methods are presented to estimate probability density functions in the proposed FOCV.

Original:  M ETHODS
Revised:  M ETHODS

Original: The PNN-RISE approach is to understand the reason why a “good” sparse ﬁtting is not physical veriﬁable, to design physical principles to guide the ﬁtting internally based on a ﬂexible physical NN structure, and to create a cost-free active learning method for extrapolation at unseen operating points.3. A “G OOD ” SPARSE FITTING CAN FAIL
Revised: The PNN-RISE approach is to understand the reason why a “good” sparse ﬁtting is not physical veriﬁable, to design physical principles to guide the ﬁtting internally based on a ﬂexible PNN structure, and to create a cost-free active learning method for extrapolation at unseen operating points.

Original: Symbolic regression (SR) is a common approach to ﬁnd equation operators ( { + , − , × , / } ) and expressions like ( x 2 , cos( x )) , etc.
Revised: Symbolic regression (SR) is a common approach to ﬁnd equation operators ( { + , − , × , / } ) and expressions like ( x 2 , cos( x )) , etc.

Original: Existing papers show that if the following three conditions are satisﬁed, SR is capable to ﬁnd the true equation.

Original: • Apply the sparse regularization (Brunton et al., 2016; Sahoo et al., 2018; Champion et al., 2019; Petersen, 2019;
Revised: Existing papers (Brunton et al., 2016; Sahoo et al., 2018; Champion et al., 2019; Petersen, 2019;

Original: Kim et al., 2020). Achieve zero training error (Brunton et al., 2016; Champion et al., 2019; Petersen, 2019). • Utilize cross validation to avoid overﬁtting (Sahoo et al., 2018; Kim et al., 2020).
Revised: Kim et al., 2020) show that if the following three conditions are satisﬁed, SR is capable to ﬁnd the true equation: 1) apply the sparse regularization, 2) achieve nearly zero training error, and 3) utilize cross validation to avoid overﬁtting.

Original: However, we observe that SR methods are prone to non-physical solutions even when all three assumptions are satisﬁed.
Revised: However, we observe that SR methods are prone to non-physical solutions even when all three assumptions are satisﬁed.

Original: For example, to learn the function y =
Revised: For example, to learn the function y =

Original: x 1 x 2 + x 2 x 3 from data, different SR methods can easily get trapped in local optima.
Revised: x 1 x 2 + x 2 x 3 from data, different SR methods can easily get trapped in local optima.

Original: To see if these observation is special, we test different cases, where Fig.
Revised: To see if these observation is special, we test different cases, where Fig.

Original: 2 shows a table summary of four different cases, with integer coefﬁcients or not.
Revised: 2 shows a table summary of four different cases, with integer coefﬁcients or not.

Original: We also test different numbers for multiplication and summation.
Revised: We also test different numbers for multiplication and summation.

Original: From allthe examples, one can see that the recovered objective has sparse coefﬁcients and negligible error (loss objective L ) at non-desirable local optimums.
Revised: From all examples, one can see that the recovered objective has sparse coefﬁcients and negligible error (loss objective L ) at non-desirable local optimums.

Original: To make the realistic case study rather than  toy examples, we use a  power system case study and visualize the objective surface in Fig. 3 (left). In a small parameter range, there are points achieving the global optimum or local optima that are very close to the global optimum.
Revised: Other than the toy examples, we conduct a realistic power system case study and visualize the objective surface in Appendix A. Similarly, for a small parameter range, there are points achieving the global optimum or local optima that are very close to the global optimum.

Original: However, only one of them is the physical ground truth.
Revised: However, only one of them is the physical ground truth.

Original: For example, one can go from the red point to a non-physical solution with an error close to zero, even when the sparsity regularization is enforced.
Revised: For example, one can go from the red point to a non-physical solution with an error close to zero, even when the sparsity regularization is enforced.

Original: Therefore, we can see that “good” sparse ﬁtting is insufﬁcient to avoid overﬁtting.
Revised: Therefore, we can see that “good” sparse ﬁtting is insufﬁcient to avoid overﬁtting.

Original: Thus, we need to avoid overﬁtting for physical systems not only by choosing sparse learning models , but also new ways to constrain the learning process and avoid non-physical solutions.3. P ROPOSED METHOD : PHYSICAL NEURAL
Revised: Thus, we need to avoid overﬁtting for physical systems not only by choosing sparse learning models , but also new ways to constrain the learning process and avoid non-physical solutions.

Original: For complex physical systems, the governing functions are formed by basic operations such as additive and multiplicative accumulations of variables.
Revised: For complex physical systems, the governing functions are formed by basic operations such as additive and multiplicative accumulations of variables.

Original: For example, the topology of power system can be represented as a graph, where the interaction between different nodes along the lines is represented by polynomials of the voltages for power ﬂow (Yu et al., 2017).
Revised: For example, the topology of power system can be represented as a graph, where the interaction between different nodes along the lines is represented by polynomials of the voltages for power ﬂow (Yu et al., 2017).

Original: To combine the beneﬁt of universal approximation and the freedom to choose activation function, we propose to use a NN as a container to learn the representation of physical equations, and thus call it a physical neural network (PNN), denoted as f .
Revised: To combine the beneﬁt of universal approximation and the freedom to choose activation function, we propose to use a NN as a container to learn the representation of physical equations, and thus call it a physical neural network (PNN), denoted as f .

Original: To train f , the training data set is { x i , y i } Ni =1 , where x i and y
Revised: To train f , the training data set is { x i , y i } Ni =1 , where x i and y

Original: i are the i th input and output of f , and N is the number of training samples.
Revised: i are the i th input and output of f , and N is the number of training samples.

Original: To construct the physical equation in PNN, the key is to develop a sparse selection mechanism to automatically select part of the variables to join summation, abstraction, multiplication, and division.
Revised: To construct the physical equation in PNN, the key is to develop a sparse selection mechanism to automatically select part of the variables to join summation/abstraction and multiplication/division.

Original: For sparse selection,some existing work utilizes sparse regularization like LASSO(least absolute shrinkage and selection operator) forthe weight updates (Kim et al., 2020), introduces extra neural network modules like gates (Liu & Laili, 2018), or conducts symbolic generators (Petersen, 2019).
Revised: For sparse selection, existing work utilizes sparse regularization like LASSO for weight updates (Kim et al., 2020), introduces extra neural network modules like gates (Liu & Laili, 2018), or conducts symbolic generators (Petersen, 2019).

Original: In general, they demand the quantiﬁcation of extra hyper-parameters (i.e., the LASSO penalty) or parameters (i.e., parameters in gates or symbolic generators), which signiﬁcantly increases the possibility of local optima.
Revised: In general, they demand the quantiﬁcation of extra hyper-parameters (i.e., the LASSO penalty) or parameters (i.e., parameters in gates or symbolic generators), which signiﬁcantly increases the possibility of local optima.

Original: In order to address the problem towards limiting local optima, we advocate the design philosophy: enforce activation functions on weights for the sparse selection of features.
Revised: In order to address the problem towards limiting local optima, we advocate the design philosophy: enforce activation functions on weights for the sparse selection of features.

Original: Thus, no extra parameters are added, and the searching is only for physical parameters.
Revised: Thus, no extra parameters are added, and the searching is only for physical parameters.

Original: To elaborate on our design, we need to answer (1) which parameters need to be activated and (2) how to enforce the activation.
Revised: To elaborate on our design, we need to answer (1) which parameters need to be activated and (2) how to enforce the activation.

Original: For parameter activation, we start from the basic symbolic operations in neural networks, e.g., multiplications and divisions in the following forms. [Equation] where z j is the j th element of the layer next to the input.
Revised: For parameter activation, we start from the basic symbolic operations in neural networks, e.g., multiplications and divisions of the form z j = Π Ni x ip ij , where z j is the j th element of the layer next to the input.

Original: x i is the i th element of the input vector variable x and p ij is the index number of x i .
Revised: x i is the i th element of the input vector variable x and p ij is the index number of x i .

Original: Note p ij ∈ {− K, · · · , − 1 , 0 , 1 , · · · , K } , where the positive integer represents the multiplication, the possibility of 0 corresponds to the sparsity, and the negative integer represents the division.
Revised: Note p ij ∈ {− K, · · · , − 1 , 0 , 1 , · · · , K } , where the positive integer represents the multiplication, the possibility of 0 corresponds to the sparsity, and the negative integer represents the division.

Original: Then, each z j is a more complex symbol for the next operation. According to Equation (1), p ij is the optimization variable.
Revised: Then, each z j is a more complex symbol for the next operation and p ij is the optimization variable.

Original: To accommodate p ij into a neural network, we use logarithm and exponential operators to achieve a linear summation format (Trask et al., 2018),
Revised: To accommodate p ij into a NN, we adopt logarithm and exponential operators to achieve a linear summation format z j = exp (cid:0) (cid:80) Ni =1 p ij ln

Original: [Equation]
Revised: ( x i ) (cid:1) (Trask et al., 2018).

Original: Thus, the linear layer with weights p ij needs to be activated for arithmetic operations and sparsity.
Revised: Thus, the linear layer with weights p ij needs to be activated for arithmetic operations and sparsity.

Original: Considering the target integer range of p ij , it’s vital to know how to control the values for parameter enforcement activation.
Revised: Considering the target integer range of p ij , it’s vital to know how to control the values for parameter enforcement activation.

Original: Thus, we ﬁrst show how to enforce p ij ∈ {− 1 , 0 , 1 } .
Revised: We ﬁrst show how to enforce p ij ∈ {− 1 , 0 , 1 } .

Original: Since tanh function can restrict the output value to be in [ − 1 , 1] , we can utilize a positive scaling factor c 1 and tanh to activate some free and optimizable parameter w ij such that tanh ( c 1 w ij ) ∈ {− 1 , 1 } , where the scaling is to make sure the output is very close to the target − 1 or 1 .
Revised: Since tanh function can restrict the output value to be in [ − 1 , 1] , we can utilize a positive scaling factor c 1 and tanh to activate some free and optimizable parameter w ij such that tanh ( c 1 w ij ) ∈ {− 1 , 1 } , where the scaling is to make sure the output is very close to the target − 1 or 1 .

Original: Subsequently, the network needs to enforce 0 for sparsity.
Revised: Subsequently, the network needs to enforce 0 for sparsity.

Original: Following the same procedure, we create another sigmoid function to achieve σ ( c 2 w (cid:48) ij ) ∈ { 0 , 1 } , where σ is the sigmoid function, c 2 is the positive scaling factor, and w (cid:48)
Revised: We similarly create another sigmoid function to achieve σ ( c 2 w (cid:48) ij ) ∈ { 0 , 1 } , where σ is the sigmoid function, c 2 is the positive scaling factor, and w (cid:48)

Original: ij is another optimizable parameter.
Revised: ij is another optimizable parameter.

Original: To summarize, we have: [Equation] where p ij ∈ ( − 1 , 1) .The visualized in Fig.
Revised: To summarize, we have p ij = tanh ( c 1 w ij ) · σ ( c 2 w (cid:48) ij ) , where p ij ∈ ( − 1 , 1) .

Original: 3 (right) shows that the multiplication of tanh and sigmoid functions not only generates a smooth optimization surface but also naturally results in limited parameters.
Revised: The multiplication of tanh and sigmoid functions not only generates a smooth optimization surface but also naturally results in limited parameters.

Original: After such a design, we have a third linear layer  to achieve y k = (cid:80) j m jk z j , where m jk denote the layer weights for linearweighted summation.
Revised: After that, a third linear layer is used to obtain y k = (cid:80) j m jk z j , where m jk denote the layer weights for linear summation.

Original: Since the summation and abstraction are incorporated into the linear layer, we complete the function y = f
Revised: Since the summation and abstraction are incorporated into the linear layer, we complete the function y = f

Original: ( x ) .
Revised: ( x ) .

Original: Using such a design as a basic unit, we can extend from {− 1 , 0 , 1 } to {− K, · · · , − 1 , 0 , 1 , · · · , K } for different physical systems.
Revised: Using such a design as a basic unit, we can extend from {− 1 , 0 , 1 } to {− K, · · · , − 1 , 0 , 1 , · · · , K } for different physical systems.

Original: Such a designintelligently converts the non-convex multiplication to a convex form of linear summation, which is efﬁcient for NN training.
Revised: Such a design converts the non-convex multiplication to a convex form of linear summation for NN training.

Original: Based on this design, we show the following property of our PNN to guarantee the ﬁnding of stationary points in the optimization.Theorem 1 (Biconvexity of PNN) . PNN is biconvex on the parameters of the logarithm layer and the linear summation layer. Then, the searching algorithm can ﬁnd the stationary points of PNN.
Revised: Based on this design, we show the following property of our PNN to guarantee the ﬁnding of stationary points in the optimization.

Original: Proof.

Original: First, we deﬁne the biconvexity as follows (Gorski et al., 2007).

Original: Deﬁnition 1 (Biconvexity) .

Original: A function φ : A × B → R is called biconvex if f ( a, b ) is convex in a for ﬁxed b ∈ B , and f ( a, b ) is convex in b for ﬁxed a ∈ A .

Original: Then, we prove PNN is biconvex. Recall for the  PNN function y = f ( x ) , the k th element can be written as: y k = (cid:80) j m jk z j = (cid:80) j m jk exp (cid:0) (cid:80) Ni =1 p ij ln
Revised: The following proposition states the ﬁnding of stationary points using PNN with the design and the proof is inlcuded in Appendix.

Original: ( x i ) (cid:1) .

Original: ∀ j, k , if m jk is known, we need to prove the convexity of f with respect to p ij , ∀ 1 ≤ i ≤ N .

Original: Since the linear combination (cid:80) Ni =1 p ij ln

Original: ( x i ) is convex with respect to p ij , and the exponential function is convex, the convex function composition implies

Original: that exp (cid:0) (cid:80) Ni =1 p ij ln

Original: ( x i ) (cid:1) is convex with respect to p ij .

Original: Therefore, ∀ j, k , if m jk is known, the function y k

Original: = (cid:80) j m jk exp (cid:0) (cid:80) Ni =1 p ij ln ( x i ) (cid:1) is convex with respect to p ij , ∀ 1 ≤ i ≤ N .On the other hand, ∀ i, j , if p ij is known, y k
Revised: Proposition 1 (Biconvexity of PNN) .

Original: = (cid:80)

Original: j m jk exp (cid:0) (cid:80) Ni =1 p ij ln

Original: ( x i ) (cid:1) is a linear combination of m jk .

Original: Therefore, y k is convex with respect to m jk given p ij .

Original: Therefore, PNNfunction y = f ( x ) is biconvex with respect to the logarithm layer and the linear summation layer.
Revised: PNN is biconvex on the parameters of the logarithm layer and the linear summation layer.

Original: The biconvexity can lead to an efﬁcient alternate convex search (ACS) (Gorski et al., 2007) to seek for optimal solutions. Namely, ACS will ﬁx one part of parameters to search for the other part iteratively. Due to biconvexity, in each iteration, ACS will solve a convex problem.
Revised: Then, the searching algorithm can ﬁnd the stationary points of PNN.

Original: Then, Corollary 4.10 in (Gorski et al., 2007) guarantees that the searched points will converge to stationary points of f ( x ) .

Original: Theorem 1 indicates that we can ﬁnd the stationary points (i.e., saddle points, local optima, and global optima) of the PNN.
Revised: Proposition 1 indicates that PNN can ﬁnd the stationary points (i.e., saddle points, local optima, and global optima).

Original: However, the following questions arises: (1) can we ﬁnd the globaloptima? (2) according to our discovery in Section 3.1, can the found global optima  represent the true physics?In the next section, we show the design of embedding physical principles to PNN to both obtain global optima and guarantee the physical meaning of the global optima. Especially, we propose sound theories to support our ideas in Theorem 2.
Revised: Subsequently, can we ﬁnd the global optima and can it represent the true physics?

Original: 3. E MBED PHYSICAL PRINCIPLES VIA CONSTRAINING PNN
Revised: 3. E MBED PHYSICAL PRINCIPLES VIA CONSTRAINING PNN

Original: After providing a ﬂexible design to reduce the parameter space for limiting possibilities of local optimums, we need comprehensive principles to avoid non-physical local optimum in Sec 3.1.
Revised: After providing a ﬂexible design to limit possibilities of local optima, we need comprehensive principles to avoid non-physical local optima.

Original: If constraints derived from these principles can be convex, they will be even more beneﬁcial in solving or reducing the issue of non-physical local optimums.
Revised: If the derived constraints can be convex, they will provide guarantees in solving or reducing the issue of non-physical local optima.

Original: For physical systems, two important perspectives are physical existence and systematic behaviors.
Revised: For physical systems, two important perspectives are physical existence and systematic behaviors.

Original: As the components are physically located, there are geographical or electrical distances, etc.
Revised: As the components are physically located, there are geographical or electrical distances, etc.

Original: Therefore, one common constraint through physical systems is the symmetric (S) property, e.g., a branch connecting two components represents a common coupling.
Revised: Therefore, one common constraint through physical systems is the symmetric (S) property, e.g., a branch connecting two components represents a common coupling.

Original: In addition to the coupling terms, the links between different components also have their physical limitations, leading to a range (R) constraint.
Revised: In addition, the links between different components also have their physical limitations, leading to a range (R) constraint.

Original: At the system level, the physical system will continue making an impact to the environment, leading to an inertia (I) concept, which is the momentum from a physical objectresults because physical systems typically have natural inertia, whether it is strong or weak.
Revised: At the system level, the physical system will continue making an impact to the environment, leading to an inertia (I) concept, which is the momentum from a physical object because physical systems typically have natural inertia, whether it is strong or weak.

Original: For man-made systems, there could be additional artiﬁcial inertia due to factors such as guaranteeing stability for sustainable operation, e.g., in power systems.
Revised: For man-made systems, there could be additional artiﬁcial inertia due to factors such as guaranteeing stability for sustainable operation, e.g., in power systems.

Original: Finally, such inertia does not only live in the temporal space but also in the input/output data space.
Revised: Finally, such inertia does not only live in the temporal space but also in the input/output data space.

Original: For example, the physical equations are typically continuous functions.
Revised: For example, the physical equations are typically continuous functions.

Original: These concepts open the door to investigate the data range beyond the training and testing data.
Revised: These concepts open the door to investigate the data range beyond the training and testing data.

Original: Such generalized inertia leads an important extrapolation (E) property across different physical systems to forecast information yet to be observed but representative of plausible scenarios..
Revised: Such generalized inertia leads an important extrapolation (E) property across different physical systems to forecast information yet to be observed but representative of plausible scenarios.

Original: The constraints based on different properties of physical systems are called RISE concept by combing their initials. For illustrating the completeness of these principles, Fig. 4 presents the relationship among the four RISE concepts.
Revised: The constraints based on different properties of physical systems are called RISE concept by combing their initials, which is demonstrated via a ﬁgure in Appendix.

Original: This subsection introduces the ﬁrst three to constrain the NN structure in Subsection 3.2; while Subsection 3.4, introduces the FOCV method for physical system extrapolation.
Revised: This subsection introduces the ﬁrst three to constrain PNN in Subsection 3.1 while Subsection 3.3 introduces the FOCV method for physical system extrapolation.

Original: Range (R): Shape Constraint on Physical Parameters: While local optima generate many possible parameters to minimize the prediction error, the true physical parameters of corresponding features usually yield speciﬁc shapes.
Revised: Range (R): Shape Constraint on Physical Parameters: While local optima generate many possible parameters to minimize the prediction error, the true physical parameters of corresponding features usually yield speciﬁc shapes (Cotter et al., 2019).

Original: For example, the range of power line admittance (reciprocal of impedance) of power distribution systems can be speciﬁed with voltage level and line materials.
Revised: For example, the range of power line admittance (reciprocal of impedance) of power distribution systems can be speciﬁed with voltage level and line materials.

Original: In addition to the bound on value range of independent parameters, there can also be shape constraints on the conditional relationships among parameters, e.g., longer lines have larger resistance than shorter lines.
Revised: In addition to the bound on value range of independent parameters, there can also be shape constraints on the conditional relationships among parameters, e.g., longer lines have larger resistance than shorter lines.

Original: Such a rule speciﬁed by the physical shape constraint function can ﬁlter out many local optima even with low errors.
Revised: Such a rule speciﬁed by the physical shape constraint function can ﬁlter out many local optima even with low errors.

Original: In order to enforce the learned physical parameters W p within the constraint, a customized gate function is proposed to ﬁlter the weights, as shown in the ﬁrst subplot of Fig. 4. [Equation] (cid:100) g [Equation]
Revised: In order to enforce the learned physical parameters W p within the constraint, a customized gate function is proposed to weights g ( · ) is a function of physical weights W p to represent the type of shape constraint.

Original: for g ( W p ) ≥ (cid:100) g p (cid:101) [Equation] (4) where g ( · ) is a function of physical weights W p to represent the type of shape constraint. For example, g ( · ) is linear if the constraint is on a speciﬁc value range of independent parameters, where (cid:98) g p (cid:99) and (cid:100)
Revised: As an example, g ( · ) is linear for speciﬁc value constraint of independent parameters, where (cid:98) g p (cid:99) and (cid:100)

Original: g p (cid:101) are the lower and upper bounds, and g (cid:48)
Revised: g p (cid:101) are the lower and upper bounds, and g (cid:48)

Original: ( W p ) are the constrained weights.
Revised: ( W p ) are the constrained weights.

Original: Symmetry (S): Constraints on Physical Couplings: For physical systems with multiple outputs, there are variable couplings among different outputs, e.g., one term is contained in two outputs.
Revised: Symmetry (S): Constraints on Physical Couplings: For physical systems with multiple outputs, there are variable couplings among different outputs, e.g., one term is contained in two outputs.

Original: Let’s use the power system as an example.
Revised: Let’s use the power system as an example.

Original: A certain amount of power ﬂowing from one place to another means that the same amount of power will be subtracted and added simultaneously at the two places.
Revised: A certain amount of power ﬂowing from one place to another means that the same amount of power will be subtracted and added simultaneously at the two places.

Original: Based on our observations, most of the local optima of NN do not satisfy such a rule.
Revised: Based on our observations, most of the local optima of NN do not satisfy such a rule.

Original: Thus, the property(the third subplot of Fig. 4) is embedded by minimizing {+L Mutual = (cid:107) W p − W Tp (cid:107) 
+}
Revised: Thus, the property is embedded by minimizing {+L Mutual = (cid:107) W p − W Tp (cid:107) 
+}

Original: [Equation]
Revised: + (cid:107) rank ( W pi ) − N mutual (cid:107) 0 .

Original: The ﬁrst term is an l 2 norm penalty on the symmetry ofthe recovered physical parameters and the second term is to further ensure that the number of mutual terms N mutual satisﬁes the system property, e.g.,the power system equation has N mutual = 2 , becausethe two nodes are connected by one power line.
Revised: The ﬁrst term is an l 2 norm penalty on the symmetry of recovered physical parameters and the second term is to further ensure that the number of mutual terms N mutual satisﬁes the system property, e.g., power system equation has N mutual = 2 , because two nodes are connected by one power line.

Original: Inertia (I): Constraints on Physical Tendency: For physical systems, the states are typically continuous and cannot change suddenly, leading to physical inertia.
Revised: Inertia (I): Constraints on Physical Tendency: For physical systems, the states are typically continuous and cannot change suddenly, leading to physical inertia.

Original: Such a property can be embedded by bounding the outputs of two adjacent time slots, shown in the second subplot of Fig.
Revised: Such a property can be embedded by bounding the outputs of two adjacent time slots.

Original: To ensure the partial convexity, the Euclidean norm is modeled as a penalty in loss function.
Revised: To ensure the partial convexity, the Euclidean norm is modeled as a penalty in loss function L Inertia = (cid:107) f

Original: [Equation]
Revised: ( x t +1 )

Revised: − f ( x t ) (cid:107) 2 .

Original: 3. M ODEL B OOSTING : C ROSS M ODEL V ALIDATION FOR P HYSICAL E XACTNESS AT U NKNOWN D ATA R EGION
Revised: C ROSS -M ODEL V ALIDATION FOR P HYSICAL E XACTNESS AT U NKNOWN D ATA R EGION

Original: Subsections 3.2 and 3.3 show how to avoid overﬁtting by considering the NN-based model design and ﬁtness of data to the physical laws.
Revised: Subsections 3.1 and 3.2 show how to avoid overﬁtting by considering the NN-based model design and ﬁtness of data to the physical laws.

Original: However, all these designs must be within the training data range.
Revised: However, all these designs must be within the training data range.

Original: What will happen beyond the training data range, e.g., a new operation point of physical systems unseen in the past?
Revised: What will happen beyond the training data range, e.g., a new operation point of physical systems unseen in the past?

Original: Actually, determining the performance region is the most useful part for physical systems modeling after understanding the physical laws, e.g., for universal data ﬁtting at arbitrary points.
Revised: Actually, determining the performance region is the most useful part for physical systems modeling after understanding the physical laws, e.g., for universal data ﬁtting at arbitrary points.

Original: For example, Fig. 5 shows three globally optimal or near globally optimal lines (red, blue, and green) that can ﬁt data equally well within the training dataset.
Revised: For example, Fig. 3 shows three globally optimal or near globally optimal lines (red, blue, and green) that can ﬁt data equally well within the training dataset.

Original: These three lines can represent the problem in Subsection 3.1 as well.

Original: One way to avoid the selection dilemma is to conduct an active inquiry out of the historical data range for a selection.
Revised: One way to avoid the selection dilemma is to conduct an active inquiry out of the historical data range for a selection.

Original: For example, Fig.illustrates that a few out-of-sample data points in grey can regularize the learning to the ground truth (dotted black line).
Revised: For example, Fig. 3 illustrates that a few out-of-sample data points in grey can regularize the learning to the ground truth (dotted black line).

Original: However, the costs of such an inquiry can be quite expensive as one inquiry may not be enough.
Revised: However, the costs of such an inquiry can be quite expensive as one inquiry may not be enough.

Original: Also, given the numerous networks that the internet of everything (IoE) has, it is infeasible and sometimes impossible to conduct an inquiry for out-of-sample estimation.
Revised: Also, given the numerous networks that the internet of everything (IoE) has, it is infeasible and sometimes impossible to conduct an inquiry for out-of-sample estimation.

Original: Therefore, how can a learning approach avoid any cost of active learning but still explore the generalization capability via some alternative active learning?
Revised: Therefore, how can a learning approach avoid any cost of active learning but still explore the generalization capability via some alternative active learning?

Original: Extrapolation (E): Constraints on Physical Validation at Unknown Regions: To conduct active learningfreely and without an inquiry cost, wepropose to implement physically meaningful forecasts via a probabilistic model for data pointsthat do not exist inthe historical data points.
Revised: Extrapolation (E): Constraints on Physical Validation at Unknown Regions: To conduct active learning without an inquiry cost, we implement physically meaningful forecasts via a probabilistic model for data points not existing in historical measurements.

Original: Such a learning objective will be combined with the learning method in the previous subsections to provide another level of physical guidance.
Revised: The learning objective is to provide another level of physical guidance for the constrained PNN in previous subsections.

Original: Speciﬁcally, an ensemble of forecasts is conductednot only for  the mean but also for the variance, as the forecast ability decreases when the inquiry point is far away from the data range.
Revised: Speciﬁcally, an ensemble of forecasts is conducted for both the mean and variance, as the forecast ability decreases when the inquiry point is far away from the data range.

Original: For such a probabilistic forecast, another important design is to embed the physical form into the forecast so that theforecast model can cross-validate the learning models.
Revised: Another important design is to embed the physical form into the forecast so that the model can cross-validate the learning models.

Original: Thus, we propose to utilize Gaussian Process (GP), which not only has computational efﬁciency in probabilistic forecasts but also provides the ﬂexibility to embed physical forms into its kernel function.
Revised: We propose to utilize Gaussian Process (GP), which not only has computational efﬁciency in probabilistic forecasts but also provides the ﬂexibility to embed physical forms into its kernel function.

Original: The forecasting process is shown in the last subplot of Fig.

Original: For training the GP, the data in represented as { x i , ˜ x i } Ni =1 , where x i represents the input samples and ˜ x
Revised: For training the GP, the data in represented as { x i , ˜ x i } Ni =1 , where x i represents the input samples and ˜ x

Original: i represents a sample point k -step later than x i .
Revised: i represents a sample point k -step later than x i .

Original: Based on such data, GP assumes prior function values in a Bayesian framework, e.g., p ( g | x 1 , · · · , x N ) =
Revised: Based on such data, GP assumes prior function values in a Bayesian framework, e.g., p ( g | x 1 , · · · , x N ) =

Original: N ( 0 , K ) , where g is a vector of latent function values, N represents normal distribution, and K is a kernelized covariance matrix.
Revised: N ( 0 , K ) , where g is a vector of latent function values, N represents normal distribution, and K is a kernelized covariance matrix.

Original: In particular, we can add different kernel mappings to calculate K based on our priors of the physical knowledge.
Revised: In particular, we can add different kernel mappings to calculate K based on our priors of the physical knowledge.

Original: For example, for power systems, the polynomial of order 2 is often utilized to represent the relationship between voltages and powers (Yu et al., 2017).
Revised: For example, for power systems, the polynomial of order 2 is often utilized to represent the relationship between voltages and powers (Yu et al., 2017).

Original: Using data with kernel knowledge, we obtain a predicting functiong (cid:63) via Bayes rule:
Revised: Using data with kernel knowledge, we obtain a predicting function via Bayes rule p ( g (cid:63) ) =

Original: [Equation]
Revised: [Equation]

Original: Based on the Gaussianity, Equation (7) has a closed-form solution g (cid:63) .
Revised: which has a closed-form solution g (cid:63) based on Gaussianity.

Original: Based on g (cid:63) , the goal is to predict the k -step later points for any input sample.
Revised: The goal is to predict the k -step later points for any input sample.

Original: Speciﬁcally, one can plug-in the input data x and obtain the distribution of the k -step points p ( g (cid:63) (cid:0) x ) (cid:1) .
Revised: Speciﬁcally, one can plug-in the input data x and obtain the distribution of the k -step points p ( g (cid:63) (cid:0) x ) (cid:1) .

Original: Thus, the GP-based forecasting not only brings the forecast point ˜ x k , e.g., the mean of p ( g (cid:63) (cid:0) x ) (cid:1) , but also the conﬁdence level α k based on the standard deviation.
Revised: Thus, the GP-based forecasting not only brings the forecast point ˜ x k , e.g., the mean of p ( g (cid:63) (cid:0) x ) (cid:1) , but also the conﬁdence level α k based on the standard deviation.

Original: Similarly, we can obtain ˜ y k and β k , where ˜ y k is the forecast point for the output data y k and β k is the corresponding conﬁdence level.
Revised: Similarly, we can obtain ˜ y k and β k , where ˜ y k is the forecast point for the output data y k and β k is the corresponding conﬁdence level.

Original: Fig.
Revised: Fig.

Original: 5 illustrates the results of GP, where the forecast points have smaller uncertainty if they are closer to the dataset boundary.
Revised: 3 illustrates the results of GP, where the forecast points have smaller uncertainty if they are closer to the dataset boundary.

Original: With the forecast results from above, we can add ˜ x k ( ˜ y k ) to the training data set, like active learning.
Revised: With the forecast results from above, we can add ˜ x k ( ˜ y k ) to the training data set, like active learning.

Original: However, our method is different as active learning, which is associated with a cost.
Revised: However, our method is different as active learning, which is associated with a cost.

Original: With the cost, active learning can provide a highly accurate label.
Revised: With the cost, active learning can provide a highly accurate label.

Original: But, our new data points are always associated with conﬁdence bounds.
Revised: But, our new data points are always associated with conﬁdence bounds.

Original: The good news is that the FOCV method can provide inﬁnite data points without a cost.
Revised: The good news is that the FOCV method can provide inﬁnite data points without a cost.

Original: Therefore, different than active learning, FOCV uses all the data points with unequal weights, e.g., the inverse of conﬁdence level, to achieve model selection.
Revised: Therefore, different than active learning, FOCV uses all the data points with unequal weights, e.g., the inverse of conﬁdence level, to achieve model selection.

Original: Therefore, we propose an integral-based regularization for extrapolation:{+extrapolation L Extrapol = 
+}
Revised: Therefore, we propose an integral-based regularization for {+extrapolation L Extrapol = 
+}

Original: [Equation]
Revised: α k β k (cid:80) Kk =1 ( ˜ y k − f ( ˜ x k )) 2 .

Original: With extrapolation (E), the proposed methods complete the derivation of RISE.
Revised: With extrapolation (E), the proposed methods complete the derivation of RISE.

Original: It’s important to note that in RISE, the factors are decoupled out of the PNN.
Revised: It’s important to note that in RISE, the factors are decoupled out of the PNN.

Original: Furthermore, the factors can be relaxed into a convex form, which signiﬁcantly increases the probability of ﬁnding the global optima.
Revised: Furthermore, the factors can be relaxed into a convex form, which signiﬁcantly increases the probability of ﬁnding the global optima.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: Theorem 2 highlights the regularization can lead to strong convexity over a set of closed and convex regions of the parameter space, i.e., the piecewise strong convexity.
Revised: Theorem 1 highlights the regularization can lead to strong convexity over a set of closed and convex regions of the parameter space, i.e., the piecewise strong convexity.

Original: Theorem 2 (Piecewise strong convexity of PNN with regularization) .
Revised: Theorem 1 (Piecewise strong convexity of PNN with regularization) .

Original: With inertia and symmetry regularization, the loss function of theproposed PNN is piecewise strongly convex in the parameter space.
Revised: With inertia and symmetry regularization, the loss function of the PNN is piecewise strongly convex in the parameter space.

Original: Proof.
Revised: Proof.

Original: For the simplicity of later derivations, we denote W as the  set of all parameters in PNN. Then, we denote l ( W ) as the loss function, i.e., mean square error, of the PNN.
Revised: For the derivation simplicity, we denote W as the parameter set of PNN and denote l ( W ) as the loss function, i.e., mean square error.

Original: The range constraint (R) and the extrapolation constraint (E) are included in the loss function as they are included in the PNN representation.
Revised: The range constraint (R) and the extrapolation constraint (E) are included in l ( W ) as they are included in the PNN representation.

Original: Subsequently, the inertia (I) and symmetry (S) regularizations help to deﬁne l 1 ( W ) =
Revised: Subsequently, the inertia (I) and symmetry (S) regularizations help to deﬁne l 1 ( W ) =

Original: l ( W ) + λ ( L Inertia + L Mutual ) as the regularized loss function for training, where λ is a positive penalty term.
Revised: l ( W ) + λ ( L Inertia + L Mutual ) as the regularized loss function for training, where λ is a positive penalty term.

Original: The second derivative of the regularized loss function can help to determine the convexity. We write the second derivative as follows:
Revised: To determine convexity, we write the second derivative of the regularized loss function d

Original: [Equation]
Revised: [Equation]

Original:  where h ( W ) is the second derivative of L Inertia + L Mutual .
Revised: tX ) + λh ( W ) , where h ( W ) is the second derivative of L Inertia + L Mutual .

Original: For inertia, Equation (6) has a quadratic form and is convex.
Revised: L Inertia has a quadratic form to be convex.

Original: For symmetry in Equation (5), the l 0 norm can be relaxed to l 1 norm in optimization.
Revised: For symmetry L Mutual , the l 0 norm can be relaxed to l 1 norm in optimization.

Original: Thus, regularization is convex and h ( W ) is a positive value.
Revised: Thus, regularization is convex and h ( W ) > 0 .

Original: However, due to the non-convexity of neural network, the ﬁrst term d 2 dt 2 | t =0{+=
+} l ( W + tX ) is generally not positive semi-deﬁnite.
Revised: However, due to the non-convexity of neural network, the ﬁrst term d 2 dt 2 | t {+=
+} l ( W + tX ) is generally not positive semi-deﬁnite.

Original: The following lemma in (Milne, 2018) proves a lower bound for d 2 dt 2 | t =

Revised: The following lemma in (Milne, 2018) proves a lower bound for d 2 dt 2 | t =

Original: l ( W + tX ) with respect to the loss value of l .
Revised: l ( W + tX ) with respect to the loss value of l .

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose ||
Revised: Suppose ||

Original: y i ||
Revised: y i ||

Original: 2 ≤  r for all 1 ≤ i ≤ N , where r is a positive value.
Revised: 2 ≤ r, r > 0 , ∀ 1 ≤ i ≤ N .

Original: The second derivative of l in direction X satisﬁes:
Revised: The second derivative of l in direction X satisﬁes

Revised: d 2 dt 2 |

Original: [Equation]
Revised: [Equation]

Original: where H is the number of hidden layers in a neural network, || · ||(cid:63) is a deﬁned norm such that
Revised: 0 l ( W + tX ) > −√ 2 H ( H − [Equation] ||

Revised: W ||

Original: [Equation]
Revised: [Equation]

Revised: ||

Revised: X

Original: According to Lemma 1, the second derivative of l islower-bounded by a term that depends on l value.
Revised: || 22 rl ( W ) 1 / 2 , where H is the number of hidden layers in a neural network, || · || (cid:63) is a deﬁned norm such that || W || (cid:63) = max 0 ≤ i ≤ H || W i

Revised: || 2 .

Revised: According to Lemma 1, the second derivative of l is lower-bounded by a term that depends on l value.

Original: Then, for regions with small loss values, the positive value h ( W ) helps to restrict d 2 dt 2 | t =

Revised: Then, for regions with small loss values, the positive value h ( W ) helps to restrict d 2 dt 2 | t =

Original: l 1 ( W + tX )
Revised: l 1 ( W + tX )

Original: to a positive number, which determines the shape of the region to be convex.
Revised: to a positive number, which determines the shape of the region to be convex.

Original: Mathematically, we deﬁne the region U ( λ, θ ) , where λ > θ > 0 , by 
Revised: Mathematically, we deﬁne the region U ( λ, θ ) , where λ > θ > 0 , by U ( λ, θ ) = { W | l ( W ) 1 / 2 || W || H − 1 (cid:63) < ( λ − θ ) h ( W ) √ 2 H ( H +1)

Original: [Equation]
Revised: r || X || 22 } .

Original: It’s obvious
Revised: It’s obvious

Original: that ∀ W ∈ U ( λ, θ ) , d 2 dt 2 | t =0 l 1 ( W + tX ) > 0 .
Revised: that ∀ W ∈ U ( λ, θ ) , d 2 dt 2 | t =0 l 1 ( W + tX ) > 0 .

Original: Namely, within the region of U ( λ, θ ) , the regularized loss function is strongly convex.
Revised: Namely, within the region of U ( λ, θ ) , the regularized loss function is strongly convex.

Original: In general, U ( λ, θ ) is not a convex set sincethere are many local optima pointsthat are disconnected and have small l values.
Revised: In general, U ( λ, θ ) is not a convex set since many local optima points are disconnected and have small l values.

Original: However, according to measure theory, we can utilize a set of closed convex sets to completely cover U ( λ, θ ) , leading to piecewise strong convexity of PNN over these closedconvex sets.
Revised: However, based on measure theory, we can utilize a set of closed convex sets to completely cover U ( λ, θ ) , leading to piecewise strong convexity of PNN over these closed sets.

Original: The piecewise strong convexity fromconvex regularization of RISE  guarantees that(1) we can ﬁnd the global optimal point for each sub-regions, and(2) the global optima satisfy RISE constraints for physical parameters.
Revised: Therefore, the piecewise strong convexity from RISE regularization guarantees that we can ﬁnd the global optimal point for each sub-regions, and the global optima satisﬁes constraints for physical parameters.

Original:  E XPERIMENTS4. D ATA PREPARATION
Revised:  E XPERIMENTS

Revised: Data Preparation.

Original: In experiments, we ﬁrst use synthetic datasets for illustration and introduce real physical system datasets of power systems and mass-damper systems as the underlying physical systems for model training and comparison.
Revised: In experiments, we ﬁrst use synthetic datasets for illustration and introduce real physical system datasets of power systems and mass-damper systems as the underlying physical systems for model training and comparison.

Original: In particular, the dataset descriptions are  as follows. Synthetic dataset . In order to visualize the learning performance, we generate random samples of x from Gaussian distribution N (0 , 1) , and obtain the outputs by y = c 1 x 1 x 2 + c 2 x 21 .
Revised: The datasets are summarized as follows: 1) Synthetic dataset is generated from quadratic functions with adding random noises.

Original:  Power system  datasets . IEEE provides standard power  systemmodels, including the grid topology, parameters, and generation models, etc., for simulations. The model ﬁles and the simulation platform, MATPOWER (MATPOWER community, 2020), are based on MATLAB. For simulations, the load ﬁles are need as the inputs to the systems.
Revised: 2) Power system datasets: The full training datasets of nodal voltage states and power injections are generated from power ﬂow simulations using real feeder models (IEEE 8-bus system and Utility feeder), one-year power consumption proﬁles, and distributed photovoltaics generation proﬁles (Yu et al., 2017).

Original: Thus, we introduce power consumption data from

Original: PJM Interconnection LLC (PJM Interconnection LLC, 2018).

Original: Such load ﬁles contain hourly power consumption in 2017 for the PJM RTO regions.

Original: With the above data, MATPOWER produces the system states of voltages and nodal power injections.

Original: Then, we utilize the voltage as input and the nodal power as the output to learn the physical mapping, for which the true physical function is used for evaluations (Yu et al., 2017).

Original: To diversify the system types, we utilize the mass-damper system and aerodynamic system to further validate our results. Motion dynamics dataset: mass dampers of in-plane wind turbine. The physical equation of the mass-damper system is representative of aircraft motion for analysis of system dynamics (van der Schaft, 2017; McRuer et al., 2014).
Revised: Motion dynamics dataset: We simulate the dynamic process of the mass-dampers systems withnodes and obtain the measurements of velocity and momentum for the test of physical equation discovery.

Original: Moreover, the active tuned mass dampers are implemented to control in-plane blade vibrations (Fitzgerald et al., 2013).

Original: Using MATLAB, we simulate the dynamic process of the mass-damper system with 10 nodes and obtain the measurements of velocity and momentum for the test of physical equation discovery.

Original: Aerodynamic dataset:aircraft weight estimation. The aircraft sensor data  is collected from the aircraft simulation in MATLAB and thecorresponding aerodynamic modelis illustrated in (Chen et al., 2018).
Revised: Aerodynamic dataset: The aircraft sensor data for weight estimation is collected from the aircraft simulation based on the aerodynamic model in (Chen et al., 2018).

Original: In this case, the goal is to recover the aerodynamic model of gross weight and angle of attack in a cambered airfoil.

Original: 4. P ERFORMANCE EVALUATION METRIC
Revised: Performance Evaluation Metrics.

Original: Prediction error : One can evaluate the modeling accuracy and extrapolation capacity of the PNN via calculating the mean square error (MSE) of the output in the testing dataset. Speciﬁcally, we deﬁne ˜ Y ∈ R N × d 2 and Y ∈ R N ×
Revised: The following metrics based on (Blasch & Sung, 2021) are used to evaluate the performance of learning physical system governing equations.

Original: d 2 as the model predicted data matrix and the true data matrix, where N is  the number of [-samples. MSE can be calculated MSE ( ˜ Y , Y ) = || ˜ Y − Y || 
-]
Revised: Prediction error is to evaluate the modeling accuracy and extrapolation capacity of [-samples. MSE can be calculated MSE ( ˜ Y , Y ) = || ˜ Y − Y || 
-]learning model via mean square error (MSE) of the output prediction.

Original: N × d 2 . Later, we plan to utilize other metrics to evaluate the performance of our model (Blasch & Sung, 2021). Physical exactness (PhyE) : To evaluate how much physics PNN extracts, we propose a metric PhyE to compare the extracted values and the  true values.
Revised: Physical exactness (PhyE) is to evaluate how much physics is extracted by comparing the recovered symbolic function and related parameters with true physical equation.

Original: Robustness to noise ratio (RNR) : We propose RNR to average all results  under different signal-to-noise ratios.
Revised: Robustness to noise ratio (RNR) is proposed to average all results tested under different signal-to-noise ratios for practical consideration.

Original: Training time and testing time : We utilize the training and testing time to evaluate the model efﬁciency.
Revised: More details are provided in Appendix.

Original: 4. B ENCHMARK METHODS AND TESTING SCENARIOS
Revised: Benchmark Methods and Testing Scenarios.

Original: The following benchmark methods are used for comparison: (1)  support vector regression (SVR) (Yu et al., 2017), (2) deep residual network (Resnet) (He et al., 2016),(3) sparse identiﬁcation of nonlinear dynamics (SINDy) (Brunton et al., 2016; Champion et al., 2019), and  equation learner (EQL ÷ ) (Sahoo et al., 2018).
Revised: The following benchmark methods are used for comparison: (1) sparse identiﬁcation of nonlinear dynamics (SINDy) (Brunton et al., 2016), (2) support vector regression (SVR) (Yu et al., 2017), (3) deep residual network (Resnet) (He et al., 2016), and (4) equation learner (EQL ÷ ) (Sahoo et al., 2018).

Original: Different scenarios are generated for testing: S 1 denotes the regular setup of dataset size and noise level (independent Gaussian noise with zero mean and 0.01 standard deviation); S 2 denotes a noisy setup with noises of 0.1 standard deviation; S 3 denotes a data-limited setup with 10% samples of original dataset.
Revised: Different scenarios are generated for testing: S 1 denotes the regular setup of dataset size and noise level (independent Gaussian noise with zero mean and 0.01 standard deviation); S 2 denotes a noisy setup with noises of 0.1 standard deviation; S 3 denotes a data-limited setup with 10% samples of original dataset available.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Fig.

Original: Although other methods can have low errors in some testing cases, they can not maintain the good performance as PNN-RISE does for all the datasets.
Revised: Although other methods can have low errors in some testing cases, they can not maintain the good performance as PNN-RISE does for all the datasets.

Original: Especially, EQL ÷ and SINDy’s performance deteriorates when the problem size becomes larger, e.g., power system datasets with data from many nodes.
Revised: Especially, EQL ÷ and SINDy’s performance deteriorates when the problem size becomes larger, e.g., power system datasets with data from many nodes.

Original: While SVR is robust against noises, it can not recover the nonlinear physical function exactly to further decrease the learning error.
Revised: While SVR is robust against noises, it can not recover the nonlinear physical function exactly to further decrease the learning error.

Original: In this case, the proposed method shows low learning error and a stable performance in both noisy and data-limited scenarios.
Revised: In this case, the proposed method shows low learning error and a stable performance in both noisy and data-limited scenarios.

Original: Moreover, the performance ofphysical function recovery is compared in Fig. 7(left).
Revised: Moreover, the performance of function recovery is compared in Fig. 5 (left).

Original: The bar plot shows the recovery rates (percentage of runs that correctly recover the ground truth symbolic expression) and the line plot with shades shows the recovery accuracy of the corresponding physical parameters.
Revised: The bar plot shows the recovery rates (percentage of runs that correctly recover the ground truth symbolic expression) and the line plot with shades shows the recovery accuracy of the corresponding physical parameters.

Original: While EQL ÷ easily fails in large power system cases,the proposed PNN-RISE has high success rate in true function recovery, merely trapped into local optima.
Revised: While EQL ÷ easily fails in large power system cases, PNN-RISE has high success rate in true function recovery, merely trapped into local optima.

Original: Forthe recovery of physical parameters, PNN-RISE not only has higher average accuracy but also has lower standard deviation than EQL ÷ .[Table]
Revised: For physical parameter recovery, PNN-RISE not only has higher average accuracy but also has lower standard deviation than EQL ÷ .

Original: 4. A BLATION STUDY FOR PHYSICAL CONSTRAINTS
Revised: 4. A BLATION STUDY FOR PHYSICAL CONSTRAINTS

Original: In order to show the performance of using different physical constraints, we provide an ablation study of the proposed NN.
Revised: In order to show the performance of using different physical constraints, we provide an ablation study of the proposed NN.

Original: First, we use the synthetic example to visualize the effect of constraints. The left-hand side of Fig.
Revised: We ﬁrst use the synthetic example to visualize the effect of constraints and show in Appendix.

Original: 6 compares the local optimum (transparent gray surface) learned by NN with the true function (solid gray surface).

Original: Though there are some overlaps, the large gap between the two surfaces causes large errors for generalization.

Original: By adding constraints, we observe in the middle of Fig. 6 that the variable coupling constraint rotates and stretches the surface to the correct shape like the maximum and minimum values of the surface. On the other hand, the parameter range constraint smoothes the surface of the local optimum. Together, the constraints push the recovered function (cyan surface) towards the ground truth, where the overlap in Fig.
Revised: Fig.

Original: 6 (right) shows the physical exactness in not only feature components but also the corresponding parameters.

Original: Subsequently, Fig.

Original: 7 (right) presents the learning results on power system datasets (IEEE 8-bus system).
Revised: 5 (right) presents the learning results on power system datasets (IEEE 8-bus system).

Original: The performance is evaluated in different test scenarios: 1) regular data acquisition, 2) limited input range (voltages within [0 . 98 , 1 . 01] for trainingdata and [0 . 90 , 1 . 10] for testing), 3) out-of-scale invariants in a governing function, and 4) limited data amount.
Revised: The performance is evaluated in different test scenarios: 1) regular data acquisition, 2) limited input range (voltages within [0 . 98 , 1 . 01] for training and [0 . 90 , 1 . 10] for testing), 3) out- of-scale invariants in a governing function, and 4) limited data amount.

Original: In general, adding physical constraints boosts the performance in different testing scenarios.
Revised: Generally, adding physical constraints boosts the performance in different testing scenarios.

Original: Especially, the constraint on mutual variables signiﬁcantly reduces the learning error by pushing the features and parameters to be symmetric for multiple-output problems.
Revised: The constraint on mutual variables signiﬁcantly reduces the learning error by pushing the features and parameters to be symmetric for multiple-output problems.

Original: When the data amount is very limited to train NN efﬁciently, the bound on the inertia of system outputs in time sequence can improve the function recovery.
Revised: When the data amount is very limited to train NN efﬁciently, the bound on the inertia of system outputs in time sequence improves the function recovery.

Original: 4. I LLUSTRATION OF THE EFFECTIVENESS OF FORECASTING - BASED PHYSICS LEARNING
Revised: 4. I LLUSTRATION OF THE EFFECTIVENESS OF FORECASTING - BASED PHYSICS LEARNING

Original:  To illustrate the effectiveness ofour model boosting algorithm enhanced by GP forecasting, we utilize 200 samples for training and propose to forecast { 500 , 1000 , 1500 , 2000 } samples out of the training dataset.
Revised: [Figure] To illustrate the effectiveness of GP forecasting, we utilize 200 samples for training and propose to forecast { 500 , 1000 , 1500 , 2000 } samples out of the training dataset.

Original: Also, we consider different SNRs in { 50 , 100 , 150 } .
Revised: Also, we consider different SNRs in { 50 , 100 , 150 } .

Original: Then, the forecast samples are integrated intoour PNN model and other benchmark models forresult comparisons.
Revised: The forecast samples are integrated into PNN model and other benchmark models for comparisons.

Original: Fig.demonstrates the result, i.e., MSE, with respect to the forecast points.
Revised: Fig. 6 demonstrates the results with respect to the forecast points.

Original: Due to the space limit, we only show results of GP-based PNN and GP-based Resnet for comparisons.
Revised: Due to the space limit, we only show results of GP-based PNN and GP-based Resnet for comparisons.

Original: The patterns of results for other benchmark methods are similar to GP-based Resent.
Revised: The patterns of results for other benchmark methods are similar to GP-based Resent.

Original: We observe thatﬁrstly, GP-based PNN performs much better than GP-based Resnet, which is consistent to observations in the previous subsections.
Revised: We observe that GP-based PNN performs much better than GP-based Resnet, which is consistent to observations in the previous subsections.

Original: Secondly, as the number of forecast data points increases, the prediction MSE value generally decreases for both PNN model and Resnet model.
Revised: As the number of forecast data points increases, the prediction MSE value generally decreases for both PNN model and Resnet model.

Original: This implies that our GP forecasting can produce accurate forecast points, suitable for all prediction models.
Revised: It implies that our GP forecasting produces accurate forecast points, suitable for all prediction models.

Original: Subsequently, when the number is larger than 500 , the results don’t show a signiﬁcant improvement.
Revised: When the number is larger than 500 , the results don’t show a signiﬁcant improvement.

Original: The reason is that forecast points far from the training set boundary are with low conﬁdence. Thus, they don’t have a big impact on learning results.
Revised: The reason is that forecast points far from the training set boundary are with low conﬁdence, which don’t have a big impact on learning results.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The tremendous expansion of internet of everything (IoE) brings intelligent connections among different devices to coordinate resources.

Original: However, the lack of physical understanding makes it hard for planning, monitoring, and control of IoE.To learn the governing physical equations from data, [Figure]
Revised: The lack of physical understanding makes it difﬁcult for planning, monitoring, and control of IoE.

Original:  the paper ﬁrst presents a fundamental challenge for data-driven methods like symbolic regression and sparse regression.
Revised: To learn governing physical equations from data, the paper ﬁrst presents a fundamental challenge for data-driven methods like symbolic regression and sparse regression.

Original: Namely, data-driven only methods cannot guarantee to ﬁnd the exact physics, as there are many low-complexity expressions to perfectly ﬁt the data.
Revised: Data-driven only methods cannot guarantee to ﬁnd the exact physics because many low-complexity expressions can perfectly ﬁt the data.

Original: To resolve the issue of ﬁnding the underlying physical laws, we propose a complete set ofprinciples to deﬁne physical constraints  to restrictthe search space.
Revised: To resolve the issue, we propose a complete set of constraints based on physical principles to restrict search space.

Original: Under these constraints, an efﬁcient physical neural network (PNNRISE) is proposed to achieve constrained neural network learning.
Revised: Thus, an efﬁcient physical neural network (PNN-RISE) is developed.

Original: Finally, the FOCV supports the scenario when (partial) physical constraints are unknown.
Revised: Finally, the FOCV supports the scenario when (partial) physical constraints are unknown.

Original: As physical constraints aim to maintain accurate performance out of samples, the forecast of data beyond the training dataset enhances model robustnessas an intelligent integration signiﬁcantly improves the model with few extra costs. 
Revised: The forecast of data beyond the training dataset enhances model robustness with few extra costs. Results on physical system datasets show the superior performance of proposed methods. B P ROOF OF T HEOREM Proof.

Original: Results on four data sets demonstrate the superior performance of the FOCV methods.
Revised: First, we deﬁne the biconvexity as follows (Gorski et al., 2007). Deﬁnition 1 (Biconvexity) . A function φ : A × B → R is called biconvex if f ( a, b ) is convex in a for ﬁxed b ∈ B , and f ( a, b ) is convex in b for ﬁxed a ∈ A .

Original: A A PPENDIX
Revised: Then, we prove PNN is biconvex. Recall for the PNN function y = f ( x ) , the k th element can be written as: y k = (cid:80) j m jk z j = (cid:80) j m jk exp (cid:0) (cid:80) Ni =1 p ij ln

Revised: ( x i ) (cid:1) .

Revised: ∀ j, k , if m jk is known, we need to prove the convexity of f with respect to p ij , ∀ 1 ≤ i ≤ N .

Revised: Since the linear combination (cid:80) Ni =1 p ij ln

Revised: ( x i ) is convex with respect to p ij , and the exponential function is convex, the convex function composition implies

Revised: that exp (cid:0) (cid:80) Ni =1 p ij ln

Revised: ( x i ) (cid:1) is convex with respect to p ij .

Revised: Therefore, ∀ j, k , if m jk is known, the function y k

Revised: = (cid:80)

Revised: j m jk exp (cid:0) (cid:80) Ni =1 p ij ln

Revised: ( x i ) (cid:1) is convex with respect to p ij , ∀ 1 ≤ i ≤ N .

Original: You may include other additional sections here
Revised: On the other hand, ∀ i, j , if p ij is known, y k = (cid:80) j m jk exp (cid:0) (cid:80) Ni =1 p ij ln ( x i ) (cid:1) is a linear combination of m jk .

Revised: Therefore, y k is convex with respect to m jk given p ij .

Revised: Therefore, PNN function y = f

Revised: ( x ) is biconvex with respect to the logarithm layer and the linear summation layer.

Revised: The biconvexity can lead to an efﬁcient alternate convex search (ACS)

Revised: (Gorski et al., 2007) to seek for optimal solutions.

Revised: Namely, ACS will ﬁx one part of parameters to search for the other part iteratively.

Revised: Due to biconvexity, in each iteration, ACS will solve a convex problem.

Revised: Then, Corollary 4.10 in (Gorski et al., 2007) guarantees that the searched points will converge to stationary points of f ( x ) .

Revised: C D ETAILS OF D ATASETS AND E XPERIMENT S ETUPS

Revised: Synthetic dataset: generating random samples of x from Gaussian distribution N (0 , 1) , and obtain the outputs by y =

Revised: c 1 x 1 x 2 + c 2 x 21 .

Revised: Power system datasets: IEEE provides standard power system models, including the grid topology, parameters, and generation models, etc., for simulations.

Revised: The model ﬁles and the simulation platform, MATPOWER (MATPOWER, 2020), are based on MATLAB.

Revised: For simulations, the load ﬁles are need as the inputs to the systems.

Revised: Thus, we introduce power consumption data from PJM

Revised: Interconnection LLC (PJM Interconnection LLC, 2018).

Revised: Such load ﬁles contain hourly power consumption in 2017 for the PJM RTO regions.

Revised: With the above data, MATPOWER produces the system states of voltages and nodal power injections.

Revised: Then, we utilize the voltage as input and the nodal power as the output to learn the physical mapping, for which the true physical function is used for evaluations (Yu et al., 2017).

Revised: To diversify the system types, we utilize the mass-damper system and aerodynamic system to further validate our results.

Revised: Motion dynamics dataset: mass dampers of in-plane wind turbine.

Revised: The physical equation of the mass-damper system is representative of aircraft motion for analysis of system dynamics (van der

Revised: Schaft, 2017; McRuer et al., 2014).

Revised: Moreover, the active tuned mass dampers are implemented to control in-plane blade vibrations (Fitzgerald et al., 2013).

Revised: Using MATLAB, we simulate the dynamic process of the mass-damper system with 10 nodes and obtain the measurements of velocity and momentum for the test of physical equation discovery.

Revised: Aerodynamic dataset: aircraft weight estimation: The aircraft sensor data is collected from the aircraft simulation in MATLAB and the corresponding aerodynamic model is illustrated in (Chen et al., 2018).

Revised: In this case, the goal is to recover the aerodynamic model of gross weight and angle of attack in a cambered airfoil.

Revised: D R ESULTS OF THE S YNTHETIC D ATASETS

Revised: In order to show the performance of using different physical constraints, we provide an ablation study of the proposed NN.

Revised: We use the synthetic example to visualize the effect of constraints.

Revised: The left-hand side of Fig.

Revised: 9 compares the local optimum (transparent gray surface) learned by NN with the true function (solid gray surface).

Revised: Though there are some overlaps, the large gap between the two surfaces causes large errors for generalization.

Revised: By adding constraints, we observe in the middle of Fig.

Revised: 9 that the variable coupling constraint rotates and stretches the surface to the correct shape like the maximum and minimum values of the surface.

Revised: On the other hand, the parameter range constraint smoothes the surface of the local optimum.

Revised: Together, the constraints push the recovered function (cyan surface) towards the ground truth, where the overlap in Fig.

Revised: 9 (right) shows the physical exactness in not only feature components but also the corresponding parameters.