A DVERSARIAL TWIN NEURAL NETWORKS : MAXIMIZING PHYSICS RECOVERY FOR PHYSICAL SYSTEM
Anonymous authors
Paper under double-blind review
A BSTRACT
The exact modeling of modern physical systems is challenging due to the expanding system territory and insufﬁcient sensors.
To tackle this problem, existing methods utilize sparse regression to ﬁnd physical parameters or add another virtual learning model like a Neural Network (NN) to universally approximate the unobserved physical quantities.
However, the two models can’t perfectly play their own roles in joint learning without proper restrictions.
Thus, we propose (1) sparsity regularization for the physical model and (2) physical superiority over the virtual model.
They together deﬁne output boundaries for the physical and virtual models.
Further, even the two models output properly, the joint model still can’t guarantee learning maximal physical knowledge.
For example, if the data of an observed node can linearly represent those of an unobserved node, these two nodes can be aggregated.
Therefore, we propose (3) to seek the dissimilarity of physical and virtual outputs to obtain maximal physics.
To achieve goals (1) − (3) , we design a twin structure of the Physical Neural Network (PNN) and
Virtual Neural Network (VNN), where sparse regularization and skip-connections are utilized to guarantee (1) and (2) .
Then, we propose an adversarial learning scheme to maximize output dissimilarity, achieving (3) .
We denote the model as the Adversarial Twin Neural Network (ATN).
Finally, we conduct extensive experiments over various systems to demonstrate the best performance of ATN over other state-of-the-art methods.
I NTRODUCTION
Internet of Everything (IoE) expands quickly to interconnect various devices.
The systematic planning, modeling, and control of IoE can bring many beneﬁts to society (Li et al., 2020).
However, it remains an open question on how to efﬁciently model various grids with different levels of system information.
For example, cyber-physical systems may be partially traceable by physical laws with limited sensing (Mulani et al., 2020).
Such scenarios have a signiﬁcant appearance nowadays on the grid edges of various physical systems due to the availability of low-cost, low-power sensor technology.
These edge areas have unequal sensors but can be used to recover some physical laws completely or partially (Divan et al., 2014;
Hu & Tang, 2020).
Learning physical equations from data is a central topic of Artiﬁcial Intelligence (Sahoo et al., 2018;
Udrescu & Tegmark, 2020).
While there are many related studies, e.g., symbolic regression (Petersen, 2019) and its variation with sparsity regularization (Brunton et al., 2016b), these methods are inapplicable to IoE or other complex physical systems due to the incomplete system observability.
To tackle the issue, (Li & Weng, 2021) builds a shallow-deep structure to learn physics in a shallow neural network and simultaneously approximate the hidden components in a deep NN.
Such a method, however, doesn’t give a clear boundary between two NNs and suffers risks of imbalanced representation power between the physical and the virtual.
Thus, there is a need to enforce restrictions for the model output within the boundary and keep the physical consistency.
Further, existing methods can hardly guarantee that the extracted physical model is operationally optimal.
For example, in the realistic operation, if the data of an observed node can linearly represent those of an unobserved node, these two nodes can be aggregated, leading to the so-called Network
Reduction (NR) (Oh, 2012; Zheng et al., 2021).
On the other hand, some components of hidden quantities that can’t be represented via the observed data can be treated as noise.
Thus, our target is to identify an optimal reduced grid with operationally maximal physics.
Namely, the reduced grid should fully represent the input-output relationship of the observed measurements under a certain noise level.
Based on the above observations, we propose Adversarial Twin NNs (ATN) for optimal system modeling, where a Physical Neural Network (PNN) represents the physical parameters of the reduced grid and a Virtual Neural Network (VNN) approximates the noise.
To ﬁnd the proper output boundary, we ﬁrst restrict the PNN using sparse regularization.
Simultaneously, we encourage PNN to approximate the ﬁnal output with priority, indirectly restricting the output of VNN.
We show that such a mechanism can be easily achieved via a skip-connection (He et al., 2016b).
Secondly, to achieve physical knowledge maximization, the output of PNN should be independent of the noise output of the VNN.
Thus, we propose an adversarial learning scheme with Embedding Neural Networks (ENNs) to extract similar features from outputs of PNN and VNN.
Adversarially, training
PNN and VNN leads to maximized output dissimilarity.
Notably, such an idea is the reverse thinking of traditional deep learning-based adversarial learning that seeks maximal similarity between two targets (e.g., feature distributions).
Namely, we aim to ﬁnd the maximized feature dissimilarity of two NNs.
Under this condition, classical training loss like binary loss in Generative Adversarial Networks (GANs) easily suffers instability.
Thus, we address the problem using the similarity and dissimilarity measures of contrastive learning (Hadsell et al., 2006).
Finally, we conduct extensive experiments over various systems to demonstrate the much better performance of our model compared to other state-of-the-art methods.
R ELATED W ORK
Physical System Identiﬁcation .
Physical system identiﬁcation is a central topic for modern physical systems, especially in the edge areas.
The target is to identify system governing equations using sensor measurements (Li & Weng, 2021).
While the problem is similar to learning underlying equations from data, the key challenge for physical system identiﬁcation is the incomplete measurement availability due to the sensor cost in a wide area of the system.
Thus, traditional methods of learning equations using symbolic regression (Schmidt & Lipson, 2009; Petersen, 2019) can easily suffer overﬁtting.
Then, (Brunton et al., 2016c; Champion et al., 2019) assume that the system is sparse and utilize Least Absolute Shrinkage and Selection Operator (LASSO) regularization to restrict physical equation representation and avoid overﬁtting.
This regularization, however, causes inaccuracy to learn physical parameters due to the penalty term.
Thus, (Li & Weng, 2021) proposes a deep-shallow architecture to learn physics with a shallow neural network as well as approximating the hidden components with a deep neural network.
However, the DNN model easily suffers from local optima, making the output boundary between the shallow neural network and the DNN deviate from the true boundary.
Consequently, the recovery accuracy of the physical parameters in the shallow neural network deteriorates.
To restrict the output boundary, (Yin et al., 2020) also assumes physical and augmented neural networks and propose a l 2 norm to regularize the parameters of the augmented NN.
However, such regularization may easily cause overﬁtting of the physical NN.
Finally, (Takeishi & Kalousis, 2021) can restricts both the physical and the virtual NNs. However, their regularization requires strict assumptions on variable distributions and the prior physical knowledge, which may not be available for general physical systems.
Adversarial Learning .
Adversarial learning is a popular approach for training two adversarial components in a NN.
Primarily, the mini-max game training helps to achieve the Nash equilibrium for optimal policy.
For example, Generative Adversarial Network (GAN) (Goodfellow et al., 2014;
Sauder & Sievers, 2019) utilizes a generator to generate fake data and, adversarially, utilize a discriminator to distinguish between the fake data and the true data.
The optimal status is to enable the generator to accurately approximate the distribution of the true data.
Such an idea is further utilized in various machine learning domains like domain adaptation (Ganin et al., 2016) to extract similar features between two domains and disentangled representation learning (Tran et al., 2017).
Contrastive Learning .
Contrastive learning seeks representation with a minimal distance of similar samples and maximal distance of dissimilar samples (Hadsell et al., 2006).
Usually, the distance is measured in an embedding space to seek the best embedding.
The elaborated contrastive loss can thus guarantee stable training due to the soft comparisons between the positive and the negative samples.
Such a technique is widely utilized in image embedding (Park et al., 2020), feature clustering (Li et al., 2021c), text recognition (Aberdam et al., 2021), etc.
M ETHODS 3. P ROBLEM FORMULATION
In this paper. we study physical systems that can be modeled as a directed weighted graph G = {V , E} , where V represents the vertex (node) set, and E ⊆
V × V represents the edge set.
All nodes in V have physical variables x and y , where x represents the system state variables, and y represents the system net outputs.
Then, the system equations can be written as y = f ( x ) .
For example, in electric grids, x denotes the voltage phasor, and y denotes the net power (i.e., the power consumption minus the power injection).
Then, f can represent the power ﬂow equation (Yu et al., 2017).
Parameters in f are usually (partially) unknown to us due to the growing system size, evolving system properties, events, and maintenance, etc.
Thus, it’s essential to estimate parameters of f using data of x and y .
However, not all nodes in V can be equipped with sensors due to the sensor cost.
Thus, we denote
V =
O ∪ U , where O represents the observable nodes, and U represents the unobservable nodes.
The physical function can then be written as [ y O , y U ] (cid:62) =
f (cid:0)
[ x O , x U ] (cid:62) (cid:1) , where (cid:62) is the transpose operation.
Subsequently, we denote the observed data samples as { x n O } Nn =1 and { y n O } Nn =1 , where N is the number of samples.
Based on the above measurements, we aim to ﬁnd an accurate mapping g θ such that y O
≈
g
θ
( x O ) .
Further, a subset of the parameters θ p ⊂ θ should represent the physical parameters in f as many as possible.
The missing quantities of x U and y U make the problem challenging as the under-estimate or overestimate of x U and y U will easily cause the inaccurate results of θ p and further hurt the generalization ability of g θ .
Subsequently, this causes negative impacts on the downstream tasks like system resource optimization, reliability evaluation, and optimal expansion.
To solve the above problem, we propose twin NNs to approximate both of them.
As shown in Fig.
1, the twin NNs try to map from x O to y O with physics consistency, where Physical Neural Network (PNN) learns physical parameters and outputs physical quantities and Virtual Neural Network (VNN) approximates the remaining quantities, i.e., the uncertainties and hidden measurements.
Thus, they jointly contribute to the ﬁnal output y O .
Before illustrating the design details, we ﬁrst introduce a moderate assumption for the system prior.
Speciﬁcally, we assume the physical bases z are known in the physical equation.
Namely, there is a mapping z = φ ( x ) such that y
=
A z , where A is a constant matrix representing system parameters.
Then, all the system non-linearity is incorporated in φ and A contains all physical information and needs to be estimated.
If we know all nodes’ measurements, a linear regression can help to identify A .
However, we only have measurements of x O and y O .
Correspondingly, we denote z =
[ z O , z U ] (cid:62) , where z O are physical bases purely calculated via x O
and z U are the bases calculated via x U or [ x O , x U ] (cid:62) .
Thus, we ﬁx the parameters from x O to z O and place the learnable parameters in the mapping from z O to y O .
Namely, we build a physical library, as shown in the left part of Fig.
Note that for some complex systems, the mapping φ may not be known, and we may demand symbolic regression-based methods to generate the base symbols ﬁrst (Petersen, 2019).
However, in this paper, we focus on the issue of incomplete system observability with the prior knowledge of φ , e.g., the quadratic and sinusoidal functions in an electric grid.
3. P HYSICAL NEURAL NETWORK ( PNN ): LEARN PHYSICS WITH SPARSE REGULARIZATION
Due to the linear relationship between z O and y O , we introduce a linear mapping g p such that h 1 = g p
( z O )
=
W 1 z O in PNN.
Usually, the system structure is sparse (Brunton et al., 2016c;
Champion et al., 2019).
To maintain the physical consistency, we enforce a sparse regularization, i.e., the LASSO term, to guarantee sparse connections of the system.
Speciﬁcally, the LASSO regularization is {+created as || W 1 || 
 =+}
(cid:80) i,j | W 1 [ i, j ] | , where W 1 [ i, j ] is the ( i, j ) th element in W 1 .
3. V IRTUAL NEURAL NETWORK ( VNN ): UNIVERSAL APPROXIMATION UNDER PHYSICAL SUPERIORITY
After showing the regularized PNN, this section will explain the modeling and beneﬁts of VNN.
The goal for VNN is to (1) approximate the remaining quantities, and (2) avoid deteriorating the output of PNN.
Namely, we want to ﬁnd a boundary to let PNN and VNN output properly.
For the ﬁrst goal, we build VNN as a fully connected neural network with multiple layers, which has a high capacity to approximate the remaining quantities.
Speciﬁcally, let the mapping of the VNN be g v , and we have h M +
= g v ( h 1 ) , where M is the number of layers for VNN.
For the second goal of ﬁnding the boundary, we need to restrict the output of VNN and encourage
PNN to output as much as possible.
On the other hand, we do not worry about the overﬁtting of the PNN as PNN has been strictly regularized via LASSO regularization to guarantee physical consistency.
Thus, we employ a skip-connection from the output of PNN directly to the ﬁnal output (He et al., 2016a).
Mathematically, we have: {+y O = h 
+}
+ g v ( h 1 ) = g p ( z O ) + g v (cid:0) g p ( z O ) (cid:1) .
The skip-connection further ensures the possibility to create physical superiority, i.e., to encourage
PNN to output as much as possible.
Namely, we encourage g p ( z O ) to directly approximate y O .
In general, we have the following objective for the twin NNs.
[Equation]
where γ and λ are positive constants for the penalty terms.
3. M INI - MAX GAME TO ADVERSARIALLY TRAIN TWIN NNS FOR MAXIMIZED PHYSICAL KNOWLEDGE
In this subsection, we propose the Adversarial Twin NNs (ATN) based on the above twin NNs.
Primarily, we show that ATN helps recover maximized physics, which is much better than previous work for physical system modeling.
As we mentioned in the Introduction Section, even if the boundary between the observed and unobserved measurements is successfully found, the physics may not be maximized.
Essentially, we aim to ﬁnd a reduced grid to govern the model and operation of observed measurements.
Mathematically, we can treat the hidden components as z U =
B z
O + e , where B is another constant matrix for the linear representation of the unobservable parts using the observable parts, and e is the noise term.
Based on the above formulation, the maximized physics can only be achieved when g p ( z O ) represents the linear relationships from both z O and z U
≈ B
z O , and g v ( z O ) represents the output transformed from the noise term e .
This inspires a clear probabilistic distance between g p ( z O ) and g v ( z O ) .
Speciﬁcally, we aim to maximize the distribution dissimilarity between g p ( z O ) and g v ( z O ) , thus achieving maximized physics.
As distribution dissimilarity can hardly be explicitly written in a neural network framework, we propose an adversarial learning framework to achieve the goal.
Speciﬁcally, we intend to drive the training process to gradually convert physical knowledge from
VNN to PNN, which needs to measure and transfer the common knowledge.
Thus, we utilize a third neural network, Embedding Neural Network (ENN), to transform the output of PNN and VNN into embedding vectors with large common knowledge, i.e., similarity of embedding distributions.
On the other hand, our PNN and VNN should dynamically adjust the parameters to achieve minimal similarity.
In general, the complete framework is shown in Fig.
1, where two ENNs convert the original output of PNN and VNN for the similarity loss in a mini-max game.
Since we have the given output pair from PNN and VNN, the task of ﬁnding the similarity of two embeddings can be formalized as a contrastive learning framework (Hadsell et al., 2006).
Thus, we deﬁne the similarity distance as follows:
[Equation]
where g e 1 and g e 2 represent the mappings of two ENNs, respectively.
Namely, we utilize the Euclidean distance in the embedding space to measure the distribution similarity.
Based on Equation (2), we subsequently deﬁne the similarity loss for training as follows:
[Equation]
where Θ e represent the set of parameters in g e 1 and g e 2 , and Θ v and Θ p represent the parameter sets for PNN and VNN, respectively.
Y is a scalar to indicate if the sample pair is similar ( Y = 0 ) or dissimilar ( Y = 1 ), and m is a threshold to achieve stability in contrastive learning.
With the loss function above, we derive the the value function for the mini-max game between PNN and VNN:
[Equation]
During the training, we denote Y = 0 for the maximization process as we want to train PNN and
VNN to maximize the distance in Equation (2).
Conversely, we denote Y = 1 in the minimization process to minimize the deﬁned distance.
The adversarial training process ﬁnally leads to the maximized dissimilarity between outputs of PNN and VNN, which implies that the physical knowledge is maximally represented in PNN.
Since we do the adversarial training, we denote our model as
Adversarial Twin NNs (ATN).
E XPERIMENTS
To extensively verify the performances of ATN, we elaborate our models on different datasets from different physical systems.
Meanwhile, we consider testing under different system observability levels, i.e., the ratio of the number of observed nodes to the total nodes, in the set { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } and different Signal-to-Noise-Ratios (SNRs) in the set { 60 , 80 , 100 , 120 } to mimic the realistic conditions.
Then, we employ 8000 samples and conduct 5 -fold cross-validation to obtain the average results for demonstration.
For all the NN training, we use Adam optimizer with a learning rate hyperparameter set { 0 .
001 , 0 . 0002 , 0 . 00005 } , and momentum parameters β 1 = 0 .
5 , β 2 = 0 .
999 to train 200 epochs for each experiment.
All the experiments are completed with a computer equipped with
Intel(R) Core(TM) i7-9700k CPU and Nvidia Geforce RTX 2080Ti GPU.
The detailed experimental settings are described as follows.
4. D ATASETS
Synthetic Data .
We ﬁrst utilize synthetic datasets for demonstration.
Speciﬁcally, we propose a 4 -node system such that:
[Equation]
The vector variable x is simulated using uniform distribution U (0 , 1) , and we construct the set z based on the physical mappings (e.g., square or cosinusoidal.) in Equation (5).
IEEE Power Systems and PJM Load Data .
There are standard power system models in IEEE, including the grid nodes, edges, and attributes.
Especially, we can download the model ﬁles and the simulation platform in MATPOWER (MATPOWER community, 2020).
In this experiment, we incorporate IEEE 85 -node systems for testing.
To conduct the simulation, we further employ realworld power consumptions in PJM Interconnection LLC (PJM) data (PJM Interconnection LLC, 2018).
The load ﬁles contain hourly electricity consumption in 2017 for the PJM RTO regions.
With the above data, MATPOWER produces the system states of voltage phasor to represent the system input x .
In the meantime, the nodal net power represents the system output y for training.
In the power system dataset, we have φ =
[ x 2 , sin x ] .
Mass-damper system data .
We can utilize the following equation to represent the governing law of the mass-damper system: ˙
q = − DRD (cid:62) M − 1 q , where q is the vector of momenta of the masses,
D is the incidence matrix of the graph, R is the diagonal matrix of the damping coefﬁcients of the damper attached to the edges, and M is the diagonal mass matrix (van der Schaft, 2017).
Using
MATLAB, we simulate the dynamic process of the mass-damper system with 10 nodes for testing.
In the mass damper system dataset, we have φ =
[ dxdt , x ] .
UF sparse matrix-based system .
We can ﬁnd a large set of sparse matrix-based networks in the UF sparse matrix systems (Texas A&M University, 2011).
In this experiment, we utilize the 274 -node system to test.
For the convenience of later presentation, we simply utilize 4 -, 10 -, 85 -, and 274 -node systems to denote the above datasets.
In the UF sparse matrix dataset, we have φ =
[ x ] .
4. B ENCHMARK M ODELS
To validate the performance of learning physical system representation, the following benchmark methods from related literature are used.
• Deep Residual network ( Resnet ).
Similar to ATN, Resnet (He et al., 2016b) also utilizes the shortcut design to pass the representation of shallow layers to deeper layers.
Such a design helps Resnet to achieve excellent performances over various machine learning tasks (Guo & Du, 2019; Ma et al., 2020).
• Sparse Identiﬁcation of Nonlinear Dynamics ( SINDy ) (Brunton et al., 2016a).
SINDy method conducts a sparse regression to recover the physical parameters.
Especially, in our paper, we assume the physical bases are known.
Thus, the essence of SINDy is to utilize a LASSO regularization to select correct bases from the built-in library.
• Deep Symbolic Regression ( DSR ) (Petersen et al., 2021).
DSR employs a risk-seeking policy gradient to train a reinforcement learning agent for symbolic regression.
• Equation Learner ( EQL ÷ ) (Sahoo et al., 2018).
EQL ÷ treat the physical bases as activation functions in a neural network.
Also, the operators like summation, multiplication, and division are included in the neural network.
Finally, the target neural network includes sparsity constraints for symbol selections.
• Physics-Consistent Neural Network ( PCNN) (Li & Weng, 2021).
PCNN has a deepshallow architecture to represent the physical model, which is similar to our ATN.
However, PCNN doesn’t include a skip-connection for physical superiority, and mini-max game for training.
In this paper, the hyper-parameters of NNs (i.e., ATN, Resnet, DSR, EQL ÷ , and PCNN) are selected via cross-validation for the best performance.
4. M ODEL EVALUATION
We propose the following metrics to evaluate the generalizability and the parameter estimation results of different models.
Generalizability .
We utilize the Mean square error (MSE) of the testing set to evaluate the model performance of predicting y O .
Parameter estimation .
To evaluate the parameter estimation performance, we utilize the so-called normalized Total Vector Error ( nTV E ) (Li et al., 2021b) to evaluate the difference between the estimated ˆ L and the true physical system parameter matrix L :
[Equation]
[Figure]
Firstly, we evaluate the model operational performance via generalizability for different grids and under different SNR levels.
Speciﬁcally, we ﬁx the number of observability to be 0 .
6 and vary SNRS in the set { 60 , 80 , 100 , 120 } .
Fig.
2 demonstrates the testing MSE results for different physical systems.
Clearly, we ﬁrst observe that our ATN model (red line) achieves the lowest MSE for all the result points.
This implies that ATN model is the most suitable model for physical system real-time operations.
The following reasons help to explain the observation.
(1) ATN vs. Resnet .
In ATN model, we embed physical bases to learn a physical equation.
However, for Resnet, no physics is learned.
Due to the high generalizability of physical models in the observed regions, ATN performs much better than Resnet.
(2) ATN vs. SINDy .
ATN and SINDy have the ability to estimate the parameters of physical equations.
However, SNIDy only utilizes LASSO as the regularization term, but ATN utilizes LASSO in PNN and employs another VNN to approximate the remaining quantities.
Thus, SINDy typically suffers underﬁtting when there are unobserved values, which deteriorates the predicting performance.
(3) ATN vs. DSR & EQL ÷ .
DSR and EQL ÷ utilize symbolic regression to sparsely select symbols to form an equation that ﬁts data.
The missing values make the symbolic regression incapable of ﬁnding the true governing law.
However, ATN can ﬁnd governing equations for the reduced grid to best ﬁt the data.
(4) ATN vs. PCNN .
Though PCNN also enjoys a twin structure to use shallow NN to learn physics and deep NN to approximate the remaining components, there is no design of the adversarial learning.
Thus, training PCNN can’t ﬁnd the maximal physics of the reduced grid for optimal operations.
Then, we observe that for all methods, as the SNR increases, their MSE results decrease.
This shows that the lower noise level helps all methods to obtain better accuracy.
Interestingly, we ﬁnd that for the synthetic system, SINDy performs much better than DSR and EQL ÷ .
For the power system, SINDy has a moderately better performance.
Then, for the rest systems, SINDy, DSR, and
EQL ÷ have similar performances.
This is because the synthetic data has the most complex physical bases ( x, x 2 , x 3 , cos ( x )) , and power system has a relative lower complexity ( x 2 , sin ( x )) .
Then, the mass-damper system and UF sparse matrix have the lowest complexity ( x ).
Though we prepare all the physical base pools for each method, SINDy can successfully select the right base with LASSO regularization in the observed region.
However, DSR and EQL ÷ with a NN structure can easily select the incorrect bases and suffer from overﬁtting.
Finally, we ﬁnd that PCNN usually performs better than SINDy, DSR, and EQL ÷ .
This shows that even without a mini-max game, the twin structure can help to identify the rough boundary of the physical and virtual outputs and obtain a more accurate approximation than methods without twin structure.
4. P HYSICAL PARAMETER ESTIMATION : ATN HAS THE MOST ACCURATE RESULTS
This subsection studies the physical-parameter-recovery ability for SINDy, DSR, EQL÷, and ATN methods.
Especially, we ﬁx the observability level to be 0 .
6 and SNR = 100 .
To make sure the ground-truth is doable for comparison, we make the nodal inputs independent of each other for each system.
Thus, it avoids the case when the obtained reduced network is different from the ground-truth network, causing the impossibility for ATN method to verify results.
Table 1 demonstrates the results of nTV E (%)( mean ± standard deviation ) for different datasets.
First, the result shows that our proposed ATN reaches the best performance due to our sophisticated design of twin architecture and adversarial learning for dynamic boundary seeking.
Secondly, as system size increases, SINDy, DSR, and EQL ÷ have the quick boosting of the nTV E (%) , almost proportional to the system size growth.
However, for the ATN method, nTV E (%) remains lower with only slight growth.
This is because the increasing system size largely promotes the number of unobserved nodes, given ﬁxed observability issues.
Thus, for SINDy, DSR, and EQL ÷ , the probability of the perfect recovery is largely decreasing for each scalar function (i.e., a function to represent y O
[ i ] for the i th element in y O ).
Speciﬁcally, recovering y O
[ i ] needs all the input variables in the function of y O
[ i ] .
However, more unobservy errors for SINDy, DSR, and EQL ÷ increase largely.
As for PCNN method, the rate of error increasing is not as high as SINDy, DSR, and EQL ÷ methods, which shows that the deep neural network in the PCNN can restrict the overﬁtting of the physical learner with joint training.
However, for our ATN method, though the chance for a perfect recovery decreases, our VNN can help identify the boundary between the observed physics and the remaining components under the mini-max game mechanism.
Therefore, ATN can still have relatively small estimation errors for different systems.
[Table]
4. A BLATION STUDY FOR DIFFERENT COMPONENTS OF ATN
In this subsection
[Table]
we conduct the ablation study for tha ATN model to test the performance of different components.
Especially, we remove the sparsity of PNN, skip-connection of VNN
[Table]
and the minimax-training, respectively.
We denote the models as ATN-SP, ATN-SK, ATN-MI, respectively.
The results are shown in Table 2.
Averagely
[Table]
for the above three models, the nTV E (%) increases by 31 .
[Table]
and 20 . 21% .
This shows that the three components have great impacts on learning the correct physics.
Further, sparsity plays the essential role as it restricts the representation of the physics.
[Table]
4. S ENSITIVITY ANALYSIS CONCERNING THE SYSTEM OBSERVABILITY AND SAMPLE NUMBERS
In this subsection, we consider the sensitivity analysis with respect to the system observability and the sample numbers.
Speciﬁcally, for the 85 -node power system, we vary the system observability,
i
[Table]
e., the ratio of the number of observed nodes to the total nodes, in the set { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } .
Secondly, we vary the sample number in { 6000 , 8000 , 10000 , 12000 , 14000 } with observability ratio to be 0 .
Finally, we always ﬁx SNR = 100 for testing.
Table 3 illustrates the results with respect to different observability.
In general, as system observability increases, all methods have a better prediction accuracy, and ATN achieves the best performance for different scenarios.
Further, with system observability level increasing from 0 . 2 to 0 . 8 , MSEs of ATN, Resnet, SINDy, DSR, EQL ÷
[Table]
and PCNN have a relative decrease of 80 .
It shows that ATN can make the most use of the new information from newly placed meters and achieve the most relative reduction.
[Table]
Table 4 demonstrates the results with respect to different sample numbers.
We ﬁnd that for different methods, when the sample number is larger than 10000 , the MSE value will almost keep stable.
This shows that 10000 is a corner point for the physical model learning in the 85 -bus power system.
Considering the high-resolution (e.g., 30 samples per second) of the Phasor Measurement Unit (PMU) (Li et al., 2021a), 10000 is an acceptable sample number for realistic training.
[Table]
C ONCLUSION AND FUTURE WORK
To accurately model modern physical systems with incomplete system observability, we propose a twin structure of the physical neural network (PNN) and virtual neural network (VNN) to simultaneously learn physics and approximate the remaining components.
Different from traditional methods, our twin structure has critical designs to ﬁnd the proper output boundary between PNN and VNN: (1) we utilize sparse regularization to maintain physical consistency of PNN output, (2) we employ skip-connections to guarantee the output of the physical superiority, thus implicitly restricting the boundary of the VNN, and (3) we propose an adversarial learning scheme to adjust the ﬁne-grained boundary for maximized physical learning.
Speciﬁcally, for (3) , we introduce embedding neural networks (ENNs) to extract similar features from outputs of PNN and VNN.
Adversarially, training
PNN and VNN leads to maximized output dissimilarity, leading to maximized physics.
To guarantee the training stability, we employ the contrastive learning loss with excellent stability properties.
Finally, we conduct extensive experiments over various systems to demonstrate the best performance of our model compared to other methods.
For the future work, we will reﬁne the design of PNN to tackle more complex physical systems with unknown physical bases.