Physics-Augmented Learning: A New Paradigm Beyond Physics-Informed Learning
Abstract
Integrating physical inductive biases into machine learning can improve model generalizability.
We generalize the successful paradigm of physics-informed learning (PIL) into a more general framework that also includes what we term physicsaugmented learning (PAL).
PIL and PAL complement each other by handling discriminative and generative properties, respectively.
In numerical experiments, we show that PAL performs well on examples where PIL is inapplicable or inefﬁcient
Introduction
Physics-informed learning (PIL) [1, 2, 3, 4, 5] has been widely used in scientiﬁc applications where physical inductive biases are applicable.
The integration of domain knowledge into machine learning can not only enhance generalization, but also make models more interpretable.
However, PIL implicitly requires physics properties to be discriminative , as opposed to generative (deﬁned below).
To complement PIL, we propose a new paradigm call physics-augmented learning (PAL) to handle generative properties, as illustrated in Figure 1.
We deﬁne and compare discriminative and generative properties in Section 2, propose PAL in Section 3 and compare it with PIL, and demonstrate PAL’s effectiveness via numerical experiments in Section 4.
[Figure]
and generative properties respectively.
Discriminative and Generative Properties
What is a property?
A property P is a mapping from an object f to a boolean variable: P ( f ) is true if f satisﬁes P , false otherwise.
For example, if f refers to an individual with age a , and P is the statement that “The individual has the age no more than 30.", then P ( f ) is true if a ≤ 30 and P ( f ) is false if a > 30 .
Inspired by Generative adversarial networks (GAN) [6], we deﬁne a generator and a discriminator associated with a property P .
Deﬁnition 2.1 (Generator) .
A generator can generate objects f that have the property P .
Deﬁnition 2.2 (Discriminator) .
A discriminator determines if an input object f has the property P or not.
To be maximally useful, a generator should be implementable as a symbolic formula or feedforward neural network, and a discriminator can be implementable as a classiﬁer or a loss function.
More speciﬁcally, we deﬁne ideal generators and discriminators as follows:
Deﬁnition 2.3 (Ideal Generator) .
An ideal generator is (1) accurate (never generate an f such that
P ( f ) is false), (2) complete (can generate any f such that P ( f ) is true), (3) efﬁcient (can generate f in polynomial time), and (4) differentiable (can exploit derivative-based optimization methods such as back propagation).
Deﬁnition 2.4 (Ideal Discriminator) .
An ideal discriminator is (1) accurate (always computes
P ( f ) correctly), (2) efﬁcient (computes P ( f ) in polynomial time), and (3) differentiable (can exploit derivative-based optimization methods such as back propagation).
In this paper, we deal with one speciﬁc ideal discriminator ˆ L such that ˆ Lf = 0 when P ( f ) is true and ˆ Lf (cid:54) = 0 when P ( f ) is false.
We now deﬁne discriminative and generative properties:
Deﬁnition 2.5 (Generative property) .
A property P is generative if there exists an ideal generator for P .
Deﬁnition 2.6 (Discriminative property) .
A property P is discriminative if there exists an ideal discriminator for P .
Let us clarify these abstract deﬁnitions with a few examples of properties below, summarized in Table 1).
A. Lagrangian property For a physics system with generalized coordinate q and velocity ˙ q , the acceleration ﬁeld ¨ q is Lagrangian if there exists a Lagrangian function L ( q , ˙ q )
such that ¨ q = ( ∇ ˙ q ∇ T ˙ q L ) − 1 ( ∇ q L − ( ∇ q ∇ T ˙ q L ) ˙ q ) [7, 8].
The Lagrangian property is generative by deﬁnition but not discriminative, because (perhaps surprisingly) there is no known efﬁcient method to determine whether a given ¨ q is Lagrangian or not [8].
B. Positive deﬁniteness By analogy with linear operators, we say that a function f ( x ) is positive deﬁnite if there exists a function g such that f ( x ) =
g ( g ( x )) .
For example, the function that time-evolves a physical system during an interval ∆ T is positive deﬁnite, since it is equivalent by time-evolving by ∆ T/ 2 twice.
The positive deﬁniteness property is generative by deﬁnition, but not discriminative to the best of our knowledge.
C. Manifest symmetry Many manifest symmetries are discriminative, with associated discriminators that can be elegantly described by partial differential equations [9].
For example, a vector ﬁeld f ( x ) is symmetric under a Lie group G if f ( g x )
=
g ( f ( x )) for all g ∈ G .
This symmetry property is discriminative because it is equivalent to ( K i x · ∇ − K i ) f
= 0 for all group generators K i [9].
It is also generative due to recent advances in equivariant neural networks [10, 11, 12, 13, 14].
D. Hidden symmetry However, hidden symmetries are not discriminative, because they require coordinate transformations to ‘generate’ manifestly symmetric objects [9].
For example, manifest
Hamiltonicity is shown to be discriminative in [9], but hidden Hamiltonicity is a generative property and that is equivalent to Lagrangianess [8].
[Table] [Table]
E. Separability :
A differentiable bivariate function f ( x 1 , x 2 ) is (additively) separable if there exists two unary functions
f 1 and f 2 such that f ( x 1 , x 2 ) =
f 1 ( x 1 ) + f 2 ( x 2 ) .
Separability is generative by deﬁnition, and also discriminative because it is equivalent to ˆ Lf ≡ ∂ 2 f/∂x 1 ∂x 2 =
F. PDE satisﬁability We say a function f ( x, t ) satisﬁes a partial differential equation (PDE) if g ( t, f, f t , f x , .. )
= 0 .
This property is discriminative by deﬁnition, with ˆ Lf = g .
It is also generative when f ( x, t ) can be efﬁciently computed by a numerical PDE solver given proper boundary conditions; this idea underlies neural ordinary/partial/stochastic differential equations [17, 18, 19].
Physics-informed Learning (PIL) and Physics-augmented learning (PAL)
In this section, we ﬁrst review physics-informed learning (PIL) and its limitations, motivating our proposed physics-augmented learning (PAL) framework.
3.1 Physics-informed Learning (PIL)
The essence of PIL is to seamlessly integrate data and mathematical physics models, and a common way is to add a soft penalty term L 2 (corresponding to physics properties) to the prediction loss
L 1 [1] (Figure 1, left panel).
PIL works for discriminative properties.
Indeed, one of its greatest successes lies in solving forward/inverse PDE problems [2, 3, 5], based on the unstated fact that satisfying a PDE is discriminative.
We clarify PIL with a toy example below.
Example: PIL for separability Suppose that our training dataset ( N samples) is generated by the oracle y =
f 0 ( x 1 , x 2 ) where f 0 : R 2 → R and that we want to use a parametrized neural network f ( x 1 , x 2 ; θ ) to ﬁt the data with a function f ( x 1 , x 2 ; θ ) that is additively separable.
PIL does this using a loss function with two terms: L ≡ L 1 + λL 2 , where the prediction loss L 1 and separability loss L 2 are
[Equation]
i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ∂ 2 f ( x ( i ) 1 , x ( i ) 2 ; θ ) ∂x 1 ∂x(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) , and the constant λ > 0 is a penalty coefﬁcient.
By deﬁnition, each discriminative property P can be written ˆ Lf = 0 for some operator ˆ L , so
L 2 ≡ | ˆ Lf | is a natural measure of property violation.
In contrast, non-discriminative properties, e.g. , being Lagrangian or positive deﬁnite, lack a known efﬁciently computable criterion ˆ Lf = 0 .
Fortunately, the PAL framework proposed below can come to rescue whenever the properties of interest are generative .
3.2 Physics-augmented Learning (PAL)
Although both PIL and PAL aim to leverage inductive biases in machine learning, their approaches are quite different: while PIL is based on regularization design, PAL is based on model design.
Useful regularization designs and model designs are only available for properties that are discriminative and generative, respectively.
In the PAL paradigm, the whole model consists of two parallel modules (see Figure 1, right panel): the ﬁrst module ( PhyGen ) strictly satisﬁes the generative property, and the second module ( Blackbox ) augments the expressive power to allow violation of the property.
The loss function consists of two terms: the standard prediction error L 1 , and a penalty term L 2 deﬁned as some norm of Blackbox module output.
The combined loss function is L =
L 1 + λL 2 .
Example: PAL for separability In PAL we have two neural modules PhyGen and Blackbox .
PhyGen satisﬁes strictly the additive separability by having two sub-networks ( f 1 ( x 1 ; θ 1 ) , f 2 ( x 2 ; θ 2 )) and outputs their sum.
Blackbox is a fully-connected neural network f 12 ( x 1 , x 2 ; θ 12 ) that can universally approximate any two-variable continuous function.
The whole prediction is thus ( f 1 ( x 1 ; θ 1 ) + f 2 ( x 2 ; θ 2 ))
+ f 12 ( x 1 , x 2 ; θ 12 ) and the prediction loss L 1 is its distance from the label y
= f 0 ( x 1 , x 2 ) .
The penalty loss is simply the function norm of the Blackbox i.e. ,
L 2 = | f 12 ( x 1 , x 2 ; θ 12 ) | .
Our PIL and PAL examples are summarized and compared in Table 2.
When should I use PAL rather than PIL?
PIL and PAL are complementary frameworks exploiting discriminative and generative properties, respectively.
As a consequence, one should resort to PAL without hesitation when the property of interest is generative and non-discriminative.
Another reason to use PAL is that it provides a model decomposition into PhyGen and Blackbox , and interpreting them potentially produces physical insights.
For example, the decomposition into conservative and non-conservative force ﬁelds enabled new-physics discovery in [8].
How to choose the hyperparameter λ ?
It was recently proven that L =
L 1 + λL 2 produces a phase transition at λ = 1 [8] 2 .
λ > 1 is the undesirable phase, so in principle one can simply choose any λ < 1 to obtain vanishing prediction loss; the numerical results of [8] suggest that λ ∈ [0 . 02 , 0 .
5] produces accurate and robust results in practice.
We verify these observations with the additive separability example in Appendix B.
In our numerical experiments, we choose λ = 0 .
2 if not stated otherwise.
Numerical Experiments
We demonstrate the effectiveness of PAL on two tasks: symbolic regression and dynamics prediction.
PAL performs well on these applications, while PIL is inapplicable or performs worse than PAL.
4.1 Symbolic Regression
The goal of symbolic regression is to ﬁnd a symbolic expression that matches data from an unknown function f .
The physics-inspired AI Feynman symbolic regression module [15, 16] tests if a dataset satisﬁes certain properties, including symmetries.
However, AI Feynman can only discovers and exploits these if they hold with high accuracy.
To relax this, we employ PAL to ﬁrst decompose the function f into two parts: a property-satisfying part f + and a property-violating residual f − .
We then apply AI Feynman to both parts separately to obtain symbolic formulas; f + satisﬁes the strict property which AI Feynman can exploit.
We experiment with three properties: additive separability, rotational invariance and positive deﬁniteness.
Training details can be found in Appendix B. Table 3 shows the symbolic regression results: for the ﬁrst two properties, PAL decomposes the function as desired while PIL can only learn the whole function; for the positivity example, PIL is not applicable while PAL can extract meaningful partial function g .
Property
Methods f + f −
Additive Separability f
= x 21 + x 22 + x 1 x
Truth ( x 21 ) + ( x 22 ) x 1 x
AI Feynman + PAL ( x 21 − 0 .
+ ( x 22 − 0 .
x 1 x 2 + 0 .
AI Feynman + PIL x 21 + x 22 + x 1 x
Rotational Invariance f = 0 .
5( x 21 + x 22 ) + 0 .
32 x
Truth 0 .
5( x 21 + x 22 ) 0 .
32 x
AI Feynman + PAL 0 .
5( x 21 + x 22 ) 0 .
31998 x
[Figure]
AI Feynman + PIL 0 .
5( x 21 + x 22 ) + 0 .
[Figure]
Positivity f = sin(sin( x ))
+
Truth sin(sin( x ))
AI Feynman + PAL g ( g ( x )) , g = − sin( x ) + 0 .
AI Feynman + PIL
Not Applicable
[Figure]
4.2 Dynamics prediction: N-body Problem
In this example, we aim to learn N -body dynamics from the initial ( t = 0) and ﬁnal ( t = 1) states of N particles.
We assume that these unit mass particles ( m = 1 ) obey the Newtonian mechanics with pairwise interactions, i.e. , particle i exerts a force f ij = f
( | x i − x j | ) x i − x j | x i − x j | 2 on particle j .
Our ground truth is that f ( r ) =
r 2 when computing ﬁnal states from initial states, but we pretend not to know f and aim to infer it solely from the initial and ﬁnal states (indicated by triangles and dots in
Figure 2).
The property P that we explore with PAL and PIL is the time-independence of f , which is clearly both discriminative and generative.
Details of loss functions, data generation and training details are included in Appendix A and C. Figure 2 shows that PAL outperforms PIL in terms of both trajectory interpolation and force recovery.
In other words, although both PIL and PAL can be applied for this example, PAL reveals the underlying dynamics much more accurately that PIL.
Conclusions & Discussions
We have proposed a new paradigm called physics-augmented learning (PAL) to effectively integrate physical properties into unconstrained neural networks.
PAL complements the well-known physicsinformed learning (PIL) paradigm, by applying in some cases where PIL is inapplicable and by outperforming PIL in some cases where both can be used.
While PIL is based on regularization design and applies to discriminative properties, PAL is based on model design and applies to generative properties.
Although PAL in its general form is explicitly formulated for the ﬁrst time in this paper, examples of it have been implicitly adopted in many successful machine learning models owing to its ability to integrate human knowledge into the model design phase.
For example, AlphaFold2 demonstrated that designing deep learning models with proper inductive biases could give superior performance for an unsolved grand challenge [20].
There is growing interest in the ML community in how inductive biases can shape deep learning models; for example, [21] discovers how two fundamental deep learning models (CNNs and Transformers) leverage their own inductive biases and tackle challenges in the computer vision domain.
We are hopeful that the PAL-PIL framework unifying inductive biases can lead to further advances in the ML community for tackling real-world challenges with optimal selection and design of inductive biases.
Acknowledgement
We would like to thank Silviu Udrescu and Jiaxi Zhao for valuable discussions. We thank the Center for Brains, Minds and Machines (CBMM) for hospitality. This work was supported by The Casey and Family Foundation, the Foundational Questions Institute, the Rothberg Family Fund for Cognitive Science and IAIFI through NSF grant PHY-
A Deﬁnitions of PIL and PAL losses
In Section 4, we conducted experiments related to four properties: additive separability, rotational invariance, positivity and time independence.
PAL applies to all of these properties, while PIL is not applicable to positivity but applies to other three properties.
In Table 4 we show the design of neural networks and loss functions for each property in the framework of PIL and PAL.
All neural networks f ∗ ( ∗ ; ∗ ) (each ∗ can be anything) are fully-connected networks which can approximate any continuous function of input variables (if wide enough).
Their differences lie in input variables.
For PIL, the only network is fully-connected and takes in all available variables.
For
PAL, we have a PhyGen and Blackbox which usually have different inputs.
Blackbox takes in all available input variables while PhyGen usually takes in fewer variables which is determined by the speciﬁc property, as detailed below.
• Additive separability.
PhyGen : f 1 takes in x 1 only, while f 2 takes in x 2 only.
The output is f 1 ( x 1 ; θ 1 ) + f 2 ( x 2 ; θ 2 ) .
Blackbox : f 12 takes in both x 1 and x 2 .
The output is f 12 ( x 1 , x 2 ; θ 12 ) .
• Rotational invariance.
PhyGen : f 1 takes in one variable R ≡ (cid:112) x 21 + x 22 only.
The output is f 1 ( R ; θ 1 ) .
Blackbox : f 2 takes in both x 1 and x 2 .
The output is f 2 ( x 1 , x 2 ; θ 2 ) .
• Positivity.
PhyGen : f 1 takes in x and outputs f 1 ( f 1 ( x ; θ 1 ); θ 1 ) .
Blackbox : f 2 takes in x and outputs f 2 ( x ; θ 2 ) .
• Time independence.
PhyGen : f 1 takes in r where r is the distance between two particles.
The output is f 1 ( r ; θ 1 ) .
Blackbox : f 2 takes in both r and t .
The output is f 2 ( r, t ; θ 2 ) . [Table] [Table] x ; θ 2 ) |
[Table]
,t PIL f [Table] [Table]
) |
[Table]
=
[Table]
[Table]
|
PAL
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
2 ) |
B Symbolic Regression Details
Neural network architecture :
In the symbolic regression experiment, all the neural networks are fully-connected networks, which have 1 or 2 input neurons (depends on the number of input variables), 2 hidden layers (width = 256 ) with LeakyReLU ( α = 0 . 2) , and a single output neuron.
Additive separability :
The function f 0 ( x 1 , x 2 ) =
( x 21 + x 22 ) + x 1 x 2 can be decomposed into the additively separable part x 21 + x 22 and a violation part x 1 x 2 .
We employ PAL To obtain the decomposition numerically ﬁrst.
we train three neural networks, f 1 ( x 1 ; θ 1 ) , f 2 ( x 2 ; θ 2 ) , and f 12 ( x 1 , x 2 ; θ 12 ) , with λ = 0 .
2 for 200 epochs.
We use the ADAM optimizer and annealed learning rate schedule i.e., { 10 − 3 , 10 − 4 , 10 − 5 , 10 − 6 } each learning rate for 50 epochs.
We show three neural networks succeed in learning the ground truth decomposition in
Figure 3(a)-(b).
After PAL training, we then apply AI Feynman to explain the outputs i.e., symbolic expression of these three neural networks:
f 1 = x 21 − 0 .
02 , f 2 = x 22 − 0 .
01 , and f 12 = x 1 x 2 + 0 .
Phase transition A recent work [8] theoretically proves that L 1 , L 2 ∼ λ has a phase transition behavior at λ = 1 3 .
λ > 1 is the undesireable phase, so in principle one can simply choose any λ < 1 to obtain the correct result.
Their numerical results suggest that λ ∈ [0 . 02 , 0 . 5] produce very accurate and robust results.
We verify these observations in the current example.
We sweep the λ region and obtained ﬁnal losses ( L 1 and L 2 ) as a function of λ in PAL and PIL, by testing λ = { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 .
We show the results in Figure 3(c)-(d).
For PAL, there is a sharp phase transition at λ = 1 and two losses are quite robust for any λ < 1 .
Intriguingly, the sharp phase transition for PIL has never been reported let alone studied before (to the best of knowledge), which will be interesting future directions.
Rotational invariance : We use PAL to decompose the function f 0 ( x 1 , x 2 )
=
12 ( x 21 + x 22 )+ ax 1 into the rotational-invariant part 12 ( x 21 + x 22 ) and violation part ax 1 ( a = 0 .
In the experiment, we take λ = 0 .
2 and jointly train f 1 ( R = (cid:112)
x 21 + x 22 ; θ 1 ) and f 2 ( x 1 , x 2 ; θ 2 ) for 200 epochs.
We use the
ADAM optimizer and annealed learning rate schedule i.e., { 10 − 3 , 10 − 4 , 10 − 5 , 10 − 6 } each learning rate for 50 epochs.
We apply AI Feynman to explain f 1 and f 2 parameterized by neural networks: AI Feynman discovers that f 1 = 0 .
5( x 21 + x 22 ) and f 2 = 0 .
31998 x 1 .
Positive Deﬁniteness : Given a function f 0 ( x ) = sin(sin( x ))
+0 , we train the the ﬁrst neural network in a nested form f 1 ( f 1 ( x ; θ 1 ); θ 1 ) for the positivity part and the second neural network for violation part.
We jointly train two networks for 2000 epochs with the ADAM optimizer.
We employ annealed learning rate schedule i.e., { 10 − 3 , 10 − 4 , 10 − 5 , 10 − 6 } each learning rate for 500 epochs.
AI Feynman discovers the symbolic expression for two networks: f 1 = − sin( x ) + 0 . 004 and f 2 = 0 .
PIL for additive separability and rotational invariance : To compare PAL and PIL, we also use
PIL to learn the additive separability and rotational invariance under the same setting.
The results of symbolic regression are x 21 + x 22 + x 1 x 2 (additive separability) and 0 .
5( x 21 + x 22 )+0 .
32 x 1 (rotational invariance).
Although PIL can obtain correctly the whole symbolic expression, it does not support decompositions into the property part and the violation part.
For the positivity example, PIL is even not applicable since positivity is non-discriminative.
[Figure]
C N-body Dynamics Details Physical Model
We consider a 2D N -particle system described by their positions x i and velocities v i ( i = 1 , · · · , N ) .
They all have unit mass m = 1 and have pairwise forces i.e., particle i exerts to particle j a force f ij = f ( r ij ) r ij r ij where r ij ≡ x i − x j , r ij ≡ | r ij | 2 and f ( · ) is the same for all pairs.
The dynamical equations for the j -th particle is:
[Equation]
Concatenating all the particle states together deﬁnes z = [ x 1 , · · · ,
[Equation]
Discretizing Eq.
(2) in time gives
[Equation]
In our simulation, the initial position and initial velocity of each particle are randomly sampled from a standard Gaussian distribution x i ∼ N ([0 , 0] T , σ 2 I 2 × 2 ) where σ = 1 .
The force between each pair of two bodies is set as f ( r ) =
r 2 .
We use Newton forward scheme as in Eq.
(3) to simulate the 5-body system for 50 steps with step size ∆ t = 0 .
The initial positions for the 5 bodies are (1 . 62 , − 0 . 61) , ( − 0 .
53 , − 1 . 07) , (0 . 87 , − 2 . 30) , (1 . 74 , − 0 .
32 , − 0 .
The initial velocities for the 5 bodies are (2 . 92 , − 4 . 12) , ( − 0 .
64 , − 0 .
77) , (2 . 27 , − 2 . 20) , ( − 0 .
34 , − 1 .
Neural network architecture
Note that our goal is to infer the interaction f ( · ) based on solely the initial z ( t = 0) and ﬁnal z ( t = 1) .
Our network is effectively a residual network with 50 blocks – it simply implements the computations as in Eq.
(3), with only f parameterized by neural networks.
To impose f to be time-independent,
PAL and PIL employ different strategies.
PAL : explicit decomposition f
( r, t ) =
f 1 ( r ; θ 1 ) + f 2 ( r, t ; θ 2 ) and penalize | f 2 ( r, t ; θ 2 ) | .
Both fand f 2 are 2-hidden-layer fully-connected networks (tanh activation) with each layer containing (1 / 2 , 200 , 200 , 1) neurons.
we take λ 1 = 1 , λ 2 = 0 .
25 , and λ 3 = 1 , where λ 1 is the weight for the ﬁnal position prediction loss, λ 2 is the weight for the ﬁnal velocity prediction loss, and λ 3 penalty coefﬁcient for f 2 ( r, t ; θ 2 ) .
We train two neural networks f 1 ( r ; θ 1 ) and f 2 ( r, t ; θ 2 ) , each repeated 100 times and stacked vertically to obtain a 100-layer Resnet.
Both f 1 and f 2 are two-hidden-layer fully-connected networks (Activation function: tanh) with each layer containing (1 / 2 , 200 , 200 , 1) neurons.
We train two networks jointly for 2000 epochs with the Adam optimizer with the learning rate 0 . 001 .
PIL : no explicit decomposition but penalize | f ( r, t 1 ) − f
( r, t 2 ) | for t 1 (cid:54) =
t 2 .
Each block is a 2-hiddenlayer fully-connected network (tanh activation) with each layer containing (1 / 2 , 200 , 200 , 1) neurons.
For the loss function, we take λ 1 = 1 , λ 2 = 0 .
25 , and λ 3 = 0 .
1 , where λ 1 is the weight for the ﬁnal position prediction loss, λ 2 is the ﬁnal velocity prediction loss, and λ 3 is the time-independence penalty loss.
The time-independence loss is simply implemented as the MSE between the outputs of two neighboring blocks of the Resnet for the same input.
We train the Resnet for 2000 epochs with an Adam optimizer with the learning rate 0 . 0001 .