DEQGAN: Learning the Loss Function for PINNs with Generative Adversarial Networks
Abstract
Solutions to differential equations are of significant scientific and engineering relevance.
Physics-Informed Neural Networks (PINNs) have emerged as a promising method for solving differential equations, but they lack a theoretical justification for the use of any particular loss function.
This work presents Differential Equation GAN (DEQGAN), a novel method for solving differential equations using generative adversarial networks to “learn the loss function” for optimizing the neural network.
Presenting results on a suite of twelve ordinary and partial differential equations, including the nonlinear Burgers’, AllenCahn, Hamilton, and modified Einstein’s gravity equations, we show that DEQGAN 1 can obtain multiple orders of magnitude lower mean squared errors than PINNs that use L 2 , L 1 , and Huber loss functions.
We also show that DEQGAN achieves solution accuracies that are competitive with popular numerical methods.
Finally, we present two methods to improve the robustness of DEQGAN to different hyperparameter settings.
1. Introduction
In fields such as physics, chemistry, biology, engineering, and economics, differential equations are used to model important and complex phenomena.
While numerical methods for solving differential equations perform well and the theory for their stability and convergence is well established, the recent success of deep learning (Krizhevsky et al., 2012;
Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017; Mnih et al., 2013; Dabney et al., 2018; Gu et al.,
2017; Silver et al., 2018) has inspired researchers to apply neural networks to solving differential equations, which has given rise to the growing field of Physics-Informed Neural
Networks (PINNs) (Raissi et al., 2019; Hagge et al., 2017;
Piscopo et al., 2019; Mattheakis et al., 2019; Stevens &
Colonius, 2020; Mattheakis et al., 2020; Han et al., 2018;
Raissi, 2018; Sirignano & Spiliopoulos, 2018).
In contrast to traditional numerical methods, PINNs: provide solutions that are closed-form (Lagaris et al., 1998), suffer less from the “curse of dimensionality” (Han et al., 2018; Raissi, 2018; Sirignano & Spiliopoulos, 2018;
Grohs et al., 2018), provide a more accurate interpolation scheme (Lagaris et al., 1998), and can leverage transfer learning for fast discovery of new solutions (Flamant et al., 2020; Desai et al., 2021).
Further, PINNs do not require an underlying grid and offer a meshless approach to solving differential equations.
This makes it possible to use trained neural networks, which typically have small memory footprints, to generate solutions over arbitrary grids in a single forward pass.
PINNs have been successfully applied to a wide range of differential equations, but provide no theoretical justification for the use of a particular loss function.
In domains outside of differential equations, data following a known noise model (e.g. Gaussian) have clear justification for fitting models with specific loss functions (e.g. L 2 ).
In the case of deterministic differential equations, however, there is no noise model and we lack an equivalent justification.
To address this gap in the theory, we propose generative adversarial networks (GANs) (Goodfellow et al., 2014) for solving differential equations in a fully unsupervised manner.
Recently, Zeng et al.
(2022) showed that adaptively modifying the loss function throughout training can lead to improved solution accuracies.
The discriminator network of our GAN-based method, however, can be thought of as “learning the loss function” for optimizing the generator, thereby eliminating the need for a pre-specified loss function and providing even greater flexibility than an adaptive loss.
Beyond the context of differential equations, it has also been shown that where classical loss functions struggle to capture complex spatio-temporal dependencies, GANs may be an effective alternative (Larsen et al., 2015; Ledig et al., 2016; Karras et al., 2018).
Our contributions in this work are summarized as follows:
• We present Differential Equation GAN (DEQGAN), a novel method for solving differential equations in a fully unsupervised manner using generative adversarial networks.
• We highlight the advantage of “learning the loss function” with a GAN rather than using a pre-specified loss function by showing that PINNs trained using L 2 , L 1 , and Huber losses have variable performance and fail to solve the modified Einstein’s gravity equations (Chantada et al., 2022).
• We present results on a suite of twelve ordinary differential equations (ODEs) and partial differential equations (PDEs), including highly nonlinear problems, showing that our method produces solutions with multiple orders of magnitude lower mean squared errors than PINNs that use L 2 , L 1 , and Huber loss functions.
• We show that DEQGAN achieves solution accuracies that are competitive with popular numerical methods, including the fourth-order Runge-Kutta and second-order finite difference methods.
• We present two techniques to improve the training stability of DEQGAN that are applicable to other GAN-based methods and PINN approaches to solving differential equations.
2. Related Work
A variety of neural network methods have been developed for solving differential equations.
Some of these are supervised and learn the dynamics of real-world systems from data (Raissi et al., 2019;
Choudhary et al., 2020; Greydanus et al., 2019; Bertalan et al., 2019).
Others are semisupervised, learning general solutions to a differential equation and extracting a best fit solution based on observational data (Paticchio et al., 2020).
Our work falls under the category of unsupervised neural network methods, which are trained in a data-free manner that depends solely on the equation residuals.
Unsupervised neural networks have been applied to a wide range of ODEs (Lagaris et al., 1998;
Flamant et al., 2020; Mattheakis et al., 2020; 2021) and PDEs (Han et al., 2018; Sirignano & Spiliopoulos, 2018; Raissi, 2018;
Stevens & Colonius, 2020), primarily use feed-forward architectures, and require the specification of a particular loss function computed over the equation residuals.
Goodfellow et al.
(2014) introduced the idea of learning generative models with neural networks and an adversarial training algorithm, called generative adversarial networks (GANs).
To solve issues of GAN training instability, Arjovsky et al.
(2017) introduced a formulation of GANs based on the Wasserstein distance, and Gulrajani et al.
(2017) added a gradient penalty to approximately enforce a Lipschitz constraint on the discriminator.
Miyato et al.
(2018) introduced an alternative method for enforcing the Lipschitz constraint with a spectral normalization technique that outperforms the former method on some problems.
Further work has applied GANs to differential equations with solution data used for supervision.
Yang et al.
(2018) apply GANs to stochastic differential equations by using “snapshots” of ground-truth data for semi-supervised training.
A project by students at Stanford (Subramanian et al., 2018) employed GANs to perform “turbulence enrichment” of solution data in a manner akin to that of super-resolution for images proposed by Ledig et al.
Our work distinguishes itself from other GAN-based approaches for solving differential equations by being fully unsupervised , and removing the dependence on using supervised training data (i.e. solutions of the equation).
3. Background 3.1. Unsupervised Neural Networks for Differential Equations
Early work by Dissanayake & Phan-Thien (1994) proposed solving initial value problems in an unsupervised manner with neural networks.
In this work, we extend their approach to handle spatial domains and multidimensional problems.
In particular, we consider general differential equations of the form
[Equation]
where Ψ( t, x ) is the desired solution, d Ψ /dt and d 2 Ψ /dtrepresent
the first and second time derivatives, ∆Ψ and ∆ 2 Ψ are the first and second spatial derivatives, and the system is subject to certain initial and boundary conditions.
The learning problem can then be formulated as minimizing the sum of squared residuals (i.e., the squared L 2 loss) of the above equation
[Equation]
where Ψ θ is a neural network parameterized by θ , D is the domain of the problem, and derivatives are computed with automatic differentiation.
This allows backpropagation
(Hecht-Nielsen, 1992) to be used to train the neural network to satisfy the differential equation.
We apply this formalism to both initial and boundary value problems, including multidimensional problems, as detailed in Appendix A.2.
3.2. Generative Adversarial Networks
Generative adversarial networks (GANs) (Goodfellow et al., 2014) are generative models that use two neural networks to induce a generative distribution p ( x ) of the data by formulating the inference problem as a two-player, zero-sum game.
The generative model first samples a latent random variable z ∼ N (0 , 1) , which is used as input into the generator G (e.g., a neural network).
A discriminator D is trained to classify whether its input was sampled from the generator (i.e., “fake”) or from a reference data set (i.e., “real”).
Informally, the process of training GANs proceeds by optimizing a minimax objective over the generator and discriminator such that the generator attempts to trick the discriminator to classify “fake” samples as “real”.
Formally, one optimizes
[Equation]
where x ∼ p data ( x ) denotes samples from the empirical data distribution, and p z ∼ N (0 , 1) samples in latent space (Goodfellow et al., 2014).
In practice, the optimization alternates between gradient ascent and descent steps for D and G respectively.
3.2.1. T WO T IME -S CALE U PDATE R ULE
Heusel et al.
(2017) proposed the two time-scale update rule (TTUR) for training GANs, a method in which the discriminator and generator are trained with separate learning rates.
They showed that their method led to improved performance and proved that, in some cases, TTUR ensures convergence to a stable local Nash equilibrium.
One intuition for TTUR comes from the potentially different loss surfaces of the discriminator and generator.
Allowing learning rates to be tuned to a particular loss surface can enable more efficient gradient-based optimization.
We make use of
TTUR throughout this paper as an instrumental lever when tuning GANs to reach desired performance.
3.2.2. S PECTRAL N ORMALIZATION
Proposed by Miyato et al.
(2018), Spectrally Normalized
GAN (SN-GAN) is a method for controlling exploding discriminator gradients when optimizing Equation 3 that leverages a novel weight normalization technique.
The key idea is to control the Lipschitz constant of the discriminator by constraining the spectral norm of each layer in the discriminator.
Specifically, the authors propose dividing the weight matrices
W i of each layer
i by their spectral norm σ ( W i )
[Equation]
where
[Equation]
and h i denotes the input to layer i .
The authors prove that this normalization technique bounds the Lipschitz constant of the discriminator above by 1 , thus strictly enforcing the 1 -Lipshcitz constraint on the discriminator.
In our experiments, adopting the SN-GAN formulation led to even better performance than WGAN-GP (Arjovsky et al., 2017; Gulrajani et al., 2017).
3.3. Guaranteeing Initial & Boundary Conditions
Lagaris et al.
(1998) showed that it is possible to exactly satisfy initial and boundary conditions by adjusting the output of the neural network.
For example, consider adjusting the neural network output Ψ θ ( t, x ) to satisfy the initial condition Ψ θ ( t, x ) (cid:12)(cid:12) t
= t 0 = x 0 .
We can apply the re-parameterization
[Equation]
) which exactly satisfies the initial condition.
Mattheakis et al.
(2020) proposed an augmented re-parameterization
[Equation]
that further improved training convergence.
Intuitively,
Equation 7 adjusts the output of the neural network Ψ θ ( t, x ) to be exactly x 0 when t
= t 0 , and decays this constraint exponentially in t .
Chen et al.
(2020) provide reparameterizations to satisfy a range of other conditions, including Dirichlet and Neumann boundary conditions, which we employ in our experiments and detail in Appendix A.2.
3.4. Residual Connections
He et al.
(2015) showed that the addition of residual connections improves deep neural network training.
We employ residual connections in our networks, as they allow gradients to flow more easily through the models and thereby reduce numerical instability.
Residual connections augment a typical activation with the identity operation.
[Equation]
where F is the activation function, x is the input to the unit,
W i are the weights and y is the output of the unit.
This acts as a “skip connection”, allowing inputs and gradients to forego the nonlinear component.
4. Differential Equation GAN
In this section, we present our method, Differential Equation
GAN (DEQGAN), which trains a GAN to solve differential equations in a fully unsupervised manner.
To do this, we rearrange the differential equation so that the left-hand side ( LHS ) contains all the terms which depend on the generator (e.g. Ψ , d Ψ /dt , ∆Ψ , etc.) and the right-hand side ( RHS ) contains only constants (e.g. zero).
During training, we sample points from the domain ( t, x ) ∼
D and use them as input to a generator G ( x ) , which produces candidate solutions Ψ θ .
We sample points from a noisy grid that spans D , which we found reduced interpolation error in comparison to sampling points from a fixed grid.
We then adjust Ψ θ for initial or boundary conditions to obtain the re-parameterized output ˜Ψ θ , construct the LHS from the differential equation F using automatic differentiation
[Equation]
and set RHS to zero.
Note that in our procedure, we add
Gaussian noise to RHS , which we found improves training of the discriminator, as described in Section 4.1.
Training proceeds in a manner similar to that of traditional GANs.
We update the weights of the generator G and the discriminator
D according to the gradients
[Equation]
[Equation]
g D = ∇ θ d 1 m m
[Equation]
where LHS ( i ) is the output of G (cid:0)
x ( i ) (cid:1) after adjusting for initial or boundary conditions and constructing the LHS from F .
Note that we perform stochastic gradient descent for G (gradient steps ∝ − g G ), and stochastic gradient ascent for D (gradient steps ∝ g D ).
We provide a schematic representation of DEQGAN in Figure 1 and detail the training steps in Algorithm 1.
Algorithm 1 DEQGAN
Input: Differential equation F , generator G ( · ; θ g ) , discriminator D ( · ; θ d ) , grid x of m points with spacing ∆ x , perturbation precision τ , re-parameterization function Φ , total steps N , learning rates η G , η D , Adam optimizer parameters β G 1 , β G 2 , β D 1 , β Dfor i = 1 to N do for j = 1 to m do
Perturb j -th point in mesh x ( j ) s
= x ( j ) + ϵ, ϵ ∼ N (0 , ∆ xτ )
Forward pass Ψ θ =
G ( x ( j ) s )
Analytic re-parameterization ˜Ψ
θ = Φ(Ψ θ )
Compute LHS ( j ) (Equation 9)
Set RHS ( j ) =end for
Compute gradients g G , g D (Equation 10 & 11)
Update generator θ g ← Adam ( θ g , − g G , η G , β G 1 , β G 2 )
Update discriminator θ d ← Adam ( θ d , g D , η D , β D 1 , β D 2 ) end for
Output: G
Informally, our algorithm trains a GAN by setting the “fake” component to be the LHS (in our formulation, the residuals of the equation) and the “real” component to be the RHS of the equation.
This results in a GAN that learns to produce solutions that make LHS indistinguishable from RHS , thereby approximately solving the differential equation.
[Table]
[Equation]
PDE
[Equation]
No
[Equation]
PDE 2 nd
No
4.1. Instance Noise
While GANs have achieved state of the art results on a wide range of generative modeling tasks, they are often difficult to train.
As a result, much recent work on GANs has been dedicated to improving their sensitivity to hyperparameters and training stability (Salimans et al., 2016; Gulrajani et al., 2017; Sønderby et al., 2016; Arjovsky & Bottou, 2017; Karnewar et al., 2019; Kodali et al., 2017; Arjovsky et al., 2017; Berthelot et al., 2017; Mirza & Osindero, 2014;
Miyato et al., 2018).
In our experiments, we found that
DEQGAN could also be sensitive to hyperparameters, such as the Adam optimizer parameters shown in Algorithm 1.
Sønderby et al.
(2016) note that the convergence of GANs relies on the existence of a unique optimal discriminator that separates the distribution of “fake” samples p fake produced by the generator, and the distribution of the “real” data p data .
In practice, however, there may be many nearoptimal discriminators that pass very different gradients to the generator, depending on their initialization.
Arjovsky &
Bottou (2017) proved that this problem will arise when there is insufficient overlap between the supports of p fake and p data .
In the DEQGAN training algorithm, setting RHS = 0 constrains p data to the Dirac delta function δ (0) , and therefore the distribution of “real” data to a zero-dimensional manifold.
This makes it unlikely that p fake and p data will share support in a high-dimensional space.
The solution proposed by (Sønderby et al., 2016; Arjovsky & Bottou, 2017) is to add “instance noise” to p fake and p data to encourage their overlap.
This amounts to adding noise to the LHS and the RHS , respectively, at each iteration of Algorithm 1.
Because this makes the discriminator’s job more difficult, we add Gaussian noise with standard deviation equal to the difference between the generator and discriminator losses, L g and L d , i.e.
[Equation]
As the generator and discriminator reach equilibrium, Equation 12 will naturally converge to zero.
We use the ReLU function because when L d > L g , the generator is already able to fool the discriminator, suggesting that additional noise should not be used.
In Section 5.2, we conduct an ablation study and find that this improves the ability of
DEQGAN to produce accurate solutions across a range of hyperparameter settings.
[Table]
4.2. Residual Monitoring
One of the attractive properties of Algorithm 1 is that the “fake” LHS vector of equation residuals gives a direct measure of solution quality at each training iteration.
We observe that when DEQGAN training becomes unstable, the
LHS tends to oscillate wildly, while it decreases steadily throughout training for successful runs.
By monitoring the
L 1 norm of the LHS in the first 25% of training iterations, we are able to easily detect and terminate poor-performing runs if the variance of these values exceeds some threshold.
We provide further details on this method in Appendix A. and experimentally demonstrate that it is able to distinguish between DEQGAN runs that end in high and low mean squared errors in Section 5.2.
[Table]
5. Experiments
We conducted experiments on a suite of twelve differential equations (Table 1), including highly nonlinear PDEs and systems of ODEs, comparing DEQGAN to classical unsupervised PINNs that use (squared) L 2 , L 1 , and Huber (1964) loss functions.
We also report results obtained by the fourth-order Runge-Kutta (RK4) and second-order finite difference (FD) numerical methods for initial and boundary value problems, respectively.
The numerical solutions were computed over meshes containing the same number of points that were used to train the neural network methods.
Details for each experiment, including exact problem specifications and hyperparameters, are provided in Appendix
A.2 and A.4.
5.1. DEQGAN vs. Classical PINNs
We report the mean squared error of the solution obtained by each method, computed against known solutions obtained either analytically or with high-quality numerical solvers (Virtanen et al., 2020; Brunton & Kutz, 2019).
We added residual connections between neighboring layers of all models, applied spectral normalization to the discriminator, added instance noise to the p fake and p real , and used residual monitoring to terminate poor-performing runs in the first 25% of training iterations.
Results were obtained with hyperparameters tuned for DEQGAN.
In Appendix
A.5, we tuned each classical PINN method for comparison, but did not observe a significant difference.
Table 2 reports the lowest mean squared error obtained by each method across ten different model weight initializations.
We see that DEQGAN obtains lower mean squared errors than classical PINNs that use L 2 , L 1 , and Huber loss functions for all twelve problems, often by several orders of magnitude.
DEQGAN also achieves solution accuracies that are competitive with the RK4 and FD numerical methods.
Figure 2 plots the mean squared error vs. training iteration for six challenging equations and highlights multiple advantages of using DEQGAN over a pre-specified loss function (equivalent plots for the other six problems are provided in
Appendix A.3).
In particular, there is considerable variation in the quality of the solutions obtained by the classical
PINNs.
For example, while Huber performs better than
L 2 on the Allen-Cahn PDE, it is outperformed by L 2 on the wave equation.
Furthermore, Figure 2(f) shows that
the
L 2 , L 1 and Huber losses all fail to converge to an accurate
[Figure]
solution to the modified Einstein’s gravity equations.
Although this system has previously been solved using PINNs, the networks relied on a custom loss function that incorporated equation-specific parameters (Chantada et al., 2022).
DEQGAN, however, is able to automatically learn a loss function that optimizes the generator to produce accurate solutions.
DEQGAN solutions to four example equations are visualized in Figure 3, and similar plots for the other experiments are provided in Appendix A.2.
5.2. DEQGAN Training Stability: Ablation Study
In our experiments, we used instance noise to adaptively improve the training convergence of DEQGAN and employed residual monitoring to terminate poor-performing runs early.
To quantify the increased robustness offered by these techniques, we performed an ablation study comparing the percentage of high MSE ( ≥ 10 − 5 ) runs obtained by 500 randomized DEQGAN runs on the exponential decay equation.
This experimental setup is detailed further in
Appendix A.7.
Table 3 compares the percentage of high MSE runs with and without instance noise and residual monitoring.
We see that adding instance noise decreased the percentage of runs with high MSE and that residual monitoring is highly effective at filtering out poor performing runs.
When used together, these techniques eliminated all runs with MSE ≥ 10 − 5 .
These results agree with previous works, which have found that instance noise can improve the convergence of other GAN training algorithms (Sønderby et al., 2016; Arjovsky & Bottou, 2017).
Further, they suggest that residual
[Figure]
monitoring provides a useful performance metric that could be applied to other PINN methods for solving differential equations.
6. Conclusion
PINNs offer a promising approach to solving differential equations and to applying deep learning methods to challenging problems in science and engineering.
Classical
PINNs, however, lack a theoretical justification for the use of any particular loss function.
In this work, we presented
Differential Equation GAN (DEQGAN), a novel method that leverages GAN-based adversarial training to “learn” the loss function for solving differential equations with PINNs.
We demonstrated the advantage of this approach in comparison to using classical PINNs with pre-specified loss functions, which showed varied performance and failed to converge to an accurate solution to the modified Einstein’s gravity equations.
In general, we demonstrated that our method can obtain multiple orders of magnitude lower mean squared errors than PINNs that use L 2 , L 1 and Huber loss functions, including on highly nonlinear PDEs and systems of ODEs.
Further, we showed that DEQGAN achieves solution accuracies that are competitive with the fourth-order
Runge Kutta and second-order finite difference numerical methods.
Finally, we found that instance noise improved training stability and that residual monitoring provides a useful performance metric for PINNs.
While the equation residuals are a good measure of solution quality, PINNs lack the error bounds enjoyed by numerical methods.
Formalizing these bounds is an interesting avenue for future work and would enable PINNs to be more safely deployed in realworld applications.
Further, while our results evidence the advantage of “learning the loss function” with a GAN, understanding exactly what the discriminator learns is an open problem.
Post-hoc explainability methods, for example, might provide useful tools for characterizing the differences between classical losses and the loss functions learned by
DEQGAN, which could deepen our understanding of PINN optimization more generally.
A. Appendix A.1. Classical Loss Functions
A plot of the various classical loss functions is provided in Figure 4.
A.2. Description of Experiments A.2.1. E XPONENTIAL D ECAY (EXP)
Consider a model for population decay x ( t ) given by the exponential differential equation ˙ x ( t ) + x ( t ) = 0 , (13) with x (0) = 1 and t ∈ [0 , 10] .
The ground truth solution x ( t ) =
e − t can be obtained analytically, which we use to calculate the mean squared error of the predicted solution.
To set up the problem for DEQGAN, we define LHS =
˙
x
+ x and RHS = 0 .
Figure 5 presents the results from training
DEQGAN on this equation.
A.2.2. S IMPLE H ARMONIC O SCILLATOR (SHO)
Consider the motion of an oscillating body x ( t ) , which can be modeled by the simple harmonic oscillator differential equation ¨ x
[Equation]
t
[Equation]
with x (0) = 0 , ˙ x (0) = 1 , and t ∈ [0 , 2 π ] .
This differential equation can be solved analytically and has an exact solution x ( t )
= sin t .
Here we set LHS =
¨ x + x and RHS = 0 .
Figure 6 plots the results of training DEQGAN on this problem.
A.2.3. D AMPED N ONLINEAR O SCILLATOR (NLO)
Further increasing the complexity of the differential equations being considered, consider a less idealized oscillating body subject to additional forces, whose motion x ( t ) we can described by the nonlinear oscillator differential equation
[Equation]
15) with β = 0 . 1 , ω = 1 , ϕ = 1 , ϵ = 0 .
1 , x (0) = 0 , ˙ x (0)
= 0 .
5 , and t ∈ [0 , 4 π ] .
This equation does not admit an analytical solution.
Instead, we use the high-quality solver provided by SciPy’s solve_ivp (Virtanen et al., 2020).
We set LHS =
¨ x +2 β ˙ x + ω 2 x
+
ϕx 2 + ϵx 3 = 0 and RHS = 0 .
Figure 7 plots the results obtained from training DEQGAN on this equation.
A.2.4. C OUPLED O SCILLATORS (COO)
Consider the system of ordinary differential equations given by
[Equation]
(16) with x (0) = 1 , y (0) = 0 , and t ∈ [0 , 2 π ] .
This equation has an exact analytical solution given by
[Equation]
Here we set
[Equation]
and RHS =
[0 , 0] T .
Figure 8 plots the result of training DEQGAN on this problem.
A.2.5. SIR E PIDEMIOLOGICAL M ODEL (SIR)
Given the ongoing pandemic of novel coronavirus (COVID-19) (Wang et al., 2020), we consider an epidemiological model of infectious disease spread given by a system of ordinary differential equations.
Specifically, consider the Susceptible S ( t ) ,
Infected I ( t ) , Recovered R ( t ) model for the spread of an infectious disease over time t .
The model is defined by a system of three ordinary differential equations
[Equation] [Equation]
where β = 3 , γ = 1 are given constants related to the infectiousness of the disease, N = S
+ I
+ R is the (constant) total population, S (0) = 0 .
99 , I (0) = 0 .
01 , R (0) = 0 , and t ∈ [0 , 10] .
As this system has no analytical solution, we use SciPy’s solve_ivp solver (Virtanen et al., 2020) to obtain ground truth solutions.
We set LHS to be the vector
[Equation]
and RHS =
[0 , 0 , 0] T .
We present the results of training DEQGAN to solve this system of differential equations in Figure 9.
A.2.6. H AMILTONIAN S YSTEM (HAM)
Consider a particle moving through a potential V , the trajectory of which is described by the system of ordinary differential equations
[Equation]
with x (0) = 0 , y (0) = 0 .
3 , p x (0) = 1 , p y (0) = 0 , and t ∈ [0 , 1] .
V x and V y are the x and y derivatives of the potential V , which we construct by summing ten random bivariate Gaussians
[Equation]
where x ( t ) =
[ x ( t ) , y ( t )] T , A = 0 .
1 , σ = 0 . 1 , and each µ i is sampled from [0 , 1] × [0 , 1] uniformly at random.
As before, we use SciPy to obtain ground-truth solutions.
We set LHS to be the vector
[Equation]
and RHS =
[0 , 0 , 0 , 0] T .
We present the results of training DEQGAN to solve this system of differential equations in
Figure 10.
A.2.7. M ODIFIED E INSTEIN ’ S G RAVITY S YSTEM (EIN)
The most challenging system of ODEs we consider comes from Einstein’s theory of general relativity.
Following observations from type Ia supernovae in 1998 (Riess et al., 1998), several cosmological models have been proposed to explain the accelerated expansion of the universe.
Some of these rely on the existence of unobserved forms such as dark energy and dark matter, while others directly modify Einstein’s theory.
Hu-Sawicky f ( R ) gravity is one model that falls under this category.
Chantada et al.
(2022) show how the following system of five ODEs can be derived from the modified field equations implied by this model.
[Equation]
where
[Equation]
The initial conditions are given by
[Equation]
where z 0 = 10 , Ω m, 0 = 0 . 15 , b =
and we solve the system for z ∈ [0 , z 0 ] .
While the physical interpretation of the various parameters is beyond the scope of this paper, we note that Equations 24 and 25 exhibit a high degree of non-linearity.
Ground truth solutions are again obtained using SciPy, and the results obtained by DEQGAN are shown in Figure 11.
A.2.8. P OISSON E QUATION (POS)
Consider the Poisson partial differential equation (PDE) given by
[Equation]
(27) where ( x, y ) ∈ [0 , 1] × [0 , 1] .
The equation is subject to Dirichlet boundary conditions on the edges of the unit square u ( x, y ) (cid:12)(cid:12)(cid:12)(cid:12) x
= =u ( x, y ) (cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
x
[Equation]
y
[Equation]
y
[Equation]
The analytical solution is
[Equation] [Equation]
We use the two-dimensional Dirichlet boundary adjustment formulae provided in Chen et al.
To set up the problem for DEQGAN we let
[Equation]
and RHS = 0 .
We present the results of training DEQGAN on this problem in Figure 12.
A.2.9. H EAT E QUATION (HEA)
We consider the time-dependent heat (diffusion) equation given by
[Equation]
31) where κ = 1 and ( x, t ) ∈ [0 , 1] × [0 , 0 . 2] .
The equation is subject to an initial condition and Dirichlet boundary conditions given by
[Equation]
(32) and has an analytical solution
[Equation]
The results obtained by DEQGAN on this problem are shown in Figure 13.
A.2.10. W AVE E QUATION (WAV)
Consider the time-dependent wave equation given by
[Equation]
34) where c = 1 and ( x, t ) ∈ [0 , 1] ×
This formulation is very similar to the heat equation but involves a second order derivative with respect to time.
We subject the equation to the same initial condition and boundary conditions as 32 but require an added Neumann condition due to the equation’s second time derivative.
[Equation]
This yields the analytical solution
[Equation]
The results of training DEQGAN on this problem are shown in Figure 13.
A.2.11. B UGERS ’ E QUATION (BUR)
Moving to non-linear PDEs, we consider the viscous Burgers’ equation given by
[Equation]
37) where ν = 0 .
001 and ( x, t ) ∈ [ − 5 , 5] ×
To specify the equation, we use the following initial condition and Dirichlet boundary conditions:
[Equation]
As this equation has no analytical solution, we use the fast Fourier transform (FFT) method (Brunton & Kutz, 2019) to obtain ground truth solutions.
The results obtained by DEQGAN are summarized by Figure 15.
As time progresses, we see the formation of a “shock wave” that becomes increasingly steep but remains smooth due to the regularizing diffusive term νu xx .
A.2.12. A LLEN -C AHN E QUATION (ACA)
Finally, we consider the Allen-Cahn PDE, a well-known reaction-diffusion equation given by
[Equation]
where ϵ = 0 .
001 and ( x, t ) ∈ [0 , 2 π ] × [0 , 5] .
We subject the equation to an initial condition and Dirichlet boundary conditions given by
[Equation]
The results are shown in Figure 16.
We see that as time progresses, the sinusoidal initial condition transforms into a square wave, becoming very steep at the turning points of the solution.
A.3. Method Comparison for Other Experiments
Figure 17 visualizes the training results achieved by DEQGAN and the alternative unsupervised neural networks that use
L 2 , L 1 and Huber loss functions for the remaining six problems.
[Figure]
A.4. DEQGAN Hyperparameters
We used Ray Tune (Liaw et al., 2018) to tune DEQGAN hyperparameters for each differential equation.
Tables 4 andsummarize these hyperparameter values for the ODE and PDE problems, respectively.
The experiments and hyperparameter tuning conducted for this research totaled 13,272 hours of compute performed on Intel Cascade Lake CPU cores belonging to an internal cluster.
[Table] [Table]
A.5. Non-GAN Hyperparameter Tuning
Table 6 presents the minimum mean squared errors obtained after tuning hyperparameters for the alternative unsupervised neural network methods that use L 1 , L 2 and Huber loss functions.
[Table]
[Figure]
A.6. Residual Monitoring
Figure 18 shows several examples of how we detect bad training runs by monitoring the variance of the L 1 norm of the
LHS (vector of equation residuals) in the first 25% of training iterations.
Because the LHS may oscillate initially even for successful runs, we use a patience window in the first 15% of iterations.
In all three equations below, we terminate runs if the variance of the residual L 1 norm over 20 iterations exceeds 0.01.
A.7. Ablation Study
To quantify the increased robustness offered by instance noise and residual monitoring, we performed an ablation study comparing the percentage of high MSE ( ≥ 10 − 5 ) runs obtained by 500 randomized DEQGAN runs for the exponential decay equation with and without using these techniques.
Figure 19 plots the results of these 500 DEQGAN experiments with instance noise added.
For each experiment, we uniformly selected a random seed controlling model weight initialization as an integer from the range [0 , 9] , as well as separate learning rates for the discriminator and generator in the range [0 . 01 , 0 . 1] .
We then recorded the final mean squared error after running DEQGAN training for 1000 iterations.
The red lines represent runs which would be terminated early by our residual monitoring method, while the blue lines represent those which would be run to completion.
Figure 19 shows that the large majority of hyperparameter settings tested with the addition of instance noise resulted in low mean squared errors.
Further, residual monitoring was able to detect all runs with MSE ≥ 10 − 5 within the first 25% of training iterations.
Approximately half of the MSE runs in [10 − 8 , 10 − 5 ] would be terminated, while 96% of runs with MSE ≤ 10 − 8 would be run to completion.