DEQGAN: Learning the Loss Function for PINNs with Generative Adversarial Networks
Anonymous Author(s)
AffiliationAddress
Abstract
Solutions to differential equations are of significant scientific and engineering relevance.
Physics-Informed Neural Networks (PINNs) have emerged as a promisingmethod for solving differential equations, but they lack a theoretical justificationfor the use of any particular loss function.
This work presents Differential Equation
GAN (DEQGAN), a novel method for solving differential equations using generative adversarial networks to “learn the loss function” for optimizing the neuralnetwork.
Presenting results on a suite of twelve ordinary and partial differentialequations, including the nonlinear Burgers’, Allen-Cahn, Hamilton, and modified
Einstein’s gravity equations, we show that DEQGAN 1 can obtain multiple ordersof magnitude lower mean squared errors than PINNs that use L 2 , L 1 , and Huberloss functions.
We also show that DEQGAN achieves solution accuracies that arecompetitive with popular numerical methods.
Finally, we present two methods toimprove the robustness of DEQGAN to different hyperparameter settings.
Introduction
In fields such as physics, chemistry, biology, engineering, and economics, differential equations areused to model important and complex phenomena.
While numerical methods for solving differentialequations perform well and the theory for their stability and convergence is well established, therecent success of deep learning [3, 10, 17, 29, 40, 47, 52, 53] has inspired researchers to applyneural networks to solving differential equations, which has given rise to the growing field of
Physics-Informed Neural Networks (PINNs)
In contrast to traditional numerical methods, PINNs: provide solutions that are closed-form [30],suffer less from the “curse of dimensionality” [16, 20, 43, 48], provide a more accurate interpolationscheme [30], and can leverage transfer learning for fast discovery of new solutions [11, 13].
Further,
PINNs do not require an underlying grid and offer a meshless approach to solving differentialequations.
This makes it possible to use trained neural networks, which typically have small memoryfootprints, to generate solutions over arbitrary grids in a single forward pass.
PINNs have been successfully applied to a wide range of differential equations, but lack a theoreticaljustification for the use of a particular loss function from the standpoint of predictive performance.
Indomains outside of differential equations, data following a known noise model (e.g. Gaussian) haveclear justification for fitting models with specific loss functions (e.g. L 2 ).
In the case of deterministicdifferential equations, however, there is no noise model and we lack an equivalent justification.
To address this gap in the theory, we propose generative adversarial networks (GANs) [14] for solvingdifferential equations in a fully unsupervised manner.
Recently, multiple works have shown thatadaptively modifying the PINN loss function throughout training can lead to improved solution accuracies [37, 57].
The discriminator network of our GAN-based method, however, can be thoughtof as “learning the loss function” for optimizing the generator, thereby eliminating the need for anexplicit loss function and providing even greater flexibility than an adaptive loss.
Beyond the contextof differential equations, it has also been shown that where classical loss functions struggle to capturecomplex spatio-temporal dependencies, GANs may be an effective alternative [32, 26, 31].
Our contributions in this work are summarized as follows:
• We present Differential Equation GAN (DEQGAN), a novel method for solving differentialequations in a fully unsupervised manner using generative adversarial networks.
• We highlight the advantage of “learning the loss function” with a GAN rather than using apre-specified loss function by showing that PINNs trained using L 2 , L 1 , and Huber losses havevariable performance and fail to solve the modified Einstein’s gravity equations [7].• We present results on a suite of twelve ordinary differential equations (ODEs) and partial differential equations (PDEs), including highly nonlinear problems, showing that our method producessolutions with multiple orders of magnitude lower mean squared errors than PINNs
that useL 2 , L 1 , and Huber loss functions.•
We show that DEQGAN achieves solution accuracies that are competitive with popular numericalmethods, including the fourth-order Runge-Kutta and second-order finite difference
methods.•
We present two techniques to improve the training stability of DEQGAN that are applicable to
other GAN-based methods and PINN approaches to solving differential equations.
Related Work
A variety of neural network methods have been developed for solving differential equations.
Some ofthese are supervised and learn the dynamics of real-world systems from data [4, 9, 15, 44].
Others aresemi-supervised, learning general solutions to a differential equation and extracting a best fit solutionbased on observational data [41].
Our work falls under the category of unsupervised neural networkmethods, which are trained in a data-free manner that depends solely on the equation residuals.
Unsupervised neural networks have been applied to a wide range of ODEs [13, 30, 34, 36] and PDEs[20, 43, 48, 50], primarily use feed-forward architectures, and require the specification of a particularloss function computed over the equation residuals.
Goodfellow et al.
[14] introduced the idea of learning generative models with neural networks andan adversarial training algorithm, called generative adversarial networks (GANs).
To solve issuesof GAN training instability, Arjovsky et al.
[2] introduced a formulation of GANs based on the
Wasserstein distance, and Gulrajani et al.
[18] added a gradient penalty to approximately enforce a
Lipschitz constraint on the discriminator.
Miyato et al.
[39] introduced an alternative method forenforcing the Lipschitz constraint with a spectral normalization technique that outperforms the formermethod on some problems.
Further work has applied GANs to differential equations with solution data used for supervision.
Yang et al.
[56] apply GANs to stochastic differential equations by using “snapshots" of ground-truthdata for semi-supervised training.
A project by students at Stanford [51] employed GANs to perform“turbulence enrichment" of solution data in a manner akin to that of super-resolution for imagesproposed by Ledig et al.
Our work distinguishes itself from other GAN-based approaches forsolving differential equations by being fully unsupervised , and removing the dependence on usingsupervised training data (i.e. solutions of the equation).
Background3.1 Unsupervised Neural Networks for Differential Equations
Early work by Dissanayake & Phan-Thien [12] proposed solving initial value problems in an unsupervised manner with neural networks.
In this work, we extend their approach to handle spatial domainsand multidimensional problems.
In particular, we consider general differential equations of the form
[Equation]
where Ψ( t, x ) is the desired solution, d Ψ /dt and d 2 Ψ
/dt 2 represent the first and second timederivatives, ∆Ψ and ∆ 2 Ψ are the first and second spatial derivatives, and the system is subject tocertain initial and boundary conditions.
The learning problem can then be formulated as minimizingthe sum of squared residuals (i.e., the squared L 2 loss) of the above equation
[Equation]
where Ψ θ is a neural network parameterized by θ , D is the domain of the problem, and derivativesare computed with automatic differentiation.
This allows backpropagation [22] to be used to trainthe neural network to satisfy the differential equation.
We apply this formalism to both initial andboundary value problems, including multidimensional problems, as detailed in Appendix A.2.
3.2 Generative Adversarial Networks
Generative adversarial networks (GANs) [14] are generative models that use two neural networks toinduce a generative distribution p ( x ) of the data by formulating the inference problem as a two-player,zero-sum game.
The generative model first samples a latent random variable z ∼ N (0 , 1) , which is used as input intothe generator G (e.g., a neural network).
A discriminator D is trained to classify whether its inputwas sampled from the generator (i.e. “fake") or from a reference data set (i.e. “real").
Informally, the process of training GANs proceeds by optimizing a minimax objective over thegenerator and discriminator such that the generator attempts to trick the discriminator to classify“fake" samples as “real".
Formally, one optimizes
[Equation]
where x ∼ p data ( x ) denotes samples from the empirical data distribution, and p z ∼ N (0 , 1) samplesin latent space [14].
In practice, the optimization alternates between gradient ascent and descent stepsfor D and G, respectively.
Further details on training and architecture are provided in Appendix A.4.
3.3 Guaranteeing Initial & Boundary Conditions
Lagaris et al.
[30] showed that it is possible to exactly satisfy initial and boundary conditions byadjusting the output of the neural network.
For example, consider adjusting the neural network outputΨ θ ( t, x ) to satisfy the initial condition Ψ θ ( t, x ) (cid:12)(cid:12) t
= t 0 = x 0 .
We can apply the re-parameterization
[Equation]
which exactly satisfies the initial condition.
Mattheakis et al.
[36] proposed an augmented reparameterization
[Equation]
that further improved training convergence.
Intuitively, Equation 5 adjusts the output of the neuralnetwork Ψ θ ( t, x ) to be exactly x 0 when t
= t 0 , and decays this constraint exponentially in t .
Chenet al.
[8] provide re-parameterizations to satisfy a range of other conditions, including Dirichlet and
Neumann boundary conditions, which we employ in our experiments and detail in Appendix A.2.
Differential Equation GAN
In this section, we present our method, Differential Equation GAN (DEQGAN), which trains a GANto solve differential equations in a fully unsupervised manner.
To do this, we rearrange the differentialequation so that the left-hand side ( LHS ) contains all the terms which depend on the generator (e.g.Ψ , d Ψ /dt , ∆Ψ , etc g
During training, we sample points from the domain ( t, x ) ∼ D and use them as input to a generator
G ( x ) , which produces candidate solutions Ψ θ .
We sample points from a noisy grid that spans D , which we found reduced interpolation error in comparison to sampling points from a fixed grid.
Wethen adjust Ψ θ for initial or boundary conditions to obtain the re-parameterized output ˜Ψ θ , constructthe LHS from the differential equation F using automatic differentiation
[Equation]
and set RHS to its appropriate value (in our examples, RHS = 0 ).
Training proceeds in a mannersimilar to traditional GANs.
We update the weights of the generator G and the discriminator Daccording to the gradients
[Equation]
[Equation]
(8) where LHS ( i ) is the output of G (cid:0) x
( i ) (cid:1) after adjusting for initial or boundary conditions and constructing the LHS from F .
Note that we perform stochastic gradient descent for G (gradient steps∝ − g G ), and stochastic gradient ascent for D (gradient steps ∝ g D ).
We provide a schematicrepresentation of DEQGAN in Figure 1 and detail the training steps in Algorithm 1.
Algorithm
Input: Differential equation F , generator G ( · ; θ g ) , discriminator D ( · ; θ d ) , grid x of m points with spacing ∆ x , perturbation precision τ , re-parameterization function Φ , total steps N , learning rates η G , η D , Adam optimizer [27] parameters β G 1 , β G 2 , β D 1 , β Dfor i = 1 to N do for j = 1 to m do
Perturb j -th point in mesh x ( j ) s
= x ( j ) + ϵ, ϵ ∼ N (0 , ∆ xτ )
Forward pass Ψ θ =
G ( x ( j ) s )
Analytic re-parameterization ˜Ψ
θ = Φ(Ψ θ )
Compute LHS ( j ) = F (cid:16) t, x , ˜Ψ θ ( t, x ) , d ˜Ψ θ dt , d 2 ˜Ψ θ dt 2 , . . .
, ∆˜Ψ θ , ∆ 2 ˜Ψ θ , . . .
(cid:17)
Set RHS ( j ) =end for
Compute gradients g G , g D (Equation 7 & 8)
Update generator θ g ← Adam ( θ g , − g G , η G , β G 1 , β G 2 )
Update discriminator θ d ← Adam ( θ d , g D , η D , β D 1 , β D 2 ) end for
Output: G
Informally, our algorithm trains a GAN by setting the “fake” component to be the LHS (in ourformulation, the residuals of the equation) and the “real” component to be the RHS of the equation.
This results in a GAN that learns to produce solutions that make LHS indistinguishable from RHS ,thereby approximately solving the differential equation.
4.1 Instance Noise
While GANs have achieved state of the art results on a wide range of generative modeling tasks, theyare often difficult to train.
As a result, much recent work on GANs has been dedicated to improvingtheir sensitivity to hyperparameters and training stability [1, 2, 5, 18, 25, 28, 38, 39, 46, 49].
In ourexperiments, we found that DEQGAN could also be sensitive to hyperparameters, such as the Adamoptimizer parameters shown in Algorithm 1.
Sønderby et al.
[49] note that the convergence of GANs relies on the existence of a unique optimaldiscriminator that separates the distribution of “fake” samples p fake produced by the generator, andthe distribution of the “real” data p data .
In practice, however, there may be many near-optimaldiscriminators that pass very different gradients to the generator, depending on their initialization.
Arjovsky & Bottou [1] proved that this problem will arise when there is insufficient overlap betweenthe supports of p fake and p data .
In the DEQGAN training algorithm, setting RHS = 0 constrains p datato the Dirac delta function δ (0) , and therefore the distribution of “real” data to a zero-dimensionalmanifold.
This makes it unlikely that p fake and p data will share support in a high-dimensional space.
The solution proposed by [1, 49] is to add “instance noise” to p fake and p data to encourage their overlap.
This amounts to adding noise to the LHS and the RHS , respectively, at each iteration of Algorithm1.
Because this makes the discriminator’s job more difficult, we add Gaussian noise with standarddeviation equal to the difference between the generator and discriminator losses, L g and L d , i.e.
[Equation]
As the generator and discriminator reach equilibrium, Equation 9 will naturally converge to zero.
Weuse the ReLU function because L d > L g indicates that the discriminator is generally performingworse than the generator, suggesting that additional noise should not be used.
In Section 5.2, weconduct an ablation study and find that this improves the ability of DEQGAN to produce accuratesolutions across a range of hyperparameter settings.
4.2 Residual Monitoring
One of the attractive properties of Algorithm 1 is that the “fake” LHS vector of equation residualsgives a direct measure of solution quality at each training iteration.
We observe that when DEQGANtraining becomes unstable, the LHS tends to oscillate wildly, while it decreases steadily throughouttraining for successful runs.
By monitoring the L 1 norm of the LHS in the first 25% of trainingiterations, we are able to easily detect and terminate poor-performing runs if the variance of thesevalues exceeds some threshold.
We provide further details on this method in Appendix A.7 andexperimentally demonstrate that it is able to distinguish between DEQGAN runs that end in high andlow mean squared errors in Section 5.2.
[-( z ) = 
-]
[-+
-]
Experiments
We conducted experiments on a suite of twelve differential equations (Table 1), including highlynonlinear PDEs and systems of ODEs, comparing DEQGAN to classical unsupervised PINNs thatuse (squared) L 2 , L 1 , and Huber [24] loss functions.
We also report results obtained by the fourthorder Runge-Kutta (RK4) and second-order finite difference (FD) numerical methods for initialand boundary value problems, respectively.
The numerical solutions were computed over meshescontaining the same number of points that were used to train the neural network methods.
Detailsfor each experiment, including exact problem specifications and hyperparameters, are provided in
Appendix A.2 and A.5.
5.1 DEQGAN vs. Classical PINNs
We report the mean squared error of the solution obtained by each method, computed againstknown solutions obtained either analytically or with high-quality numerical solvers [6, 54].
Weadded residual connections between neighboring layers of all models, applied spectral normalization
[Table]
No
[Table]
PDE
[Table]
No
[Table] to the discriminator [Table] added instance noise to the p fake and p real , and used residual monitoring toterminate poor-performing runs in the first 25% of training iterations.
Results were obtained withhyperparameters tuned for DEQGAN.
In Appendix A.6, we tuned each classical PINN method forcomparison, but did not observe a significant difference.
Table 2 reports the lowest mean squared error obtained by each method across ten different modelweight initializations.
We see that DEQGAN obtains lower mean squared errors than classical
PINNs that use L 2 , L 1 , and Huber loss functions for all twelve problems, often by several orders ofmagnitude.
DEQGAN also achieves solution accuracies that are competitive with the RK4 and FDnumerical methods. [Figure]
We perform ten randomized trials and plot the median
We smooth the values using a simple moving average with window size 50 .
Figure 2 plots the mean squared error vs. training iteration for six challenging equations and highlightsmultiple advantages of using DEQGAN over a pre-specified loss function (equivalent plots for theother six problems are provided in Appendix A.3).
In particular, there is considerable variation inthe quality of the solutions obtained by the classical PINNs.
For example, while Huber performsbetter than L 2 on the Allen-Cahn PDE, it is outperformed by L 2 on the wave equation.
Furthermore,
Figure 2f shows that the L 2 , L 1 and Huber losses all fail to converge to an accurate solution to themodified Einstein’s gravity equations.
Although this system has previously been solved using PINNs,the networks relied on a custom loss function that incorporated equation-specific parameters [7].
DEQGAN, however, is able to automatically learn a loss function that optimizes the generator toproduce accurate solutions.
DEQGAN solutions to four example equations are visualized in Figure4, which shows that the ODE solutions are indistinguishable from those obtained using a numericalintegrator. Similar plots for the other experiments are provided in Appendix A.2.
5.2 DEQGAN Training Stability: Ablation Study
In our experiments, we used instance noise to adaptively improve the training convergence of
DEQGAN and employed residual monitoring to terminate poor-performing runs early.
To quantify the increased robustness offered by these techniques, we performed an ablation study comparingthe percentage of high MSE ( ≥ 10 − 5 ) runs obtained by 500 randomized DEQGAN runs on theexponential decay equation.
Figure 3 plots the results of these 500 DEQGAN experiments with instance noise added. For eachexperiment, we uniformly selected a random seed controlling model weight initialization as an integerfrom the range [0 , 9] , as well as separate learning rates for the discriminator and generator in therange [0 . 01 , 0 . 1] . We then recorded the final mean squared error after running DEQGAN trainingfor 1000 iterations. The red lines represent runs which would be terminated early by our residualmonitoring method, while the blue lines represent those which would be run to completion.
We seethat the large majority of hyperparameter settings tested with the addition of instance noise resulted inlow mean squared errors.
Further, residual monitoring was able to detect all runs with MSE ≥ 10 − 5 .
Approximately half of the MSE runs in [10 − 8 , 10 − 5 ] would be terminated, while 96% of runs with
MSE ≤ 10 − 8 would be run to completion.
[Table]
Table 3 compares the percentage of high MSE runs with and without instance noise and residualmonitoring.
We see that adding instance noise decreased the percentage of runs with high MSE andthat residual monitoring is highly effective at filtering out poor performing runs.
When used together,these techniques eliminated all runs with MSE ≥ 10 − 5 .
These results agree with previous works,which have found that instance noise can improve the convergence of other GAN training algorithms[1, 49].
Further, they suggest that residual monitoring provides a useful performance metric thatcould
[Figure]
Conclusion
PINNs offer a promising approach to solving differential equations and to applying deep learningmethods to challenging problems in science and engineering.
Classical PINNs, however, lack atheoretical justification for the use of any particular loss function.
In this work, we presented
Differential Equation GAN (DEQGAN), a novel method that leverages GAN-based adversarialtraining to “learn” the loss function for solving differential equations with PINNs.
We demonstratedthe advantage of this approach in comparison to using classical PINNs with pre-specified lossfunctions, which showed varied performance and failed to converge to an accurate solution to themodified Einstein’s gravity equations.
In general, we demonstrated that our method can obtainmultiple orders of magnitude lower mean squared errors than PINNs that use L 2 , L 1 and Huberloss functions, including on highly nonlinear PDEs and systems of ODEs.
Further, we showed that
DEQGAN achieves solution accuracies that are competitive with the fourth-order Runge Kutta andsecond-order finite difference numerical methods.
Finally, we found that instance noise improvedtraining stability and that residual monitoring provides a useful performance metric for PINNs.
Whilethe equation residuals are a good measure of solution quality, PINNs lack the error bounds enjoyedby numerical methods.
Formalizing these bounds is an interesting avenue for future work and wouldenable PINNs to be more safely deployed in real-world applications.
Further, while our resultsevidence the advantage of “learning the loss function” with a GAN, understanding exactly what thediscriminator learns is an open problem.
Post-hoc explainability methods, for example, might provideuseful tools for characterizing the differences between classical losses and the loss functions learnedby DEQGAN, which could deepen our understanding of PINN optimization more generally.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] Our claims are evidenced by the experimental results in Section 5.
(b) Did you describe the limitations of your work?
[Yes] We discussed limitations anddirections for future work in Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] While our
research is focused on the study of differential equations and does not hold particularly
poignant ethical consequences, we discussed future research directions for ensuring that our method can safely be deployed in real-world applications in Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See the footnote
on page 1 (link is currently hidden to preserve anonymity).
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix A.2 and A.4.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We conducted an ablation study that includes a sensitivity
analysis of our method.
See Appendix A.7.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]See the footnote on page 1 (link is currently hidden to preserve anonymity).
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants
and
screenshots if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A AppendixA.1 Classical Loss Functions
A plot of the various classical loss functions is provided in Figure 5.
A.2 Description of ExperimentsA.2.1 Exponential Decay (EXP)
Consider a model for population decay x ( t ) given by the exponential differential equation˙ x ( t ) + x ( t ) = 0 , (10) with x (0) = 1 and t ∈ [0 , 10] .
The ground truth solution x ( t ) =
e − t can be obtained analytically,which we use to calculate the mean squared error of the predicted solution.
To set up the problem for DEQGAN, we define LHS =
˙
x
+ x and RHS = 0 .
Figure 6 presents theresults from training DEQGAN on this equation.
A.2.2 Simple Harmonic Oscillator (SHO)
Consider the motion of an oscillating body x ( t ) , which can be modeled by the simple harmonicoscillator differential equation¨ x ( t ) + x ( t ) = 0 , (11) with x (0) = 0 , ˙ x (0) = 1 , and t ∈ [0 , 2 π ] .
This differential equation can be solved analytically andhas an exact solution x ( t )
= sin t .
Here we set LHS =
¨ x + x and RHS = 0 .
Figure 7 plots the results of training DEQGAN on thisproblem.
A.2.3 Damped Nonlinear Oscillator (NLO)
Further increasing the complexity of the differential equations being considered, consider a lessidealized oscillating body subject to additional forces, whose motion x ( t ) we can described by thenonlinear oscillator differential equation
[Equation]
(12) with β = 0 . 1 , ω = 1 , ϕ = 1 , ϵ = 0 .
1 , x (0) = 0 , ˙ x (0)
= 0 .
5 , and t ∈ [0 , 4 π ] .
This equation does notadmit an analytical solution.
Instead, we use the high-quality solver provided by SciPy’s solve_ivp[54].
We set LHS =
¨ x + 2 β ˙ x + ω 2 x + ϕx 2 + ϵx 3 = 0 and RHS = 0 .
Figure 8 plots the results obtainedfrom training DEQGAN on this equation.
A.2.4 Coupled Oscillators (COO)
Consider the system of ordinary differential equations given by
[Equation]
(13) with x (0) = 1 , y (0) = 0 , and t ∈ [0 , 2 π ] .
This equation has an exact analytical solution given by 
[Equation]
cos
[Equation]
(cid:19) (14)
Here we set
[Equation]
(15) and RHS =
[0 , 0] T .
Figure 9 plots the result of training DEQGAN on this problem.
A.2.5 SIR Epidemiological Model (SIR)
Given the ongoing pandemic of novel coronavirus (COVID-19) [55], we consider an epidemiologicalmodel of infectious disease spread given by a system of ordinary differential equations.
Specifically,consider the Susceptible S ( t ) , Infected I ( t ) , Recovered R ( t ) model for the spread of an infectiousdisease over time t .
The model is defined by a system of three ordinary differential equations
[Equation]
where β = 3 , γ = 1 are given constants related to the infectiousness of the disease, N = S
+ I
+ R isthe (constant) total population, S (0)
= 0 .
99 , I (0) = 0 .
01 , R (0) = 0 , and t ∈ [0 , 10] .
As this systemhas no analytical solution, we use SciPy’s solve_ivp solver [54] to obtain ground truth solutions.
We set LHS to be the vector
[Equation]
=
[Equation]
(17) and RHS =
[0 , 0 , 0] T .
We present the results of training DEQGAN to solve this system of differentialequations in Figure 10.
A.2.6 Hamiltonian System (HAM)
Consider a particle moving through a potential V , the trajectory of which is described by the systemof ordinary differential equations
[Equation]
18) with x (0) = 0 , y (0) = 0 .
3 , p x (0) = 1 , p y (0) = 0 , and t ∈ [0 , 1] .
V x and V y are the x and yderivatives of the potential V , which we construct by summing ten random bivariate Gaussians
[Equation]
(cid:88)
[Equation]
where x ( t ) =
[ x ( t ) , y ( t )] T , A = 0 .
1 , σ = 0 . 1 , and each µ i is sampled from [0 , 1] × [0 , 1] uniformlyat random.
As before, we use SciPy to obtain ground-truth solutions.
We set LHS to be the vector
[Equation] [Equation]
and RHS =
[0 , 0 , 0 , 0] T .
We present the results of training DEQGAN to solve this system ofdifferential equations in Figure 11.
A
[Figure]
2.7 Modified Einstein’s Gravity System
[Figure] [Figure]
The most challenging system of ODEs we consider comes from Einstein’s theory of general relativity.
Following observations from type Ia supernovae in 1998 [45], several cosmological models have beenproposed to explain the accelerated expansion of the universe.
Some of these rely on the existenceof unobserved forms such as dark energy and dark matter, while others directly modify Einstein’stheory.
Hu-Sawicky f ( R ) gravity is one model that falls under this category.
Chantada et al.
[7] show howthe following system of five ODEs can be derived from the modified field equations implied by thismodel. 
[Equation]
where
[Equation]
The initial conditions are given by
[Equation]
where z 0 = 10 , Ω m, 0 = 0 . 15 , b =
and we solve the system for z ∈ [0 , z 0 ] .
While the physicalinterpretation of the various parameters is beyond the scope of this paper, we note that Equations 21and 22 exhibit a high degree of non-linearity.
Ground truth solutions are again obtained using SciPy,and the results obtained by DEQGAN are shown in Figure 12.
For ease of visualization, we plot the predictions and residuals for each equation separately. A.
2.8 Poisson Equation (POS)
Consider the Poisson partial differential equation (PDE) given by
[Equation]
[Equation]
where ( x, y ) ∈ [0 , 1] × [0 , 1] .
The equation is subject to Dirichlet boundary conditions on the edgesof the unit square
[Equation]
The analytical solution is
[Equation]
We use the two-dimensional Dirichlet boundary adjustment formulae provided in Chen et al.
Toset up the problem for DEQGAN we let
[Equation]
[Equation]
and RHS = 0 .
We present the results of training DEQGAN on this problem in Figure 13.
A.2.9 Heat Equation (HEA)
We consider the time-dependent heat (diffusion) equation given by
[Equation]
(28) where κ = 1 and ( x, t ) ∈ [0 , 1] ×
The equation is subject to an initial condition and Dirichletboundary conditions given by
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12) (cid:12) x
[Equation]
29) and has an solution
[Equation]
The results obtained by DEQGAN on this problem are shown in Figure 14.
A.2.10 Wave Equation (WAV)
Consider the time-dependent wave equation given by
[Equation]
(31) where c = 1 and ( x, t ) ∈ [0 , 1] × [0 , 1] .
This formulation is very similar to the heat equationbut involves a second order derivative with respect to time.
We subject the equation to the sameinitial condition and boundary conditions as 29 but require an added Neumann condition due to theequation’s second time derivative.
[Equation]
This yields the analytical solution
[Equation]
The results of training DEQGAN on this problem are shown in Figure 14.
A.2. 11 Bugers’ Equation BUR)
Moving to non-linear PDEs, we consider the viscous Burgers’ equation given by
[Equation]
where ν = 0 .
001 and ( x, t ) ∈ [ − 5 , 5] ×
To specify the equation, we use the following initialcondition and Dirichlet boundary conditions:
[Equation]
As this equation has no analytical solution, we use the fast Fourier transform (FFT) method [6] toobtain ground truth solutions.
The results obtained by DEQGAN are summarized by Figure 16.
Astime progresses, we see the formation of a “shock wave” that becomes increasingly steep but remainssmooth due to the regularizing diffusive term νu xx .
A.2.12 Allen-Cahn Equation (ACA)
Finally, we consider the Allen-Cahn PDE, a well-known reaction-diffusion equation given by
[Equation]
(36) where ϵ
= 0 .
001 and ( x, t ) ∈ [0 , 2 π ] × [0 , 5] .
We subject the equation to an initial condition and
Dirichlet boundary conditions given byu ( x
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12) t = =4 sin( x ) u ( x, y ) (cid:12)(cid:12)(cid:12)(cid:12) x
= =u (
[Equation]
y ) (cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
The results are shown in Figure 17.
We see that as time progresses, the sinusoidal initial conditiontransforms into a square wave, becoming very steep at the turning points of the solution.
A.3 Method Comparison for Other Experiments
Figure 18 visualizes the training results achieved by DEQGAN and the alternative unsupervisedneural networks that use L 2 , L 1 and Huber loss functions for the remaining six problems.
[Figure]
A.4 DEQGAN Training and ArchitectureA.4.1 Two Time-Scale Update Rule
Heusel et al.
[23] proposed the two time-scale update rule (TTUR) for training GANs, a method inwhich the discriminator and generator are trained with separate learning rates.
They showed that theirmethod led to improved performance and proved that, in some cases, TTUR ensures convergence toa stable local Nash equilibrium.
One intuition for TTUR comes from the potentially different losssurfaces of the discriminator and generator.
Allowing learning rates to be tuned to a particular losssurface can enable more efficient gradient-based optimization.
We make use of TTUR throughoutthis paper as an instrumental lever when tuning GANs to reach desired performance.
A.4.
Spectral Normalization
Proposed by Miyato et al.
[39], Spectrally Normalized GAN (SN-GAN) is a method for controlling exploding discriminator gradients when optimizing Equation 3 that leverages a novel weightnormalization technique.
The key idea is to control the Lipschitz constant of the discriminator byconstraining the spectral norm of each layer in the discriminator.
Specifically, the authors proposedividing the weight matrices W i of each layer i by their spectral norm σ ( W i )
W SN,i =
W i σ ( W i ) , (38) where
[Equation]
(39) and h i denotes the input to layer i .
The authors prove that this normalization technique bounds
the
Lipschitz constant of the discriminator above by 1 , thus strictly enforcing the 1 -Lipshcitz constrainton the discriminator.
In our experiments, adopting the SN-GAN formulation led to even betterperformance than WGAN-GP [2, 18].
A.4.3 Residual Connections
He et al.
[21] showed that the addition of residual connections improves deep neural networktraining.
We employ residual connections in our networks, as they allow gradients to flow more easilythrough the models and thereby reduce numerical instability.
Residual connections augment a typicalactivation with the identity operation.
[Equation]
where F is the activation function, x is the input to the unit, W i are the weights and y is the outputof the unit.
This acts as a “skip connection", allowing inputs and gradients to forego the nonlinearcomponent.
A.5 DEQGAN Hyperparameters
We used Ray Tune [33] to tune DEQGAN hyperparameters for each differential equation.
Tables 4and 5 summarize these hyperparameter values for the ODE and PDE problems, respectively.
Theexperiments and hyperparameter tuning conducted for this research totaled 13,272 hours of computeperformed on Intel Cascade Lake CPU cores belonging to an internal cluster
[Table]
[Table]
[Table]
A.6 Non-GAN Hyperparameter Tuning
Table 6 presents the minimum mean squared errors obtained after tuning hyperparameters for thealternative unsupervised neural network methods that use L 1 , L 2 and Huber loss functions
[Table] [Table]
A.7 Residual Monitoring
[Figure]
Figure 19 shows several examples of how we detect bad training runs by monitoring the variance ofthe L 1 norm of the LHS (vector of equation residuals) in the first 25% of training iterations.
Becausethe LHS may oscillate initially even for successful runs, we use a patience window in the first 15%of iterations.
In all three equations below, we terminate runs if the variance of the residual L 1 normover 20 iterations exceeds 0.01.