Adversarial Sampling for Solving Differential Equations with Neural Networks
Abstract
Neural network-based methods for solving differential equations have been gaining traction.
They work by improving the differential equation residuals of a neural network on a sample of points in each iteration.
However, most of them employ standard sampling schemes like uniform or perturbing equally spaced points.
We present a novel sampling scheme which samples points adversarially to maximize the loss of the current solution estimate.
A sampler architecture is described along with the loss terms used for training.
Finally, we demonstrate that this scheme outperforms pre-existing schemes by comparing both on a number of problems
Introduction
Differential equations are ubiquitous in all engineering and science disciplines.
Hence, substantial research goes into designing novel methods and improving pre-existing methods to solve differential equations.
With the rise of deep learning, one approach which has gained traction is to use neural networks to solve these equations in an unsupervised manner.
This has been used to solve a variety of differential equations such as ordinary differential equations (ODEs) [1, 2, 3, 4], partial differential equations [4, 5, 6, 7] (PDEs), and eigenvalue problems [8].
This holds certain advantages as opposed to numerical methods like ﬁnite difference such as (a) instead of obtaining solution values at discretized points, we get a closed and differentiable solution function [1], (b) it is more effective in solving high dimensional PDEs by faring better against the "curse of dimensionality" [5], (c) numerical errors are not accumulated in each iteration [4], (d) initial and boundary conditions are satisﬁed by construction [1, 2].
One method employed is to use a neural network (possibly in a reparametrized form to satisfy the initial/boundary conditions) to represent a candidate solution and minimize the loss corresponding to the differential equation over a number of iterations [1, 7].
This is done by sampling points over the domain using some predeﬁned scheme (for example, uniform/equally spaced) and minimizing the loss function on these points in each iteration.
However, such sampling schemes are either blind to the equation being solved or require one to carefully design custom sampling schemes to improve efﬁciency.
We propose a novel method which samples points in an adversarial manner by ﬁnding points in the domain at which the current loss is high.
This is followed by the solver minimizing the loss at these very points by updating the weights of the solver network
We demonstrate the efﬁciency of this scheme on a variety of problems.
Methodology
We ﬁrst describe the vanilla method and then talk about adversarial sampling.
2.1 Vanilla Approach
Although the framework described is readily extendable to harder ODEs/PDEs (as demonstrated in the results 2) , in this section we consider a ﬁrst-order ordinary differential equation F to be solved for x ∈ [ a, b ] :
[Equation]
given the initial condition y ( x 0 ) =
y 0 .
Let the function learned by the neural network be y N .
The method employed to exactly impose the initial conditions is to reparameterize y N to get ˆ y as [1, 7]:
[Equation]
For a ﬁxed ˆ y , we may exactly calculate the (partial) derivatives using automatic differentiation.
In further text, we write F ( x, ˆ y, ˆ y (cid:48) ) as just F ( x, ˆ y ) .
Hence for a given ˆ y , we may express the loss function at a point x as:
[Equation]
Hence, to solve the equation, we must minimize the expected loss :
[Equation]
At each iteration, we minimize the empirical loss at points x = { x 1 , x 2 , . . .
, x n } : [Equation]
The routine way to sample points x 1 , x 2 , . . .
, x n is to use some pre-speciﬁed scheme like uniform/equally-spaced sampling and to update the parameters of our neural network y N in each iteration to improve our estimate ˆ y .
2.2 Adversarial Sampling
Using a predeﬁned sampling scheme has some clear drawbacks.
First, it is agnostic to the equation being solved as well as our current estimate ˆ y .
An efﬁcient sampling scheme is expected to take into account where our current estimate is incorrect and sample extra points from that region to converge faster.
When ﬁtting the 1D exponential decay equation ( u x = − λu ), for example, sampling equally both before and after the elbow would be inefﬁcient.
This is especially valid in cases where the number of sampled points is limited and insufﬁcient to cover the entire space (increasingly valid in higher dimensions).
We present a sampling scheme that is dependent on the current estimate ˆ y .
This is done by using a neural network to represent a variable sampling distribution implicitly similar to the generator network of a GAN [9].
In each iteration, the sampler is trained to produce points which maximize the loss of the solver (and a secondary loss).
Thus, it competes with the solver whose weights are updated to minimize the loss at these very points.
2.3 Architecture of the Adversarial Sampler
We use a GAN-like [9] generator to produce samples.
It is a fully connected neural network that uses a noise vector z to produce a vector of sampled points x .
We use tanh activation in the last layer and rescale the output of the sampler to ensure the points lie in the speciﬁed input domain.
2.4 Regularization
It is observed that if the sampler is purely optimized with the objective of maximizing ˆ L (ˆ y, x ) (residual loss corresponding to the DE at samples x ), it tends to collapse all samples to one single point of high loss.
This is akin to the phenomenon of mode collapse in GANs [9].
Therefore it is essential to regularize the sampler by penalizing distributions with low entropy.
For this, we use an additional loss term D k .
Given points { x 1 , x 2 , . . .
, x n } , we deﬁne d k ( x i ) to be the sum of distances of x i from its k nearest neighbors.
Hence for x = { x 1 , x 2 , . . .
, x n } the secondary loss term is deﬁned as follows:
[Equation]
This is implemented using a kd-tree [10], which is queried for each point to get its k -nearest neighbors in order to calculate D k .
Other methods of enforcing entropy constraints were experimented with but this proved to be faster and more effective in practice.
2.5 Training iteration
In each training iteration, we ﬁrst sample points from the sampler to get x = { x 1 , x 2 , . . .
, x n } .
This leads us to two losses, namely ˆ L solver and ˆ L sampler .
[Equation]
In one single training iteration, we ﬁrst calculate ˆ L solver and update the parameters of y N .
Next we evaluate ˆ L sampler and update the parameters of the sampler.
Experiments
We demonstrate the results with a number of differential equations as shown in Table 1.
Table 2 contains the results, where the comparison is done by ﬁrst setting a target loss and maximum iterations.
The maximum iterations are chosen in such a way so that both sampling schemes take equal time to arrive at them.
Then by ﬁxing the number of points, we compare the average time taken to arrive at the target loss or maximum iterations(whichever comes ﬁrst).
We also provide the average loss achieved, which might be higher than the target loss in case the maximum iterations are reached without achieving the target loss. We evaluate the mean-squared error loss w.r.t.
the analytic solution in the case of ODEs where this is available, and in the case of PDEs we use the validation loss which is the residual loss evaluated on a grid of (32 , 32) equally spaced points.
All average data is reported over 10 trials.
[Table]
[Table]
Exponential decay ( γ = − 5 ) u x [Equation] e
{+[Equation] [0 , 30] u | x =
 = 0 .+} Logistic Equation ( γ = − 1 , M = 1 ) u x = γu ( M − u ) [0 , 10] u | x =
= 0 .
Radial part of H-atom ( n = 1 , l = 0 )
[Equation]
[-l +1)2 x 2 (cid:1) u [0 , 30] u | x =
-][Equation]
[Equation]
Radial part of H-atom ( n = 2 , l = 0 )
[-u xx = 2 (cid:0) 12 n 2 − 1 x + l l +1)2 x 2 (cid:1) u [0 , 30] u | x =
-][Equation]
[Equation]
Laplace Equation[-u xx + u yy =[0 , 1] × [0 , 1] u | x =
-]
[Equation]
[-| x =1 = 0 u | y =
 = 0 , u | y = =-]problems.
[Table]
Conclusion
In this paper
[Table]
we show that adversarial sampling works in practice.
It is able to improve the efﬁciency of ﬁtting and leads to lower loss.
It is also observed that in order to avoid collapse, we must regularize by penalizing sampling distributions with low entropy.
Future work would involve exploring this technique to solve high-dimensional PDE(s) where it holds more promise
[Table]
Acknowledgments and Disclosure of Funding
The authors would like to thank the anonymous reviewers for their valuable feedback on this manuscript.