N EURAL N ETWORK D IFFERENTIAL E QUATION S OLVERS ALLOW UNSUPERVISED ERROR ANALYSIS AND CORRECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural Network Differential Equation (NN DE) solvers have surged in popularity due to a combination of factors: computational advances making their optimization more tractable, their capacity to handle high dimensional problems, easy interpretability, etc.
However, most NN DE solvers suffer from a fundamental limitation: their loss functions are not explicitly dependent on the errors associated with the solution estimates.
As such, validation and error estimation usually requires knowledge of the true solution.
Indeed, when the true solution is unknown, we are often reduced to simply hoping that a “ low enough ” loss implies “ small enough ” errors, since explicit relationships between the two are not available.
In this work, we describe a general strategy for efficiently constructing error estimates and corrections for Neural Network Differential Equation solvers.
Our methods do not require a priori knowledge of the true solutions and obtain explicit relationships between loss functions and the errors.
In turn, these explicit relationships allow for the unsupervised estimation and correction of the model errors.
I NTRODUCTION
Deep learning has heralded new methods for many scientific disciplines – the field of numerical methods for differential equations has been no exception (1; 2; 3; 4).
Deep neural network based differential equations (NN DE) solvers have been proposed under a variety of different names (PINNs (2), DGM (3), etc) – each catering to various classes of problems.
However, they all share certain common features: the use of a known differential equation (DE) as the central component of an appropriate loss function, the use of existing knowledge (boundary conditions, experimental/synthetic data, etc) to constrain the search for solutions, randomized optimization methods that sample from the domain of interest at a requisite resolution, etc.
We investigate another common facet of many NN DE solvers: the lack of unsupervised error quantification/correction methods to estimate model errors without prior knowledge of the solution.
Most solvers use the equation based loss functions as a surrogate measure for the error.
However, while such a measure is intuitively related to the error in the solution model, an explicit description of that connection is mandatory, if error quantification is to be done without knowing the solution.
We achieve these goals by explicitly relating the loss terms and the model error.
We showcase how these connections allow results on the model error that don’t rely on prior knowledge of solution.
We propose techniques by which these results can be used to build significantly more efficient NN DE solvers, with only marginal increases in computational complexity.
We formalize our ideas into four theorems, two inequalities, and two algorithms.
We validate our claims with a collection of numerical experiments on several non-trivial DEs (including nonlinear PDEs).
For the sake of readability and simplicity, all proofs have been rigorously presented in the appendices, while the main text simply reports the results and discusses their significance.
An associated codebase is also provided, with inbuilt options for the DEs already studied as part of this work.
However, the codebase has been designed so that the users may easily add their own
DEs of interest (the assumptions under which this work is valid should be general enough for a wide variety of DEs from many different scientific disciplines).
N EURAL N ETWORK D IFFERENTIAL E QUATION SOLVERS
For ease of discussion, we let D
⊆ R
d denote some closed, bounded, path connected domain of interest for some differential equation (DE).
Let
∂ D ⊆
D be the portion of the domain over which some constraint conditions on the solution exist (usually obtained as boundary conditions, empirical data, etc).
Assume that our chosen DE, when given unique constraints over ∂ D , admits a unique solution Φ : D → R D .
We wish to consider a (possibly non-linear) equation operator F : G → H , where G and H are some suitable spaces of functions over D ( Φ ∈ G ).
We decompose F as
[Equation]
where L represents the term(s) which depend linearly on Φ , N the represents the term(s) which depend non-linearly on Φ , and C are the terms independent of Φ .
This additive decomposition into linear, nonlinear, and constant terms is always possible: for linear DEs, N ≡ 0 ≡ C .
We have
F [Φ] =
L [Φ] + N [Φ] + C = 0 . Let us assume we wish to construct an NN based approximation N : D → R D .
Let us also assume the NN uses analytic activation functions so that N ∈ C ∞ .
Let W be its width (neurons per hidden layer) and D N be its depth (number of hidden layers).
Finally, let w ≡ { b 11 , w 111 , w 112 , . . .
, b 21 , w 211 , w 212 , . . . } ∈ M ⊆
R M be the M weights and biases of the NN.
2. E XISTENCE AND COMPLEXITY
Feedforward NNs with ReLU activation and W ≥ d + 4 can arbitrarily well approximate any
Lebesgue integrable function Φ : D → R D w.r.t.
L 1 norm, provided that D is some compact subset of R d and enough depth D N is provided to the NN (5).
The same holds for NNs with ReLU activation and D N ≥ log 2 ( d + 1) provided that they are wide enough (6).
Hence, there exist w such that N can arbitrarily approximate any Φ ∈ C k over D , given large enough ( W, D N ) .
These theoretical guarantees may be practically realized by leveraging the minimum regularity expected from Φ .
Recall W k, ∞ ( D ) is the Sobolev space of order k , on compact D ⊆ R d , w.r.t.
the
L ∞ norm.
Let us assume that Φ ∈ W k, ∞ ( D ) .
There exists some N with ( W, D N ) that can εapproximate Φ : D → R , if W = d + 1 , D N =
O ( diam ( D ) /ω
− 1 f ( ε ))
d (7), where
[Equation]
There also exist N that ε - approximate Φ : D → R , if M = O ( ln ( 1 ε ) ε − d/k ) , D N =
O ( ln ( 1 ε ))
2. O PTIMIZATION
The loss function L to train N usually takes the following form (2; 3; 4):
[Equation]
where ∥ · ∥ p is the usual p -norm on R D .
Variants of Stochastic Gradient Descent (SGD) are almost always capable of eventually reaching adequately small loss values for large enough ( W, D N ) under such loss (9).
Thus as L → 0 during optimization, the uniqueness of Φ implies N → Φ over D (formally shown in Theorem 1).
2. A LTERNATIVE PARAMETERIZATION OF THE CONSTRAINT CONDITIONS ON ∂ D
Sometimes, the constraint conditions (such as initial or boundary conditions) may be enforced by using the following parametrization as the model N for Φ , at some arbitrary x 1 ∈ D :
[Equation]
where x 2 ∈ ∂ D is some appropriate nearest constraint point to x 1 .
Here, N o represents the NN and dist ( x 1 , x 2 ) is some metric that enforces that the model N is always exact over ∂ D , while allowing the NN flexibility to learn Φ elsewhere (ex: 1 − e −∥ x 1 − x 2 ∥ , as used in (10)).
This parametrization can eliminate the need for the E x 2 ∈ ∂ D term in Eq. 2.
Our work is applicable in such scenarios too.
E RROR BOUNDS AND C ONVERGENCE FOR NN DE S OLVERS
Define Φ ϵ 1 :=
Φ − N as the error in our solution model.
Three simple observations about N , Φ ϵ 1 , the convergence of L to 0, and their inter-plays underpin the entirety of this work:
F [ N ] is not explicitly dependent on Φ ϵ 1 : there is no explicit relation between L and Φ ϵover D − ∂ D , where Φ is unknown.
We don’t know if/how E (cid:104) ∥ Φ ϵ 1 ∥ (cid:105) → 0 as L → 0 . 2.
Thus, Φ ϵ 1 associated with N is not estimable over D − ∂ D in standard NN DE solvers.
Optimization performance saturates when N settles around a local minima(s) of L .
In its most general form, the first problem can be immensely intractable and is often handled separately for different kinds of DEs (3; 10; 11; 12; 13).
In this section, we present generic results (proofs in Appendix B) which strongly combat this problem for a large collection of DEs originating in scientific domains, including many nonlinear PDEs.
Our results will be valid for any F [ N ] , s.t. F [ N ] = 0 = ⇒ N = Φ , thus allowing a larger class of loss function designs as well.
Theorem 1.
Let G and H be two Banach spaces.
Suppose that F : G ⊃ U → H is Fréchet differentiable in U , that the derivative of F at Φ ∈ U is invertible, and F [Φ] = 0 uniquely.
Then, there exists a neighbourhood V ⊂ H of 0 small enough such that
[Equation]
Note that the existence of the Frechet derivative of F (denoted D F ) is necessary if gradient descent or related methods are to work.
As such, only the assumption that D F − 1 exists within some open neighbourhood of Φ is imposing additional structure.
Theorem 1 is a powerful result backing the wisdom of using NN DE solvers.
However, the same assumptions allow us to go much further:
Theorem 2.
Under the same assumptions as Theorem 1, we have
[Equation]
= O [Equation] A common observation made for NN DE solvers is that E (cid:104) ∥
F [ N ] ∥ (cid:105) ∝ E (cid:104) ∥
Φ ϵ 1 ∥ (cid:105) .
We codify that as a theorem over a relaxed class of assumptions on F as Theorem 2, validating the intuitive wisdom of using the L as a surrogate for error: especially when the associated constant of proportionality for the Lipschitz continuous F is of order unity. Some famous examples where Theorems 1 andare valid are non-degenerate Hamiltonian systems, nonlinear Poisson-Boltzmann equation, Heat and
Poisson’s equation with homogeneous boundary conditions, etc.
Our assumptions even guarantee an exponentially convergent optimization phase somewhere at some rate in G (see Lemma 3 in Appendix B). However, we can go even further and explicitly estimate that rate with one additional assumption: Theorem 3.
Let G, H be Hilbert spaces, U ⊂ G an open subset, and N ( t ) be the NN model after t iterations.
Assume F ∈ C 2 ( U ; H ) , and that D F [Φ] is invertible at Φ .
Then for every ϵ > 0 , there exists an R > 0 , s.t.
for all initial conditions N (0) ∈
B R (Φ) ⊂ U , the gradient descent equation
[Equation]
has a solution that satisfies
[Equation]
where
[Equation]
σ min > 0 implies exponential convergence at least at that rate is always possible.
However, while we mitigate the concerns raised in the first two observations, Theorems 1 - 3 still only weakly/globally describe Φ ϵ 1 over D .
We can obtain stronger estimates on ∥ Φ ϵ 1 ∥ by assuming structural information on the Φ dependent terms L , N and on the interactions they might have with each other.
In particular, assume that D N is a positive (or negative) definite operator (such an assumption is often taken to avoid degenerate systems).
Define N s =
Φ − s Φ ϵ 1 .
Further, let:
[Equation]
Let F max be the maximum of ∥ F [ N ] ∥ over D for some model N that has finished optimizing.
We then have the following inequality on ∥ Φ ϵ 1 ∥ (see Appendix C for assumptions and proof):
[Equation]
A variation of Inequality 4 in Appendix C showcases how NN DE solver design can encode other knowledge/guesses on the system’s behavior.
Those can be used to fine-tune the assumptions that go into obtaining the presented inequality and lead us to modifications needed on the bound.
However, even with the stronger inequalities on ∥ Φ ϵ 1 ∥ (which might still give severe overestimates, if H min is not properly controlled for), we don’t yet have information regarding pointwise behavior of Φ ϵ 1 over D − ∂ D .
There is a scarcity of work on algorithms that can explicitly estimate Φ ϵ 1 over D − ∂ D , using only the information available to the model N .
More often than not, precise error estimations rely on actual knowledge of Φ over entirety of D .
In the next section, we shall provide an unsupervised algorithm to estimate Φ ϵ 1 over D − ∂ D , without knowledge of Φ .
This endeavor, combined with the third observation will directly lead to another useful application.
For any nontrivial DE, it is improbable that N over D will be the exact solution once a stable minima(s) of optimization is reached (stability is meant here in the sense of a converged model that is unlikely to find a significantly better minima in a reasonable amount of iterations).
Indeed, it is demonstrably impossible if the DE does not admit closed form solutions, since N is always a closed form expression.
Any further training would not provide any meaningful gains in accuracy.
However, in that regime, even a moderately good estimate N ϵ 1 for Φ ϵ 1 would be immediately useful: we could simply use N + N ϵ 1 as a better model (albeit, at additional optimization/model memory costs).
Finally, just like Theorems 1 - 3 remove the ambiguity associated with standard NN DE solvers, we will find similar results reliably eliminate such ambiguities vis a vis the error model N ϵ 1 .
E RROR E STIMATION AND C ORRECTION
We begin by noticing the following:
[Equation]
where N ′ [ N , Φ ϵ 1 ] :
= N [ N + Φ ϵ 1 ] − N [ N ] .
Thus, we have a new equation and its operator F
[Equation]
with Φ ϵ 1 as the only unknown quantity.
The uniqueness of Φ implies that if N is fixed, then Φ ϵ 1 is a unique solution to the DE given by Eq. 6, and we have
[Equation]
Note that N ′ is obtained such that terms explicitly dependent on Φ don’t appear in Eq.
5: that is what allows tight bounds on Φ ϵ 1 in (10), without depending upon explicit knowledge of Φ (Inequality 4 is a strong generalization of that result).
As a motivating example, consider the cases when the nonlinearities in F are degree 2 polynomials: N [ ] =
In that case, we have:
[Equation]
The transformation technique itself (algebraic manipulations, Taylor expansions, etc) is not important: we only need it to be such that F [ N ] , Φ ϵ 1 appear in relation to each other over all of D .
Indeed, we are always able to directly leverage the unsimplified N [ N ] − N [ N + Φ ϵ 1 ] form of N ′ (or even the F [ N + Φ ϵ 1 ] = 0 equation) 2 for optimization purposes in the proposed Algorithm 1. is sometimes profitable to obtain an explicit, separable expression for N ′ [ N , Φ ϵ 1 ] .
For example, whenever Taylor expansions can be used,
[Equation]
where T i ( N ) are the i th order Taylor terms.
These can be useful for analysis, since N , Φ ϵ 1 appear in multi- plicatively separable terms in the new expressions ((10) used these forms to bound Φ ϵ 1 ).
However, we obtained
Inequality 4 in a way that renders such transformations superfluous: our bounds stand under weaker assumptions.
1: procedure E RROR C ORRECT 2: initialize NN DE solver N : D → R .
3: while N not converged do ▷ Initial model N over D 4:
Optimize N for a step using L (Eq. 2) 5: end while 6:
Freeze the parameters of N ▷ Fix N to start error analysis 7: initialize Error estimation model N ϵ 1 .
8: while N ϵ 1 not converged do ▷ Error estimation over D 9:
Optimize N ϵ 1 for a step using L ϵ (Eq. 8) 10: end while 11: return N + N ϵ▷ Error correction over D 12: end procedure
Eq. 6 immediately hints at a method to estimate Φ ϵ 1 : we use Eq.
6 with another network model, N ϵ 1 , in the manner we used N with Eq. 2, to obtain an approximation for Φ ϵ 1 .
More precisely, once we have a converged model N that has saturated its capacity to model Φ , we use:
[Equation]
(7) to define a new loss function for the error model N ϵ 1 :
[Equation]
Alternatively, since the parameters of N are kept unchanged, we can use the equivalent form:
[Equation]
In problems with homogeneous constraint conditions, Φ( ∂ D ) [Equation] 0 , we may replace the second term in Eq.
8 by E (cid:104)
∥N ϵ 1 ( x 2 ) + N ( x 2 )
∥ p (cid:105) , since Φ ϵ 1 ( ∂ D ) = [-
-]0 − N ( ∂ D ) =
−N ( ∂ D ) .
The same reasoning underlying L = 0 = ⇒ N = Φ , allows that L ϵ 1 = 0 =
⇒ N ϵ 1 =
Φ ϵ 1 .
Indeed, analogs of Theorems 1 - 3 and Inequality 4 are obtained directly, as stated in a generalized form in the next subsection.
In particular, our knowledge of F [ N ] allows us substantial control over how to estimate Φ ϵ 1 , whereas the knowledge of F 1 [ N ϵ 1 ] (or alternatively F [ N + N ϵ 1 ] ) allows us bounds on ∥ Φ ϵ 1 − N ϵ 1 ∥ .
The ability to internally and reliably estimate Φ ϵ 1 associated with N is not only useful for error analysis purposes, but also for refining N .
Algorithm 1 describes the procedure.
Note that the difficulty of estimating Φ and Φ ϵ 1 should be roughly equivalent since Φ ϵ 1 has at least the same regularity as Φ .
As such, the optimization costs of N ϵ 1 per iteration will not be substantially larger than those of N (see discussion/experiments in the next sections).
Finally, note that Alg. 1 provides relative corrections to some model N .
We do not claim that only error correction can provide improvements in performance to the extent shown in Fig.
We only claim that error analysis and correction are useful and available tools for any model N that might have been obtained via some technique based on NN DE solvers.
It should always be possible to error correct N using the strategy we are describing, as long as the associated assumptions are satisfied.
[Equation]
4. H IGHER ORDER E RROR ESTIMATIONS AND CORRECTIONS
Consider some sequence of networks N ϵ 1 , N ϵ 2 , ...., N ϵ i − 1 , each modeling the errors Φ ϵ 1 , Φ ϵ 2 , ...., Φ ϵ i , associated with error corrected solution models N , N 1 , ..., N i − 1 .
Here, N j =
N + N ϵ
+ ...
+ N ϵ
j is the j th order corrected model.
Assume the i − 1 order error correction model
N ϵ
i − 1 has saturated its capacity to model Φ ϵ i − 1 .
We can obtain a higher order corrected model N i by setting up a new network N ϵ i to estimate Φ ϵ i .
The new network N ϵ i is optimized using:
[Equation]
Figure 1 provides a visual representation of the recursive process involved in building i th order corrected models.
Appendix A demonstrates how to arrive at these results.
We have direct analogs of Theorems 1, 2, and 3 for the i th order errors and error models as:
Theorem 4.
Under the assumptions above on F , the analogous result holds for F i for all i ∈ N , i.e. we can choose a neighbourhood U i ⊂ G of 0 small enough
such that if N ϵ i ∈
U i then
[Equation]
Furthermore,
[Equation]
Lastly, under gradient descent, initial condition N ϵ i (0) has a solution that satisfies
[Equation] Inequality 4 becomes (under the same structural assumptions as Inequality 4):
[Equation]
Algorithm 2 in the appendix provides the pseudo-code to setup higher order corrections.
W HY ERROR CORRECTION WORKS AND HOW WE CAN MAKE IT BETTER
NN DE solvers transform the problem of estimating Φ into one of finding an appropriate N in some suitably chosen functional space of NN models.
The use of gradient descent type methods turns this into a stochastic dynamical process in that space.
The probability that a real network N will end up exactly modeling Φ within a finite number of iterations is 0 (assuming F is even moderately non-trivial).
Note this is true without even considering that N is constrained to be a closed form model, whereas most non-trivial DEs don’t have such solutions.
As such, a real network N is doomed to languish in some inadequate minima(s) of optimization, no matter how much we optimize it.
Its capacity to approximate Φ is necessarily limited and once that capacity is saturated, further training of N is a waste of computational resources.
Error correction expands that capacity for approximating Φ , by introducing new trainable parameters (as components of N ϵ 1 ) which exist exclusively for modeling the error Φ ϵ 1 =
Φ − N .
Since the parameters of the base network N are frozen, we have locked in the amount of performance already achieved on the base model.
Further, since we assume error correction is being turned on when the base model’s approximation capacity is effectively saturated (which is certain to happen), the gains we make with N ϵ 1 are truly unachievable without it.
We have put forth a claim that error correction is not significantly more complex or resource hungry than the standard NN DE solver method: the reason being that Φ and Φ −N are in the same functional space, have the same regularity, etc.
Essentially, we meant that Φ ϵ 1 is no more complicated than Φ .
However, it bears noting that modeling Φ ϵ 1 using N ϵ 1 can be a very different kind of problem than modeling Φ using N .
For example, Φ ϵ 1 presumably lies in a different scale of magnitude than Φ and might present itself as a different kind of structure over D (for example, in one sense, it might be significantly more or less "oscillatory" than Φ ).
Its optimization might also require its own set of hyper-parameters, algorithm choices, activations, etc. This is where our control over the sampling of F (and F 1 ) comes into play. Theorems 1 - 3 and
Inequality 4 allow us to estimate both the scale and complexity of Φ ϵ 1 , relative to N .
The correlation expected between Φ ϵ 1 and F [ N ] means even crude analysis of F [ N ] is enough to point us in the correct direction, vis a vis the scale and complexity we expect N ϵ 1 to model.
For example, we can bound the expected scale of Φ ϵ 1 by simply analysing the max of ∥ F [ N ] ∥ over
D : the linear correlation result already gives us one way to weakly bound Φ ϵ 1 .
The strong inequality 4 may allow for more precise control, when its assumptions are satisfied.
Similarly, over a domain D , we could perform a FFT analysis of F [ N ] and gauge how oscillatory
N ϵ 1 should be relative to N (since we have control over both N and F [ N ] and we expect Φ ϵ 1 to be linearly correlated to F [ N ] ).
Alternatively, we could also analyse the distribution of ∥ F [ N ] ∥ over D to understand where the equation is being badly solved, and switch from uniform to targeted sampling over those subdomains.
Other methods of estimating relative complexity of features should also be possible, since we have complete access to the behavior of F [ N ] and F 1 [ N ϵ 1 ] .
The code we provide allows the user to supply their own estimates for relative scale and complexity of N ϵ 1 wrt N , while keeping our preferred defaults.
We note again, that in no way is prior knowledge of Φ assumed: only the information already known before or computed using the NN DE solver is being used.
Domain knowledge of an expert building an NN DE solver for a system of their interest is thus rewarded during error analysis and correction.
Finally, note that error correction inherits one major issue prevalent within machine learning: the ambiguity in determining if a real model N has reached (or nearly reached) its capacity for effectively modelling Φ .
The issue is caused because of lack of work on a priori or in situ convergence guarantees. It is extremely tough to gauge whether a loss trajectory has flattened due to saturation in modeling capacity or it is simply in a non-trivial region of dynamics from which it could escape at any time.
If error correction is attempted while the base model itself is fully capable of providing significant increases in accuracy, the additional resources will simply be wasted (and may even prove to be counter-productive).
Indeed, in our numerical experiments, we found that convergence of the base model N was a practical pre-requisite for significant gains via error correction.
However, in our proposed approach, the issue is still somewhat mitigated.
Error correction can not cost significantly more than the standard methods (it can at worst be 2X expensive).
Since it can lead to very significant improvements in accuracy when used at the appropriate time, it can always be used as an unsupervised, post-processing/refinement routine without significant risks.
N UMERICAL E XPERIMENTS
Let us exemplify the above discussion with an example DE that contains terms with linear, nonlinear, and no dependence on the solution.
In particular, we investigate a variant of the nonlinear
Poisson-Boltzmann equation (nPBE), which serves as an important tool for the study of electrostatic interactions with widespread applications in biology and medicine (14).
We will choose
L [ ] = − ∆[ ] , N [ ] ≡ sinh [ ] , C ≡ g , where g is some independent function/term.
Thus, we define:
[Equation]
For some solution attempt N : R d → R and associated error Φ ϵ 1 =
Φ − N , we get
[Equation]
Thus, replacing Φ ϵ 1 with an error model N ϵ 1 , we get the residual equation for the new loss as:
[Equation]
Now, to setup N ϵ 1 as the error NN DE solver (after halting the optimization of N ), we can use Eq.
14 to obtain an approximation of Φ ϵ 1 .
We not only get an estimate for Φ ϵ 1 , we also get to use N ϵ 1 as a correction term, while retaining the advantages that make N a lucrative option.
6. E RROR CORRECTION RESULTS
We choose D =
[ − π, π ] d ⊂ R d , and randomly sample points from this domain as training data.
The exact form of our chosen DE, considered over D with solution Φ : D → R and homogeneous boundary conditions Φ( ∂ D ) = 0 , is:
[Equation]
The solution to this DE, which showcases significant scale and feature variation over D (Fig. 3), is:
[Equation]
Note that the solvers never see this solution, and only have access to their own parameters and the operator F during optimization.
We only use the exact solution to verify that our claims hold true.
Given a network N to model the true solution Φ , we optimize N using the mean squared loss L :
[Equation]
The error correction model N ϵ 1 : D → R is effectively optimized using the following loss:
[Equation]
As discussed before, L → 0 = ⇒ F [ N ( x )] → 0 , over x ∈ D , which implies N → Φ .
Similarly,
L ϵ 1 → 0 = ⇒ F 1 [ N ϵ 1 ( x )] → 0 , which implies N ϵ 1 → Φ ϵ 1 .
We use Eq.
16 to benchmark efficacy of our proposed algorithm for ( ω, d ) choices of (5 , 2) and (1 , 4) .
The goal of the experiments was to figure out what impact error correction would have, if used in lieu of the standard approach, while running for an equivalent number of training iterations.
As such, choices on width, depth, architecture, etc were kept the same between N and N ϵ 1 .
The experiments also demonstrate the effects of invoking error correction at different stages of training (Fig. 2).
[Figure]
To present and analyse our results, we define a computational cost metric, τ : (time per iteration), and a computational performance metric, Relative Error (RE):
[Equation]
where N i : D → R is the i th order corrected, NN DE based approximation for Φ over D .
K represents the number of sampled points in D for the calculation.
In Fig.
2, we plot the relative error dynamics of the various choices made.
Notice the very immediate gains made by error correction. [Table]
We also investigate alternative scenarios, where the slightly higher consumption of resources is used in other ways, to quantify ablation possibilities. We conduct two sets of examples. For size ablation experiments
[Table] we allow N to be proportionally larger, while training it for the same number of training iterations (say T ). For time ablation experiments, we allow the standard algorithms a proportionally higher number of iterations.
We define the Return on Investment or ROI of each choice c = (error correction, size, or time ablation) as: [Equation] As Table 1 shows
[Table]
error correction consistently provides the best ROI.
The associated codebase with this work provides numerical experiments on some other nonlinear/chaotic DEs (it is designed to be compatible with any DE operator with permissible Frechet derivatives and allow any finite order of correction).
The results on those systems are also tabulated in Table 1 (first order error correction was injected at 50 % of the run-time, for all other examples).
All results are in line with our expectations.
The additional systems are described in appendix D.
C ONCLUSIONS
The surging popularity of NN DE solvers presents exciting possibilities in many scientific fields: their capacity to sidestep the curse of dimensionality (15), general advances in computing/GPU power, and their easy interpret-ability make for a powerful combo.
As such, it is important that these solution models be capable of validation over domains where true solutions are not available.
Our work proposes theorems/methods that fix this deficiency for many NN DE solvers.
Summarily, NN model errors are unambiguously estimable, and profitably so, if the assumptions of Theorem 1 - 3 hold. For systems where the assumptions of Theorems 1 - 3 do not hold, existing work still leads us to expect that NN DE solvers ( and thus, error correction) will work. However, for those systems, error correction converts the ambiguity associated with the models N into the ambiguity associated with the estimate N ϵ 1 .
These unreliable estimates of Φ ϵ 1 can lead to more accurate models, but our work does not rigorously predict when or how this happens.
However, since this ambiguity is present for all NN DE solvers prior to our work, unsupervised error analysis and correction can only be an improvement upon the existing situation (even when it is unreliable).
Future work will focus on extending our results rigorously to more systems, so that even larger classes of DEs may come within the remit of reliable error estimation and correction.
Lastly, note that the suggested ideas are not radically different than those that already exist for classical numerical methods: higher order corrections are about as old as the field itself.
By pairing the many significant advantages of modern NNs with those old ideas, we simply hope to have presented a blueprint that will be useful for a wide class of scientific problems that are being tackled using NN DE solvers.
A S ETTING UP HIGHER ORDER CORRECTIONS
Using Eq.
equation we are able to define new loss L ϵ 1 using F 1 [ N ϵ 1 ] (or equivalently, F [ N + N ϵ 1 ] ).
Define Φ ϵ 2 :=
Φ ϵ 1 − N ϵ 1 =
Φ − ( N + N ϵ 1 ) and use the same trick used to derive Eq.
5 to realize that,
[Equation]
[Equation]
Let us define the 1 st order corrected model as N 1 :
=
N + N ϵ 1 , and with a slight abuse of notation,
F 1 [ N 1 ] := F [ N + N ϵ 1 ] .
Thus, we may rewrite Eq.
19 as:
[Equation]
Note the functional similarity to Eq. 6.
Further, to setup a 2 nd order correction estimate N ϵ 2 for Φ ϵ 2 , we need to minimize over the following residual, similar to Eq. 7:
[Equation]
In general, given an ( i − 1) th order corrected model N i − 1 using N i − 1 = N + N ϵ
+ N ϵ
+ .....
+ N ϵ
i − 1 , we can define Φ ϵ i :=
Φ − N i − 1 and F i −
[ N i − 1 ] as the corresponding residual for the preceding equations.
Then, Φ ϵ i is the unique solution to the following equation
[Equation]
The error can be estimated using a differential equation solver (e.g. a NN) with residual:
[Equation]
Figure 1 depicts the extension of the error estimation and correction framework as a graphical model.
Notice that the initial solver estimator and the error estimators may be pre-fetched before carrying out the optimization steps.
Since Φ ϵ i =
Φ − N i − 1 is always known exactly over ∂ D for all i , we can optimize the i th order error model N ϵ
i using L ϵ i :
[Equation]
Algorithm 2 demonstrates the procedure for higher order estimation and correction.
The uniqueness of Φ as a solution to F [ ] also implies the uniqueness of Φ ϵ i as a solution to F i [ ] .
Unpacking the definition of F i and N i − 1 we find that
[Equation]
and therefore the solution to this equation is still unique and convergence of L ϵ i
→ 0 implies F i → 0 , and thus also N ϵ i
→ Φ ϵ i .
Theorem 3 follows naturally from Theorems 1 and 2.
Inequality 11 follows naturally from Inequality 4.
Algorithm 2 describes the proposed implementation. Algorithm
1: procedure E RROR C ORRECT ( m ) 2: initialize NN DE solver N : D → R 3: initialize Error estimators {N ϵ i } mi = 4:
Train N , and define ˆ N := N
▷ ˆ N serves as a dummy variable 5: for i = 1 , . .
, m do 6:
Train N ϵ i until loss converges 7:
Save parameter states of N ϵ i 8: ˆ N :=
ˆ N + N ϵ i 9:
Freeze parameters of ˆ N 10: end for 11: return N + (cid:80) mi =1 N ϵ i ▷
Equivalently, return ˆ N 12: end procedure
B T HEOREMS ON THE RELATIONSHIPS BETWEEN Φ ϵ 1 , F [ N ] , AND L
This work is centered around three theorems. We begin this section by summarizing and contextualizing those results. Theorem 1 verifies the intuitive expectation that F [ N ] → 0 implies the fitness of N as an approximation of Φ .
We do this by taking advantage of the uniqueness of Φ as a solution, alongside the fact that any gradient descent method can only be successful, iff the gradient itself exists in a well defined sense.
Theorem 2 naturally follows as a consequence of Theorem 1.
It adds a strong a priori expectation we should have from NN DE solvers that satisfy the assumptions of Theorem 1 and tells us that their errors should be quantifiable in some sense, if information about F [ N ] is sampled (which is always possible up to the desired resolution).
Additional structure leads us to stronger quantification capabilities (surmised in Inequalities 4 and 11 and proved in appendix C).
Finally, assume a mapping F : G ⊃ U
→ H between two Hilbert spaces, e.g. representing a non-linear PDE mapping between function spaces.
Further, assume it is twice differentiable and at the solution Φ , satisfying F (Φ) = 0 , D F [Φ] , is an invertible linear map.
Then, the gradient descent procedure guarantees convergence at a rate
[Equation]
where ϵ > 0 can be chosen sufficiently small if ∥N (0) − Φ ∥ ≤ R , where R depends on ϵ .
σ min here is given by the minimum of the spectrum of (D F [Φ]) † D F [Φ] which is strictly positive as D F [Φ] is invertible.
Theorem 3 guarantees that exponentially converging NN models for Φ exist somewhere in
G , and that gradient descent will allow us that level of performance.
We need only find some initial model from that region.
Together, the three theorems give us an idealized framework for describing NN DE solvers under iterative optimization.
Real NN DE solvers differ from this idealized framework in two ways: their optimization is a discrete process and the subspace within which real NN models lie (say G M , where
M is number of parameters in our model), while being capable of coming arbitrarily close to Φ , seldom contains it.
The real optimization is a discrete approximation of the ideal trajectory: further, the empirical trajectory is actually a projection of that discrete trajectory onto G M .
Note that for finite dimensional approximation of the problem on a subspace G M ⊂ G of dimension
M , the corresponding constant σ G M min is greater or equal to σ min .
Furthermore, if G M ⊂ G m then σ G M min ≥ σ G m min .
Thus, one can expect the error correction procedure to allow for an exponential improvement at each step, iff σ min > 0 , i.e. exactly in the cases when D F [Φ] is invertible.
Finally, Theorem 4 follows trivially from the definitions of F i , N i − 1 , Φ ϵ i , and Theorems 1 - 3.
We now move to rigorously prove Theorems 1 - 3 below.
B. I NVERSE F UNCTION T HEOREM AND T HEOREMS 1 AND
We quickly summarize the Banach space version of the inverse function theorem that allows us to establish that F [ N ] → 0 implies N → Φ (for further details see for example (16)).
We also establish another lemma we will use to estimate the convergence rate of NN DE solvers.
Let G and H be two Banach spaces, and U ⊂ G an open subset.
A continuous map between the two Banach spaces F : G ⊃ U → H is said to be Fréchet differentiable at a point x ∈ U iff there exists a linear bounded operator L x : G → H such that
[Equation]
If the map x (cid:55)→ D F [ x ] =
L x is continuous, then one says that F ∈ C 1 ( U ; H ) .
Analogously C( U ; H ) denote the functions which are twice differentiable, and if H =
R we drop H from the notation.
Lemma 1.
Let F : U → H be a C 1 -map.
Suppose that there exists a point x 0 ∈ U such that D F ( x 0 ) is an isomorphism (i.e. it has a continuous inverse).
Then there exists a neighbourhood V ⊂ H of
F ( x 0 ) and a C 1 function F − 1 : V → G that is a local inverse of F .
Proof.
See Theorem 5.2 of (16). Lemma 2.
Assume L ∈ C 2 ( U ) .
L is strongly convex at N ∈ U , if for all N ′ ∈ G [Equation] N [Equation]
≥ µ ∥N ′
∥ 2 .
Proof.
This follows by a simple application of Taylor’s theorem as in the finite dimensional case.
For
Taylor’s theorem for functions on Banach spaces, cf.
Theorem 1.
Suppose that F ∈ C 1 ( U ; H ) , that the derivative of F at Φ ∈ U is invertible, and F (Φ) = 0 .
There is a neighbourhood V ⊂ H of 0 small enough such that
[Equation]
Proof.
By Lemma 1, we can choose neighbourhoods Φ ∈ U ′ ⊂
U and 0 ∈ V ⊂ H such that
F : U ′
→ V is a diffeomorphism.
Then, if N ∈ U ′ the continuity of F − 1 implies
[Equation]
which is the assertion.
Theorem 2.
Under the same assumptions as above we
[Equation]
Proof.
Since by Lemma 1, F − 1 is differentiable at 0 , it follows that it is locally Lipschitz continuous around 0 , implying that
[Equation]
The constant of proportionality is approximately given by λ − 1min where
[Equation]
the eigenvalue with minimal absolute value in the spectrum of D F [Φ] .
The nPBE example from section 6 fits nicely into this paradigm as the map
[Equation]
is continuous and continuously Fréchet differentiable with Fréchet derivative
[Equation]
which is everywhere continuously invertible as a linear map.
The constant of proportionality is approximately 1.
Theorems 1 and 2 guarantee that there exist adequate models for Φ , if the loss is going down sufficiently enough .
However, we still have no clue about how fast or slow this convergence is going to be and how to convert these existence results into actual estimates on Φ ϵ 1 (and higher order Φ ϵ i ).
We develop those results below, with minimal additional assumptions.
B. G RADIENT DESCENT IN H ILBERT SPACES
We assume that G is a Hilbert space, i.e. in addition to being a Banach space, it is also equipped with an inner product ⟨· , ·⟩ such that ⟨ v, v ⟩ = ∥ v ∥ 2 for all v ∈ G .
Assume that we are given a loss function L : G ⊃ U → R that is an element of C 1 ( U ) . We denote by ∇L ( N ) the unique element of
G , s.t. for all Ψ ∈ G
D
[Equation]
If ∇L is a locally Lipschitz continuous or equivalently if L is a locally L -smooth function, then the
ODE
[Equation]
[Equation]
has a unique local solution, cf.
By locally L -smooth, we mean that around a minimum Φ , there exists R > 0 such that for all N , N
′ ∈
B R (Φ)
[Equation]
Furthermore, for µ > 0 , we call the function L locally µ -strongly convex around a minimum Φ , if there exists R > 0 such that for all N , N ′ ∈
B R (Φ)
[Equation]
We need one more lemma to have all the tools we will use to get Theorem 3.
Lemma 3.
Let L ∈ C 1 ( U ) be a locally L -smooth, locally µ -strongly convex function around a minimum Φ , s.t.
L ≥ 0 and L (Φ)
= 0 .
Then for any initial condition N ∈ B R (Φ) , a ball where L is both L -smooth and µ -strongly convex, the Gradient Descent equation 23 converges exponentially at rate with rate µ 2 towards Φ , i.e.
[Equation]
Proof.
From the strong µ -convexity it follows that
[Equation]
Solving this differential inequality, we find
[Equation]
Lemma 3 guarantees the existence of an exponentially convergent regime of optimization, as long as we assume that D F exists and L is µ -strongly convex.
However, it is a non-constructive statement: we have no information on what behavior should be expected from µ .
We can make it constructive with an additional assumption on the existence of D 2 F .
B. C ONSTRUCTIVELY ESTIMATING THE RATE OF C ONVERGENCE
Assume that F is in C 2 ( U ; H ) .
Furthermore, we require that H is also a Hilbert space.
From the definition of Fréchet differentiability we can write F as
[Equation]
where ∥ R 1 ( N ) ∥
=
o ( ∥N − Φ ∥ 2 ) .
Then our loss function L ( N ) := ⟨ F [ N ] , F [ N ] ⟩ is strictly convex in a neighbourhood of Φ as
[Equation]
where we used that L (Φ)
{+= 
+}
and defined
[Equation]
as D F [Φ] is invertible.
Furthermore, our loss function can be written as
[Equation]
where | R 2 ( N ) |
= o ( ∥N − Φ ∥ 3 ) and R 2 is also a C 2 -function, and its second derivative is o ( ∥N − Φ ∥ ) .
Thus, for every ϵ > 0 there exists R > 0 such that for all N in the ball of radius R around Φ
[Equation]
Then, we have that for all N ∈ B R (Φ)
[Equation]
It follows that L is µ -strongly convex in the ball of radius R around Φ with constant µ = (1 − ϵ ) σ min , and thus the gradient descent flow N ( t ) converges exponentially at rate (1 − ϵ ) σ min.
We have thus proven Theorem 3.
Theorem 4 follows immediately from the results in appendix B.
We will begin our attempt to ascertain bounds on ∥ Φ ϵ 1 ∥ with classical Hamiltonian systems, to exemplify the intuition and the technique involved in the generic bound we wish to develop.
Once that is achieved, we will show how a generalization follows in a straightforward manner.
C. H AMILTONIAN SYSTEMS
To start, let us remember Φ ϵ 1 :
=
N − Φ =
( N 1 − ϕ 1 , . . .
, N D − ϕ D ) for a D -dimensional dynamical system to be solved on a domain of interest [0 , T ] .
Note that D is necessarily even for a Hamiltonian dynamical system.
Let us construct a worst case scenario for the norm of Φ ϵ 1 .
We write the equation in terms of the Hamiltonian formulation:
[Equation]
where H is the appropriate Hamiltonian, and J is the symplectic matrix
[Equation]
and I D/ 2 is the identity matrix.
The NN DE solver is trained using F [ N ] given by
[Equation]
The equation above represents D separate differential equations in a vector form.
Since F [Φ]( t ) =
for all t we can write F [ N ] as follows, suppressing the time dependence
[Equation]
(27) where N s :=
Φ − s Φ ϵ 1 , and D 2 H (Φ) is the Hessian matrix of H with components ∂
i ∂ j H ( N ) .
In order to be able to extract a meaningful error bound from Eq.
equation 27 we need to make some structural assumption: We assume that at a time t Φ ϵ 1 ∈ [0 , T ] at which ∥ Φ ϵ 1 ∥ obtains its maximum
[Equation]
Note that the assumption of focusing all error in only one component of Φ , as done in (10), already implies the above, but not vice versa.
As such, we are using a much weaker assumption than (10).
Then, for time t Φ ϵ 1 ∈ [0 , T ] , we have
[Equation]
We next need an estimate on the matrix norm of the inverse of R for every t ∈ [0 , T ] .
We can ignore
J as it is an orthogonal matrix and only consider
[Equation]
We can do this if we know that D 2 H ( x ) is a strictly positive or strictly negative definite matrix for all x ∈ D ⊂ R D in our domain of interest, which is a standard assumption for the motion to be non-degenerate.
W.l.o.g., assuming that D 2 H is positive definite, and setting
[Equation]
where λ min is the smallest eigenvalue of the matrix D 2 H ( x ) , we have for all t ∈ [0 , T ]
[Equation]
Finally, letting
[Equation]
we reach the an estimate on ∥ Φ ϵ 1 ∥ as:
[Equation]
Additionally, if we assume that at the time t ˙Φ ϵ 1 at which | ˙Φ ϵ 1 | reaches its maxima,
[Equation]
we have the following bound on ˙Φ ϵ 1 as well
[Equation]
C. B OUNDS ON ∥ Φ ϵ 1 ∥ IN MORE GENERAL SETTINGS
Eq.
27 generalizes beyond Hamiltonian systems in a straightforward manner for F [ ] ≡ L [ ]+ N [ ]+ C as:
[Equation]
The generalized version of the R Φ ϵ 1 · ˙Φ ϵ = 0 assumption is that at the x Φ ϵ 1 ∈ D that ∥ Φ ϵ 1 ∥ takes its maxima in, we have:
[Equation]
The next assumption we generalize is the one made to have non-degeneracy in the solutions we were trying to model.
As such, we assume that D N is a positive definite (or negative definitive) operator.
From then on, we define H min in a similar manner.
More precisely, we say:
[Equation]
where [Equation] Finally, let H min = max { H min 1 , H min 2 } . With that in place, we obtain the following inequalities for ∥ Φ ϵ 1 ∥ :
[Equation]
The generalized version of the assumption on x ˙Φ ϵ where | ˙Φ ϵ | attains its maxima similarly gives:
[Equation]
The assumption that (cid:104) R ( x Φ ϵ 1 )Φ ϵ 1 ( x Φ ϵ 1 ) (cid:105) ·
L [Φ ϵ 1 ]( x Φ ϵ 1 ) = 0 is critical in allowing us strong estimates on ∥ Φ ϵ 1 ∥ .
Superficially, it might seem like too strong of an assumption (and even somewhat of a non-sequitur).
However, note that L [ ] in a DE is often the domain derivative term (we denote it as ∇ x in this discussion, where x ∈ D represents an arbitrary element in the domain.
As such, ∇ x are the spatial derivatives for spatial DEs, time derivatives for ODEs, a spatio-temporal one for space-time DEs, etc).
When ∥ Φ ϵ 1 ∥ achieves its maxima, Φ ϵ 1 · ∇ x (Φ ϵ 1 ) = 0 .
When L ≡ ∇ x , we indeed have Φ ϵ 1 · L (Φ ϵ 1 ) = 0 .
As such, what we are really assuming is that the R term effectively has Φ ϵ 1 as its eigenvector on x Φ ϵ 1 (since that means R Φ ϵ 1 · ∇ x (Φ ϵ 1 ) is 0).
For example, such is the case for
Hamiltonian systems without mixed position-momentum terms.
What we are really constraining, is the behavior exhibited by the operator N in a system of interest.
If the problem of interest comes from a scientific domain with which the user has familiarity, they may supply a different sort of assumption based on those considerations.
For example, we might be able to say:
[Equation]
at the x Φ ϵ 1 where ∥ Φ ϵ 1 ∥ takes its maxima.
The generalized bound simply changes to:
[Equation]
D A DDITIONAL NUMERICAL EXPERIMENTS
As part of validating our claims, we also performed numerical experiments on several scientific ODEs and PDEs.
We chose our experiments such that the set of examples showcases non-trivial spatial and temporal phenomena amongst the combined examples (chaos, high dimensional domains, etc).
Table 1 presents the median results from the collection of randomized experiments conducted for each system.
All numerical experiments were done using a 2019 MacBook Pro with a 2.6 GHz 6-Core Intel Core i7 processor and 16 GBs of 2667 MHz DDR4 RAM.
We describe the chosen systems below:
D. H ENON H EILES
Henon Heiles is represented by the following ODE governing the dynamics of Φ ≡ { x, y, p x , p y } † :
[Equation]
(36) where I is the 2 × 2 identity matrix.
We picked [0 , 6 π ] as the time domain of interest.
All Φ(0) ≡ { x (0) , y (0) , p x (0) , p y (0) } were picked
s.t. H ( z (0))
< 16 , x (0) ∈ [ − √ 32 , √ 32 ] , and y (0) ∈ [ − 0 .
5 , 1 − √ 3 | x | ] , which corresponds to the subset of the phase space that has bounded orbits.
We implement the NN DE solver prescribed in (10).
The models each had 2 hidden layers, withsine activation functions per layer.
The base models were trained for 50000 iterations using ADAM, with an error correction made at 25000 iterations.
We sampled 200 time points were iteration.
D. N ONLINEAR P OISSON B OLTZMANN EQUATION
We have already given details of the equation in the main paper, alongside the associated choices of dimensionality and frequency parameter.
Below, we plot what the ( ω, d ) =
(5 , 2) system should look like when modeled without (a) and with (b) error correction, alongside the true solution (c).
To quantify the visual, we also quote the mean average value of ∥N∥ , ∥N + N ϵ 1 ∥ , ∥ Φ ∥ respectively.
[Figure]
N and N ϵ 1 were fully connected Neural Networks, with 4 hidden layers, each with 50 sine activation functions.
The base model was trained via ADAM for 400000 iterations, while error models were trained for a varying number of iterations, depending on when the error correction models were activated (see Fig. 2).
We sampled 1024 points per iteration.
The ( ω, d ) = (1 , 4) model was trained for 50000 iterations, with 8192 points sampled per iteration.
Both NN DE solvers are implemented using the methods prescribed in (2)
D. N ONLINEAR QUARTIC OSCILLATOR
The nonlinear quartic oscillator is represented by the following ODE governing the dynamics of Φ ≡ { x, p x } :
[Equation]
We sample Φ(0) such that { x (0) , p x (0) } ∈ [ − 1 , 1] × [ − 1 , 1] each time.
We implement the NN DE solver prescribed in (10).
The models had 2 hidden layers each, withsine activation functions per layer.
The base models were trained for 50000 iterations using ADAM, with an error correction made at 25000 iterations.
We sampled 200 time points were iteration.
The associated codebase allows the user to add other systems as per their choice.