Performance-Guaranteed ODE Solvers with Complexity-Informed Neural Networks
Abstract
Traditionally, we provide technical parameters for ODE solvers, such as the order, the stepsize and the local error threshold.
However, there is no guarantee for performance metrics that users care about, such as the time consumption and the global error.
In this paper, we provide such a user-oriented guarantee by using neural networks to ﬁt the complex relationship between the technical parameters and performance metrics.
The form of the neural network is carefully designed to incorporate the prior knowledge from time complexity analysis of ODE solvers, which has better performance than purely data-driven approaches.
We test our strategy on some parametrized ODE problems, and experimental results show that the ﬁtted model can achieve high accuracy, thus providing error guarantee for ﬁxed methods and time guarantee for adaptive stepsize methods.
Introduction
The choice of stepsize for ODE solvers has great inﬂuences on the computational time and error, which are two fundamental performance metrics when solving differential equations.
Existing approaches to choose the stepsize usually make the solver satisfy either the time or error constraint by providing some technical parameters.
On the one hand, the ﬁxed stepsize method like forward Euler is often used in real-time simulator of complex systems [1] which requires strict time guarantee of the method.
On the other hand, the adaptive stepsize method with error control strategy is used in settings when the precision of the solution is required.
In both cases, no convenient method has been developed to choose the appropriate hyper-parameters of ODE solvers to satisfy the requirements.
In recent years, artiﬁcial neural networks are developed for the design of new ODE solvers [2] or the construction of differentiable integrators (NeuralODE)[3] to facilitate the parameter identiﬁcation of dynamic systems.
Based on the pioneering work of NeuralODE, the proxy ODE simulators (hypersolvers) are constructed using neural networks to reduce the time complexity of the inference [4].
To our best knowledge, no efforts have been spared to use neural networks for the purpose of guaranteeing the user-oriented performance.
In our work, we will ﬁll this gap by selecting ODE solvers with complexity-informed neural networks, which is a combination of the mathematical model and the data-driven approach.
To unify the two control goals, we utilize neural networks to ﬁt the complex relationship between user-oriented metrics (the error or time) and technical parameters for given ODE problems.
Using our approach, we can provide strict global error control for ﬁxed stepsize method and approximated time guarantee for adaptive stepsize method, which enlarges the usability for existing ODE solvers.
Methodology
We consider solving the initial value problem for a given ODE:
[Equation]
Many solvers are available to obtain numerical solutions of u ( t ) for an interval [ t 0 , t max ] .
For explicit
Runge-Kutta (RK) methods, u n is updated with the following scheme to approximate the value u ( t n ) .
[Equation]
In the above formula, ψ ( t, u, h ) is a scheme-speciﬁc function for different RK methods.
For example, forward Euler method has ψ ( t, u, h ) =
f ( t, u ) .
Depending on whether h n changes or not, the methods can be divided by the ﬁxed stepsize method ( h n = h is pre-speciﬁed) and the adaptive stepsize method ( h n is adjusted at each step).
In the above, we have described the numerical routine to provide the values of two performance metrics Q 1 and Q 2 .
For these two metrics, our goal is to ﬁnd their optimal tradeoff, which is controlled by varying the threshold (abbreviated as th).
Let g th be a function which maps a problem (parametrized by p ) to the most suitable solver (parametrized by s , which is speciﬁed by a certain integration method and stepsize control strategy).
Then for a given th, g th is determined by
[Equation]
The optimal g th for (3) makes us achieve the best for the ﬁrst performance metric Q 1 , while satisfying the threshold requirement for the second performance metric Q 2 .
Within this article, we focus on two performance metrics: accuracy (global error) and efﬁciency (time complexity).
The function g th should be pre-trained and well generalized on problems, so that in usage, we don’t need to spend extra computational cost.
2.1 Global error control for ﬁxed stepsize method
The time complexity for ﬁxed stepsize method is inverse proportional to the adopted stepsize h while the relationship between the error and h is unknown.
To provide error control in such a case, we replace abstract variables of (3) in the following way: Q 1 specially means T 1 , Q 2 means the global error (abbreviated as err), and th means the error tolerance (abbreviated as tol). [Equation]
[Equation]
Once we ﬁx the p and RK method in use, the minimal T is reached when the equality constraint is satisﬁed.
We then need to ﬁnd an inverse mapping from tol to h such that err ( p, RK , h ) = tol.
Since the local truncation error is of the order O ( h q ) where q is the order of the RK method, the relationship between log h and log( err ) is approximately linear.
Therefore, we assume the inverse mapping has the following form
[Equation]
For a given problem parametrized by p , the intercept function C ( p ) (represented by a neural network) and the slope k can be learned after obtaining multiple triads of ( p i , h i , err i ) .
In latter experiments, we show that (5) performs better than ﬁtting the functional relationship from ( h, err ) to h directly by an MLP with the same model complexity.
The loss function can be constructed by MSE or the quantile loss.
When more strict error control is preferred, the latter loss is adopted for a small quantile value q .
In inference stage, the learned model is used directly to obtain the step h given the prescribed error level tol for ﬁxed stepsize method.
To interpret the learned model visually, we consider solving a simple ODE problem by classical RK with order 4 [5].
The integration interval is [0 , t max ] , and the solution is a two-dimensional spiral.
[Equation]
For this ODE problem, the parameter p is concatenated by { b, x 0 , y 0 , t max } .
To train the model described by (5), we ﬁrst sample multiple data of ( p, h ) from bounded intervals and obtain the corresponding global errors in (cid:96) ∞ norm.
Using an MLP with one hidden layer to describe C ( p ) , we obtain the model illustrated in Figure 1 by varying one input variable at each time.
Since (5) captures the order condition, the ﬁtted curve matches the true relationship exactly as shown in (a).
For other inputs, (5) differs not far from the numerical approximation of the true relationship, as shown in (b,c).
By using the quantile loss, all the ﬁtted curves in this example problem lie below the corresponding true one, thus the strict error control is achieved in such a case.
− − − − −6 log(err) −3.
−2. −2.
−1. −1.
l o g ( h ) complexity-informed nn ground truth (a) t max = 2 π, b =1.
3.0 log(tmax) −2.
−2. −2.
−2. −2.
−1.
l o g ( h ) complexity-informed nn true (b) log( err )
= − 8 , b =−0.
1.0 log(b) −2.
−2.
−2. −2.
−2. −2.
−1.
l o g ( h ) complexity-informed nn ground truth (c) t max = 2 π, log( err ) = −
2.2 Time control for adaptive stepsize method
In this subsection, our goal is to provide time guarantee for adaptive stepsize method.
We focus on the proportional-integral stepsize controller (PIController) [6], which is a classical control strategy for explicit RK methods.
PIController uses the following formula to obtain the new step size h n + h n +1 = h n (cid:18) sc err n (cid:19)
[Equation]
(cid:16) err
[Equation]
sc (cid:17)
[Equation]
(7) β 1 , β 2 and sc (abbreviation of step control) are three parameters while sc has the greatest inﬂuence on the computational time.
When β 2 = 0 , the controller is called the integral controller (IController), which only uses err n , the local error estimation at t = t n , to determine the new stepsize h n +1 .
To provide time control for the controller in (7), let Q 1 be the global error and Q 2 be the computational time T in (3), then
[Equation]
In (8), T ∗ is the threshold number of times to evaluate f ( t, u ) in (1), which is proportional to the computational time approximately.
Similar to the methodology used in subsection 2.1, once p and RK are chosen, an inverse mapping from T ∗ to sc is required to provide computational time guarantee for adaptive stepsize method.
Though adaptive strategy is adopted, by experiments we ﬁnd the error order is still O ( h q ) approximately.
Therefore, our goal is to ﬁt the model with the following form
[Equation]
After obtaining triples of ( p i , sc i , T i ) , we can learn the model (9) in training stage and infer the sc given the evaluation times T for adaptive stepsize method.
Experiments 3.1 Fixed stepsize methods
We consider ﬁxed stepsize RK methods with order 2 (Midpoint), 3 (BS3), 4 (RKF4 [7])
and(DP5 [8]).
Besides, three different ODE problems are considered: Spiral(6), LotkaVolterra [9] and
Brusselator[10].
We generate 6400 data for training, and the size of the test dataset is 1280.
The evaluation metric is R 2 (coefﬁcient of determination) on test dataset.
The quantile loss with quantile value 10% is used to guarantee the error does not surpass the prescribed error level with larger probability.
All neural networks used in this experiment are equipped with one hidden layer with size no more than 3.
From Table 1 we see that using the complexity-informed model in (5) performs better than using MLP or decision-tree based ﬁtting with the signiﬁcance level around 5%.
Another observation is that incorporating the different RK methods as input to the network model does not decrease the performance of the ﬁtting, as shown in the column of "All" of Table 1.
The RK methods are treated as categorical features and pre-processed by one-hot embedding before training.
3.0 log(tmax) −2.
−2.6 −2.
−2.2 −2.
−1.8 −1.
−1.
l o g ( h ) complexity-informed nn MLP decision tree true (a) t max = 2 π, b =−0.
1.0 log(b) −2.
−2.
−1. −1.
−1.
l o g ( h ) complexity-informed nn MLP decision tree gro nd tr th (b) log( err ) = − 8 , b =complexity-informed nn MLP decision tree ground truth − − − − −6 log(err)
−5.0 −4.
−4.0 −3.
−3.0 −2.
−2.0 −1.
l o g ( h ) (c) t max = 2 π, log( err ) = −
The ﬁtted result of different methods are drawn in Figure 2, from which we can see that the curve by the complexity-informed neural network is nearest to that of the ground truth.
[Table]
LotkaVolterra
[Table]
Brusselator
[Table]
To verify that the predicted stepsize can achieve the desirable error level
[Table]
we integrate the model into existing ODE solvers and obtain the work-precision diagram for different RK methods with ﬁxed
[Figure]
stepsize strategy.
From Figure 3a, we see that for each error level ( 10 − 5 to 10 − 2 ), the actual error (shown in each mark on the curve) is near the left hand side of the corresponding dotted grey line.
3.2 Adaptive stepsize methods
Using the same network architecture as complexity-informed in 3.1 and MSE loss, we obtain the ﬁtting results for adaptive stepsize methods with different controllers, which are listed in Table 2.
[Table]
From Table 2, we see that the model (9) has similar ﬁtting ability for different controllers and ﬁts better for lower order RK methods.
Furthermore, given the expected time T ∗ , we can use the model (9) to get sc and solve the ODE problem to obtain the actual T .
Then we verify whether the actual evaluation time is equal to the expected time by plotting Figure 3b.
We see that all lines are near T =
T ∗ approximately, thus demonstrating the feasibility of our method.
Future Work
Two directions of further improvement will be considered in the future.
Firstly, for now we choose the most decisive variable to guide the ﬁtting of user-oriented target.
To further improve the accuracyefﬁciency tradeoff, we should include as many solver parameters as possible, such as the order of
RK method in (4), and β 1 , β 2 in (7).
Besides, uniform random sampling is adopted to obtain training data for a speciﬁc ODE problem in this paper.
In the future, we will explore how to sample more efﬁciently while keeping the training accuracy, which holds the potential to signiﬁcantly improve the learning accuracy and efﬁciency, especially when the ODE systems are of high dimensions.