N ON -L INEAR O PERATOR A PPROXIMATIONS FOR I NITIAL V ALUE P ROBLEMS
A BSTRACT
Time-evolution of partial differential equations is fundamental for modeling several complex dynamical processes and events forecasting, but the operators associated with such problems are non-linear.
We propose a Pad´e approximation based exponential neural operator scheme for efficiently learning the map between a given initial condition and the activities at a later time.
The multiwavelets bases are used for space discretization.
By explicitly embedding the exponential operators in the model, we reduce the training parameters and make it more data-efficient which is essential in dealing with scarce and noisy real-world datasets.
The Pad´e exponential operator uses a recurrent structure with shared parameters to model the non-linearity compared to recent neural operators that rely on using multiple linear operator layers in succession.
We show theoretically that the gradients associated with the recurrent Pad ´ e network are bounded across the recurrent horizon.
We perform experiments on non-linear systems such as Korteweg-de Vries (KdV) and Kuramoto–Sivashinsky (KS) equations to show that the proposed approach achieves the best performance and at the same time is data-efficient.
We also show that urgent real-world problems like epidemic forecasting (for example, COVID19) can be formulated as a 2D time-varying operator problem.
The proposed Pad ´ e exponential operators yield better prediction results ( 53% ( 52% ) better MAE than best neural operator (non-neural operator deep learning model)) compared to state-of-the-art forecasting models.
I NTRODUCTION
Predicting the future states using the current conditions is a fundamental problem in machine learning, robotics, autonomous aerial / ground / underwater systems and cyber-physical systems (Xue & Bogdan (2017)).
Such problems fall under the umbrella of a common term, the “Initial Value Problems” (IVPs).
The basic structure of IVP involves a first-order time-evolution along with nonlinear operators.
The class of IVPs spans the domain of physics (modeling gravitational waves (Lovelace, 2021)), neuroscience (Hodgkin-Huxley model (Zhang et al., 2020)), engineering (fluid dynamics (Wendt, 2008)), water waves (tsunami (Elbanna et al., 2021)), mean field games (Ruthotto et al., 2020; Bogdan & Marculescu, 2011), to list just a few.
Within the current pandemic context, the applications areas like epidemiology (Kermack–McKendrick model (Kermack et al., 1991; Diekmann et al., 2021)) are of tremendous interest.
Operators
The use of deep learning to solve the IVP like problems for predictions has been exploiting within the framework of convolutional neural networks (CNNs)
(Bhatnagar et al., 2019; Guo et al., 2016), and time-evolution by employing multiple layers (Khoo et al., 2020).
The multi-layered deep networks with CNNs are suitable to solve problems with a large number of training samples.
Moreover, because of the image-regression like structures, such models are restricted to the specifications of the input size.
Another research direction aims at solving and modeling the partial differential equations (PDEs) versions of the IVPs for a given instance.
The works of (Kochkov et al., 2021) model the IVP solution as NNs for modeling the turbulent flows.
Along the same lines, we have physics-informed neural networks (PINNs) (Raissi et al., 2019; Wang et al., 2021b) that utilize PDE structure for defining the loss functions.
Such models are not applicable within the context of a complete data-driven scenario, or for the setups where the exact PDE structure is not known, for example, modeling the climate, epidemic, or unknown physical and chemical phenomena.
Finally, we have the works of Neural Operators that are completely data-driven and input-resolution independent schemes (Li et al., 2020b;c;a; Gupta et al., 2021; Bhattacharya et al., 2020; Patel et al., 2021).
Most of these approaches tried to efficiently work with the integral kernel operators, for example, Graph Nystr¨om sampling in (Li et al., 2020b), convolution approximation in (Li et al., 2020a), or the multiwavelets compression in (Gupta et al., 2021).
Apart from solving non-homogeneous linear differential equations, the PDE operators are mostly non-linear.
To tackle the non-linear behavior, these prior works use a multi-cell architecture with non-linearity (for example, ReLU).
To work with the IVP like problems, and be data-efficient, we aim to adopt explicitly the non-linear operator (exponential) that appears in the IVP solutions.
Exponential Operators
The exponential of linear transformation has been a subject of research for the last 150 years (Laguerre, 1898).
In the simplest form, the exponential operator appears as a solution of: dydt “ at, y p 0 q “ y 0 as y p t q “ e at y 0 (for more general examples, see Table 1).
With applications ranging from control systems theory (converting continuous to discrete systems) to solving partial differential equations (Cox & Matthews, 2002; Kassam & Trefethen, 2005), the exponential function of operators is a subject of active research.
In deep learning, the exponential function to model non-linearity is used in (Andoni et al., 2014).
Recently, the exponential operators have also been explored in the field of computer vision for generative flows in (Hoogeboom et al., 2020).
Pad´e Approximation
Although one approach to implementing an exponential operator could be attained through a Taylor series representation, this operator function is prone to errors (Abramowitz & Stegun, 1965).
Scale-and-squaring (SSQ) methods are commonly suggested approaches to deal with the errors (Lawson, 1967).
In addition to SSQ, the Pad ´ e approximation which represents an analytic function as the ratio of polynomials achieves state-of-the-art accuracy in computing exponential operators (Fasi & Higham, 2019).
Industry standard numerical toolboxes (for example,
MATLAB, SciPy) use the Pad ´ e approximation based approach to compute the matrix exponential expm (Al-Mohy & Higham, 2009).
Matrix exponential via Pad ´ e representation requires dense matrix computations (for example, inverse and higher-order polynomials).
Such operations are not numerically feasible, in-general, for the inputs with large size.
However, the commonly used operators like convolution (possibly, multi-layered) have parameters that are fixed beforehand and are much less than the input dimension.
A suitable approach, therefore, is a neural architecture based
Pad ´ e approximation.
Our strategy, in this work, is to explicitly embed the exponential operators in the neural operator architecture for dealing with the IVP like datasets.
The exponential operators are non-linear, and therefore, this removes the requirement of having multi-cell linear integral operator layers.
While with sufficient data in-hand, the proposed approach may work similarly to the existing neural operators with a large number of training parameters.
However, this is seldom a feasible scenario for the expensive real-world experiments, or on-going recent issues like COVID19 prediction.
Here, the current work is helpful in providing data-efficiency analytics, and is useful in dealing with scarce and noisy datasets (see Section 3.3).
To the advantage of Pad´e approximation, the exponential of a given operator can be computed with the pre-defined coefficients (see Section 2.3) and a recurrent polynomial mechanism.
Our Contributions
The main novel contributions of this work are summarized as follows: ( i ) For the IVPs, we propose to embed the exponential operators in the neural operator learning mechanism.
( ii )
By using the Pad´e approximation, we compute the exponential of the operator using a novel recurrent neural architecture that also eliminates the need for matrix inversion.
( iii )
We theoretically demonstrate that the proposed recurrent scheme, using the Pad´e coefficients, have bounded gradients with respect to (w.r.t.)
the model parameters across the recurrent horizon.
( iv ) We demonstrate the data-efficiency on the synthetic 1D datasets of Korteweg-de Vries (KdV) and Kuramoto–Sivashinsky (KS) equations, where with less parameters we achieve state-of-the-art performance.
( v ) We formulate and investigate the epidemic forecasting as a 2D time-varying neural operator problem, and show [Table] that for real-world noisy and scarce data, the proposed model outperforms the best neural operator architectures by 53% and best non-neural operator schemes by 52%.
O PERATORS FOR I NITIAL V ALUE P ROBLEM
We formalize the partial differential equations (PDEs) version of the Initial Value Problem studied in this work in Section 2.1.
Section 2.2 summarizes the multi-resolution analysis using multiwavelets for space-discretization.
Section 2.3 describes the proposed use of canonical exponential operators and presents a novel architecture using Pad ´ e approximation.
2. I NITIAL V ALUE P ROBLEM
The initial value problem (IVP) for PDEs can be written in its general form as follows.
[Equation]
where, u t is the first-order time derivative of u , F is a time-varying differential operator (non-linear in-general) such that F : R ` Y t
u ˆ B Ñ B with B being a Banach space.
Usually, the system in eq.
(1) is required to satisfy a boundary condition such that Bu p x, t q “ 0 , x P B Ω @ t in the solution horizon, and B Ω is the boundary of the computational region Ω with B some linear function.
Pertaining to our work, the operator map problem for IVP can be formally defined as follows.
Operator Problem Given A and U as two Sobolev spaces H s,p with s ą 0 , p “ 2 , an operator T is such that T : A Ñ U .
For a given τ ą 0 and two functions u 0 p x q and u p x, τ q , in this work, we take the operator map as T u 0 p x q “ u p x, τ q with x P Ω .
Table 1 summarizes a few examples of the IVP and their solutions.
The exponential operators are ubiquitous in the IVP solutions and, therefore, are important to study.
One issue, however, is that the exponential operators are non-linear and unlike convolution like operators, there does not exist a general way to diagonalize them (Fourier transform diagonalizes convolution operator) for an efficient representation.
Previous work on neural operators (Li et al., 2020c;a; Gupta et al., 2021) modeled the non-linear operators in one way or another by using multiple canonical integral operators along with non-linearity (for example, ReLU).
In this work, we directly produce an exponential operator approximation.
First, we discuss an efficient basis (multiwavelets) for space discretization of the input / output functions in Section 2.2.
2. M ULTI -R ESOLUTION A NALYSIS
The multi-resolution analysis (MRA) aims at projecting a function to a basis over multiple scales.
The wavelet basis (e.g., Haar, Daubechies) are some popular examples.
Multiwavelets further this operation by using the family of orthogonal polynomials (OPs), for example, Legendre polynomials for an efficient representation over a finite interval (Alpert et al., 2002).
The multiwavelets are wide useful in the sparse representation of the integral operators with smooth kernels.
In addition, the multiwavelets also sparsify the exponential functions of the strictly elliptic operators (Beylkin &
Keiser, 1997).
However, we do not rely on this assumption in this work.
Here, we briefly introduce the MRA and refer the reader to Gupta et al.
for a more detailed discussion.
Notation We begin by defining the space of finite interval polynomials as V kn “ t f | f are polynomials of degree ă k defined over interval p 2 ´ n l, 2 ´ n p l ` 1 qq for all l “ 0 , 1 , . . .
, 2 n ´ 1 , and assumes 0 elsewhere u .
The V kn are contained in each other for subsequent n or,
[Equation]
The orthogonal component of these polynomial spaces is termed as multiwavelet space W kn and are defined such that
[Equation]
The orthonormal basis of V k 0 are OPs ϕ 0 , ϕ 1 , . . .
ϕ k ´
and we have used appropriately normalized shifted Legendre Polynomials in this work.
The basis for V kn and W kn are ϕ njl p x q “ 2 n { 2 ϕ j p 2 n x ´ l q and ψ njl p x q “ 2 n { 2 ψ j p 2 n x ´ l q , respectively, for l “ 0 , 1 , . . .
, 2 n ´ 1 and j “ 0 , 1 , . . .
, k ´ 1 .
Finally, an important trick for representing the operator T in the multiwavelet basis is called nonstandard (NS) form (Beylkin et al., 1991).
The NS form decouples the interactions of the scales and is useful in obtaining an efficient numerical procedure.
Using NS form, the projection of operator T is expanded using a telescopic sum as follows.
[Equation]
where, P n : H s, 2 Ñ V kn is the projection operator, T n “ P n TP n , Q n : H s, 2 Ñ W kn such that Q n “ P n ´ P n ´ 1 , and L is the coarsest scale under consideration p
L ě 0 q .
Therefore, the NS form of the operator is a collection of the triplets t
A i , B i , C
i u ni “ L ` 1 and P L TP L with A i “ Q i
TQ
i , B i “ Q i TP
i ´ 1 and C
i “ P i ´
1 TQ i .
In this work, we aim to model A i , B i , C i as the exponential operators to better learn the IVP by explicitly embedding the non-linear operators into the multiwavelet transformation.
This is not straightforward due to the non-linearity of exponential functions.
We are now in shape to present the main contribution of the current work in the Section 2.
where we discuss an implementable neural approximation of the exponential operators.
2. E XPONENTIAL O PERATOR A PPROXIMATIONS
Due to the nature of first-order time-evolution equations, the exponential operators appear in the solution of IVP as discussed in Section 2.1.
Being an analytic function, the exponential also assumes a Taylor series expansion.
However, the approximation error by truncation is (Abramowitz & Stegun, 1965) and may require a large number of coefficients.
We now discuss a better approximation for the non-linear functions.
Pad´e Approximation Given an analytic function f p z q , z P C at 0 and let p, q P N , the r p { q s Pad´e approximation of f at 0 is a rational polynomial r pq p
z q “ A pq p z
q{ B pq p z q , where A pq p z q and B pq p z q are polynomials of degree p and q , respectively.
For our work, f p z q “ e z , and the r p { q s Pad´e approximation to the exponential function is written as A pq p z q “ ř pj “ 0 a j z j and B pq p z q “ ř qj “ 0 b j z j , where
[Equation]
Note that, a 0 “ b 0 “ 1 , and we now discuss the exponential of operators.
Given an operator L (possibly non-linear), the r p { q s Pad´e approximation for e L can be written as
[Equation]
For L being a linear transformation, the inverse in eq.
(6) is computed as a matrix inverse, for example when evaluating a matrix exponential.
Even convolution operator can be represented as the circulant matrix (shift-invariant kernel) operation.
But storing the big circulant matrix is not a numerically efficient solution.
Instead, we now discuss a neural architecture that emulates the Pad´e approximation for operators in eq.
(6) while avoiding taking the inverse.
Pad´e Exponential Model We compute the operator exponential using the r p { q s Pad´e approximation via a recurrent neural architecture in Figure 1.
First, the denominator polynomial B pq p L q is evaluated using the left recurrent network.
Second, the output is passed through a non-linear layer which we implement as v “ σ p Wu ` b q with σ p .
q being the ReLU function.
Note that the non-linearity layer is applied to the input channels, and therefore, its size is independent of the input spatial dimension.
Finally, the output of the non-linear layer is passed through another recurrent network implementing the numerator polynomial A pq p L q .
Note that, both polynomial recurrent networks use same operator
L , or in other words, the parameters are shared across the network.
With a j , b j fixed-beforehand, the total trainable parameters of the Pad´e exponential model are t θ L , W, b u .
One issue with the recurrent architectures is the possibility of gradients explosion when evaluated over a large horizon.
In such cases, techniques like gradient clipping is used as a workaround.
For a given p, q , the proposed network runs the recurrent loops for a total of p ` q ´ 2 .
We show that the proposed r p { q s Pad´e network does not suffer from the issue of gradient explosion, and the boundedness of the gradients is established through the following result:
Theorem 1.
Given a linear operator L “ L p θ L q , a non-linearity layer v “ σ p Wu ` b q , and p, q P N , at points of differentiability, the gradients of the operation x ÞÑ y “ F p x ; θ L , W, b q : “ r p { q s e L p x q using the r p { q s Pad´e network in Figure 1 are bounded in operator norm by
[Equation]
The detailed proof is provided in the Supplementary Materials (Appendix E).
Note that, n θ “ | θ L | is such that n θ !
M , where M is the dimension of the input.
For example, the convolution operator has a fixed kernel independent of the size of the input, the Fourier-based convolution in (Li et al., 2020a) has k m Fourier modes independent of the input dimension.
Next, as noted earlier, the non-linearity layer σ p Wu ` b q is applied to the input channels (instead of input spatial dimensions), therefore, the gradient bounds do not scale with high input-resolutions, apart from the dependence from ∥ x ∥which could be pre-normalized.
Finally, the Pad´e neural model is integrated with the multiwavelet transform by substituting each of A, B, C , from Section 2.2, with r p { q s e L A , r p { q s e L A , r p { q s e L A .
The complete flow-diagram by plugging-in the Pad ´ e model is shown in Appendix D.
E XPERIMENTS
Here, we empirically evaluate the proposed model and compare against the existing approaches.
We consider several synthetic PDE datasets as well as a real-world example of pandemic prediction (COVID-19).
For the data, a similar input/output structure is used as in the recent works of neural operator architectures.
Specifically, the input function u 0 p x q and the output function u p τ, x q for some pre-specified τ ą 0 , are evaluated at M discretized locations of the domain Ω .
This yields a single training sample of p u
p
x
i q , u p τ, x i q , x i P Ω , 1 ď i ď M .
In total, we take N training samples, and unless stated otherwise, N “ 1000 and we test on 200 samples for the synthetic datasets.
Pad´e Model Implementation
The operator L in Figure 1 is fixed as a single-layered convolution operator for 1D datasets, and 2-layered convolution for 2D datasets.
For getting the input/output operator mapping, the multiwavelet transform is used only for discretizing the spatial domain.
The
Pad´e neural model easily fits into the sockets of the multiwavelet transformation based neural operator as sketched in Figure 7 (Appendix D).
The multiwavelet filters are obtained using shifted Legendre
OPs with degree k “ 4 .
In contrast to the work of Gupta et al.
(2021), only a single cell of [Equation] [Figure] multiwavelet transform is used in the current work because the non-linearity in the operator is explicit via Pad ´ e neural operator.
This saves a lot of trainable parameters and yields a compact model which is suitable when dealing with scarce noisy real-world data as we see in Section 3.3.
The numerator / denominator polynomials degrees p { q for the Pad ´ e approximation is fixed as p p, q q “ p 5 , 6 q for 1D models and p 4 , 2 q for 2D models.
While the authors do not advocate that this is the best possible choice, an ablation study for 1D data is performed in Appendix B.1.
Benchmark Neural Operators We compare against the recently proposed neural operator works with the proposed Pad ´ e exponential model ( Pad´e Exp ).
The graph neural operator ( GNO ) was proposed in (Li et al., 2020b).
A multi-level version of the graph neural network ( MGNO ) in (Li et al., 2020c).
LNO A low-rank representation of the integral operator kernel and then using multiple layers with non-linearity, which also emulates unstacked DeepONet (Lu et al., 2020).
A convolution approximation to the canonical integral kernel and then Fourier transform to diagonalize in (Li et al., 2020a) as FNO .
MWT Leg utilize multiwavelets for spatial projections using Legendre OPs and uses multi-cell structure along with ReLU non-linearity.
The Pad´e Exp model delivers state-of-the-art (Sota) performance on a range of datasets, both synthetic and real-world.
With less number of parameters, the proposed model shows promise for small datasets as shown in Section 3.3.
Training parameters All neural operator models are trained using Adam optimizer with a learning rate of 0 . 001 and decay of 0 .
after every 100 steps.
The loss function is taken as the relative L error.
For synthetic datasets we train for a total of 500 epochs and for real-world COVID-19 dataset we train for a total of 750 epochs.
All experiments are done on an Nvidia A 100 40 GB GPUs.
3. K ORTEWEG - DE V RIES E QUATION
The Korteweg-de Vries (KdV) equation is a one-dimensional non-linear PDE used to model the non-linear shallow water waves.
For a given field u p x, t q , the KdV PDE takes the following form:
[Equation]
[Equation]
For Table 1 notations, we have L “ ´ B 3 B x 3 , f p u q “ ´ 0 .
25 u 2 , and N “ BB x .
The neural operator learns the mapping of the initial condition u 0 p x q to the solutions u p x, t “ 1 q .
The initial condition is generated in Gaussian random fields according to u 0 „ N p 0 , 7 4 p´ ∆ ` 7 2 I q ´ 2 .
5 q with periodic boundary conditions.
The data set is obtained by solving the equation using the fourth-order stiff time-stepping scheme known as ETDRK4 (Cox & Matthews, 2002) with a resolution of 2 10 , and datasets with lower resolutions are obtained by sub-sampling the highest resolution dataset.
For evaluation, firstly, we vary the total training samples ( N ) for all the operator models.
We sample randomly and uniformly 5 times the training subset from the complete data N “ 1000 , and for each [Figure] Networks s =s =s
=s
=s
[Table] [Table] sampling the models are evaluated. For consistency, each model is fed the same training sub-sample
and the results are shown in Figure 2 (Left).
We see that Pad ´ e Exp model has the sharpest decay compared to the other state-of-the-art neural operators on varying N .
This shows that the proposed model is data-efficient and works well when less data is available.
Next, we also evaluate the Pad ´ e
Exp model on the complete data N “ 1000 but by varying the input resolution as shown in Table 2.
The proposed model performs consistently better for all the input resolutions.
3. K URAMOTO -S IVASHINSKY (KS) EQUATION
The Kuramoto-Sivashinsky (KS) equation is a fourth-order non-linear PDE derived to model the diffusive instabilities in a laminar flame front.
For a given field u p x, t q , the KS PDE takes the following form:
[Equation]
The KS equation is also time-advection and according to the Table 1 notations, we have: L “ ´ B 2 B x 2 ´ B 4 B x 4 , f p u q “ ´ 0 .
5 u 2 , and N “ BB x .
The neural operator learns the mapping of the initial condition u 0 p x q to the solutions u p x, t “ 1 q .
Similarly to the KdV equation in Section 3.1, the initial condition is sampled from a Gaussian random field u 0 „ N p 0 , 5 4 p´ ∆ ` 5 2 I q ´ 2 .
5 q with periodic boundary conditions.
The equation is numerically solved using chebfun package (Driscoll et al., 2014) with a resolution of 2 10 , and datasets with lower resolutions are obtained by sub-sampling the highest resolution data set.
We evaluate the proposed model on the KS equations and compare with the existing works in a similar experimental setup as for the KdV equation in Section 3.1.
The reduced training experiment results are shown in Figure 2 (Right), where we have uniformly and randomly sub-sampled the training samples from the complete dataset.
For consistency, we have evaluated all models on the same sub-sampled training set.
We again observe that the proposed Pad´e Exp model with its compact structure has the steepest decay of relative L2 error compared to the recent neural operator works.
[Table]
[Figure]
[Table]
[Figure]
By using the exponential operators
[Figure]
the IVPs can be efficiently solved as we witness from Figure 2.
Finally
[Figure]
we also evaluate the proposed model for the complete data but varying the input resolution s (see results in Table 10, Appendix B.4).
We again conclude that the proposed model attains better performance than state-of-the-art approaches for all the resolutions [Figure]
3. E PIDEMIC F ORECASTING : COVID-19 STUDY
The epidemic forecasting problem refers to the prediction of future counts of infected individuals, recovered individuals and deaths using the current observational data.
A variety of compartmental models exist to model the epidemic spread with certain assumptions (see related work, Appendix A).
The dynamics of the epidemic spread by modeling their time-evolution behavior is not exactly known, and may not apply to any problem, for example, the recent COVID-19 pandemic.
Neural operators, providing a complete data-driven approach, are capable to learn PDE agnostic maps .
Consequently, we show that the epidemic forecasting can be formulated as an operator map learning problem.
Dataset The COVID-19 data set 3 from April 12th 2020 to August 28th 2021 is provided by Johns
Hopkins University (Dong et al., 2020).
We take the data of 50 US States, and for each state, we have the total counts of daily reported confirmed (C), recovered (R), and deaths (D).
We normalize the data of each state by their total population.
Therefore, we have a daily collection of 2-dimensional data of size 50 ˆ 3 .
Deep learning Benchmarks
In addition to the neural operators, we also compare against the stateof-the-art deep learning techniques: an auto-regressive fully connected ( FC ) network, Sequence to
Sequence ( Seq2Seq ) (Salinas et al., 2019; Rangapuram et al., 2018) and Transformers (Vaswani et al., 2017) utilizing encoder-decoder structure.
For Seq2Seq, we have used LSTM architecture for encoder and decoder.
Neural ODE (Chen et al., 2019) utilizing latent ODE for time-series forecasting.
Operator Map
The operator task is to learn the map between the 14 consecutive counts (C, R, D) to next 7 days data for each of the 50 US states.
Let d t be the 50 ˆ 3 array on day- t , then the operator map can be written as follows.
[Equation]
Forecasting The COVID19 forecasting benchmarks are presented in Table 3.
Due to data scarcity (484 samples in total), we do a 10-fold resampling of the dataset to obtain train/test samples and the averaged results are presented for all models.
We see that the proposed Pad ´ e exponential model achieves better performance than existing approaches especially in the presence of scarce and noisy data setup.
The Seq2Seq performs best among all non-neural operator models.
In terms of the total mean averaged error (MAE) (for C, R, and D counts), the Pad ´ e exponential achieves a 53% improvement over the best neural operator (MWT Leg), and 52% over the best non-neural operator model (Seq2Seq).
The percentage improvement over the FC model and the difference between the total model parameters w.r.t.
FC are shown in the last column.
A sample forecasting is shown for
US states in Figure 3.
We also show the corner cases (best and worst test sample) predictions in the
Figure 8 (Appendix B.5) along with the best/worst prediction states.
A BLATION E XPERIMENTS
The ablation experiments are presented in the Supplementary Materials (Appendix B.1).
The following experiments are performed; ( i )
Comparison between the proposed Pad ´ e approximation vs Taylor
Series neural operator, ( ii ) the variations of p, q for the r p { q s Pad ´ e approximation model in Figure 1, and ( iii ) variation of the non-linearity module in the Pad ´ e neural model.
F UTURE D IRECTIONS AND C ONCLUSION
Time-evolution analysis of PDEs and forecasting the future states from a set of observations is fundamental for studying a wide range of complex systems in physics, chemistry, geoscience, neuroscience, system biology, social, political, and climate sciences.
In many such contexts, we need to solve an initial value problem consisting of a first-order time-evolution of several nonlinear spatial operators.
To efficiently solve these initial value problems while also overcoming data science challenges (e.g., scarcity in the number of samples, samples corrupted by unknown noise types and sources), we proposed a combined multiwavelet and Pad´e approximation based exponential neural operator architecture.
The proposed model has order-magnitude fewer parameters (see Table 3) and attains data-efficiency.
The IVPs deal with first-order time evolution, and thus, the exponential operator appears.
In a more general setup, the higher-order time derivatives, or even fractional time derivatives should be considered to model the non-Markovian dynamics.
In such cases, Reisz transform like approach helps.
This is an interesting future direction for neural operator design, where data helps identifying the time-evolution order. /COVID-
A CKNOWLEDGEMENT
We are thankful to the anonymous reviewers for providing their valuable feedback which improved the manuscript.
We gratefully acknowledge the support by the National Science Foundation Career award under Grant No. CPS/CNS-1453860, the NSF award under Grant CCF-1837131, MCB-1936775,
CNS-1932620, the U.S. Army Research Office (ARO) under Grant No. W911NF-17-1-0076, the
Okawa Foundation award, the Defense Advanced Research Projects Agency (DARPA)
Young Faculty
Award and DARPA Director Award under Grant No. N66001-17-1-4044, an Intel faculty award,
a
Northrop Grumman grant, and Google cloud.
A part of this work used the Extreme Science and
Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562.
The third author has been supported in part by a NSF award under grant
DMS-2108900 and by the Simons Foundation.
The views, opinions, and/or findings contained in this article are those of the authors and should not be interpreted as representing the official views or policies, either expressed or implied by the Defense Advanced Research Projects Agency, the Army
Research Office, the Department of Defense or the National Science Foundation.
Epidemic Forecasting In Epidemiology, the compartmental models are very common, where the population is assigned into discrete categories (for example, Susceptible, Infected, Recovered), also termed as SIR models which date back to 1927 (Kermack et al., 1991; Diekmann et al., 2021).
Along the lines, several works exist that model the time-evolution of epidemic using various assumptions.
The work of Chen et al.
(2020) uses a ridge regression based prediction.
An extension of SEIR (E for
Exposed) compartments to sub-populations moving across different places in (Pei & Shaman, 2020).
A convolution filter based approach in (Wang et al., 2020), where the total infected cases are modeled as convolution of novel cases with proportion of infected counts over time.
A susceptible EIR model in (Zou et al., 2020b) incorporating the unreported cases.
The work of Davis et al.
(2020) which simultaneously models the transmission rate among US states.
A physics-based model in (Wang et al., 2021a) that uses auto-differentiation using Runga-Kutta methods to estimate the model parameters.
B A DDITIONAL R ESULTS B. A BLATION S TUDY
Taylor vs Pad´e Exponential Approximation We compare the Pad´e approximation based neural model in Figure 1 vs Taylor series approximation for evaluating the exponential of the operator L .
The Taylor series version is written as
[Equation]
where, L t is the truncated length of the series.
We evaluate the truncated Taylor series approximation vs r p { q s Pad´e approximation in Figure 1 for same number of maximum polynomial degree, i.e.,
L t “ max p p, q q .
For both models, we give input with varied degree of fluctuations (to make the data more challenging) and the output is governed by the KdV equation (see Section 3.1).
The fluctuated input is sampled from Random fields Filip et al.
(2019) with controllable parameter λ with lower values attaining higher fluctuations and the input u 0 p x q reaches the Brownian motion limit with λ Ñ 0 .
We see in Table 4 that the Pad´e approximation model works better compared to Taylor series based expansion for incorporating the exponential operators for each λ .
L’Hospital Approximation
The exponential exp p x q can also be approximated as lim n Ñ8 p 1 ` x { n q n using L’Hospital rule (Krantz, 2004).
For a given operator L , the L’Hospital based exponential operator approximation can be written as
[Equation]
n
[Table]
times , where, I is the identity operator and n is the length of truncation of operator compositions.
We compare L’Hospital approximation with the Pad´e model in Table 4 for the same degree of operator polynomials, i.e., n “ max p p, q q .
For the same experimental settings as taken in the Taylor approximation comparison, we see that the proposed Pad´e model performs better than the L’Hospital-based exponential approximation for all fluctuations strengths.
Varying Pad´e polynomials degree The numerator/denominator polynomials degree p, q in eq.
(5) for r p { q s Pad ´ e approximation model in Figure 1 are hyper-parameters.
We vary the p, q over a 2D grid of r 1 , 10 s ˆ r 1 , 10 s and evaluate for the KdV equation settings (s=1024) as mentioned in the
Section 3.1 and the results are shown in Figure 4.
We make the following observations, namely; (i)
For p “ q “ 1 , i.e., when there is no operator L and only the non-linear layer σ p Wu ` b q , the model has the worst performance of 0 . 1063 .
This shows the importance of operator L in the Pad´e model.
(ii)
By increasing the p, q in either direction, the performance improves as we attain better approximation with higher degree polynomials.
The best performance is achieved as p p, q q “ p 10 , 1 q .
(iii)
The performance is roughly similar in the mid-range of p and q ’s, as long as p ą 1 and q ą 3 for the 1D experiment on KdV. Note that, although larger values of p and q are preferable but they also incur the additional cost of run-time which scales linearly with p ` q as evident from the recurrent structure in
Figure 1.
[Table]
Structure of the non-linear MLP We have used the non-linear layer σ p Wu ` b q in the Pad ´ e neural model for all the experiments with σ p .
q =ReLU.
Now, we vary the depth of this layer from 1 to(each layer is followed by ReLU non-linearity) to see the individual contribution of this layer towards the final test performance.
The experiments settings are same as for Table
and we fix s “ 1024 .
Upon increasing the depth we do not see any improvements, in fact, after having three layers, the performance starts to degrade.
A possible reason could be Denominator polynomial getting affected by the deeper non-linearity layer.
B. P REDICTION AT HIGHER RESOLUTIONS
Train
Test s =
s =
s =s= 0.0423 (0.0473) 0.0440 (0.0511) 0.0450 (0.0544) s= 0.0229 (0.0315) 0.0250 (0.0374) 0.0263 (0.0427) s= 0.0124 (0.0230) 0.0148 (0.0305) 0.0162 (0.0372)
B. T RAINING /E VALUATION WITH DIFFERENT p p, q q
Due to the recurrent structure of the Pad´e approximation model in Figure 1, with only trainable parameters t θ L , W, b u , the model can be trained and tested on different values of p p, q q by just varying the recurrence loop lengths of the numerator and denominator polynomial.
We setup this experiment on the KdV equation (same setting as in Section 3.1).
The Pad´e model is trained for p p, q q “ p 5 , 6 q and p 4 , 2 q
and then tested on r 1 , 10 s ˆ r 1 , 10 s grid of p, q tuples as shown in Figure 5. [Figure] [Figure]
p 0. q     p 0.
We observe that the best performance (lowest relative L2 error) results when the train and test p p, q q matches.
In addition, we observe that for a fixed q , the best performance results when p is such that p/q ratio is similar as the training ratio.
For example, in Figure 5 (Left), we see that lowest value for each column is obtained when p { q is closest to 5 { 6 while in (Right) we see lowest value of each column when p { q “ 2 .
The boundary cases diverge because either p, q “ 1 does not have any operator L in the corresponding polynomial as evident from eq.
When trained for p p, q q “ p 4 , 2 q , the best performance for q ą 5 saturates to p “ 10 because of the maximum limit on the experimental grid p p “ 10 q .
Overall, this experiment suggests that the relative length of the polynomials during evaluation should be similar to training setting for best performance.
B. K URAMOTO -S IVASHINSKY E QUATION
Comparison of relative error for different neural operators evaluated over KS equation is presented in
Table 10.
B. COVID-19 F ORECASTING
Additional forecasting results for COVID-19 by considering the corner cases are presented in Figure 8.
B. 2D N AVIER -S TOKES E QUATIONS
The Navier-Stokes (NS) equations are 2D time-varying PDEs describing the motion of viscous fluid substances.
The NS equations can describe many physical processes and have wide range of practical uses.
In this paper, to compare with the state-of-the-art models (Li et al., 2020a; Gupta et al., 2021) under the same conditions, we use the same data sets that have been published in (Li et al., 2020a), where the NS equations take the following form:
[Equation]
where u is the velocity, w is the vorticity such that w “ ∇ ˆ u .
The incompressible flow is modeled via divergence condition as ∇ ¨ u p x, t q “ 0 .
We set the experiments to let the neural operator map the first 10 time units to last T ´ 10 time units of vorticity w .
The initial condition is generated in Gaussian random fields according to w 0 „
N p 0 , 7 32 p´ ∆
` 7 2 I q ´ 2 .
5 q with periodic boundary conditions and the forcing function is f p x q “ 0 .
1 p sin p 2 π p x 1 ` x 2 qq ` cos p 2 π p x 1 ` x 2 qqq .
We experiment with different viscosities ν , final time T , and the number of training pairs N : p i q ν “ 1 e ´ 3 , T “ 50 , [Table]
[Table] [Table] [Table] N “ 1000 ; p ii q ν “ 1 e ´ 4 , T “ 30 , N “ 1000 ; p iii
[Table]
ν “ 1 e ´ 4 , T “ 30 , N
[Table] [Table]
ν “ 1 e ´ 5 , T “ 20 , N “ 1000 on a 64 ˆ 64 grid
[Table]
The time-varying 2D data is modeled as 3D operator learning problem. We implemented the Pad´e exponential model by taking L as 2-layered 3D CNNs with ReLU non-linearity and p “ q “ 4 in      epochs
[Figure]
Figure 1.
A total of 4 layers of multiwavelet skeleton (Figure 7) with k “ 3 are concatenated using ReLU nonlinearity.
The results are reported in Table 7, and we observe that Pad ´ e model performs well compared to the recent state-of-the-art neural operator approaches.
For the less data setup of
N “ 1000 with ν “ 1 e ´ 4 , we see in Figure 6 that the Pad´e Exp model quickly converges to the lowest value compared to the other neural operator approaches.
B. N UMERICAL V ALUES
The numerical values for the training samples variation experiment for the
KdV and KS equation in the Figureare shown in Table 8 and Table 9, respectively.
[Table]
C N OTATIONS
[Table]
Learning
[Table]
spaces H s,p Sobolev spaces such that constituent functions and their weak derivatives
[Table]
order s have finite L p norms
[Table]
V kn t f | f are polynomials of degree ă k defined over interval p 2 ´ n l, 2 ´ n p l ` 1 qq for all l “ 0 , 1 , . .
, 2 n ´ 1 , and assumes 0 elsewhere u
W kn
Orthogonal space to V kn such that W kn À V kn “ V kn `
P n
Projection operator such that P n : H s, 2 Ñ V kn
Q n
Projection operator such that Q n : H s, 2 Ñ W kn
L
Coarsest scale of the multiwavelet transform
Pad´e Approximation } . }
Operator norm } . }
Euclidean L-2 norm r p { q s f
Pad ´ e approximation of function f with numerator/denominator polynomial degree p/q, respectively θ
L
Parameters used to represent the operator L θ n
Number of parameters used to represent the operator L x ÞÑ y “ F p x ; Θ q
Mapping from x to y using function F with parameters set Θ
D M ULTIWAVELET E XPONENTIAL O PERATOR A RCHITECTURE
The Pad´e neural operator based multiwavelet transform model is shown in Figure 7.
The input to the model is s p n ` 1 q , where n “ log p M q , and the output is U p n q s , where n is the finest scale (or the log of input resolution).
For a detailed description of multiwavelet transform, we refer the reader to (Gupta et al., 2021).
E P ROOF OF T HEOREM
First note that, for the Pad´e approximation r p { q s e x , the polynomial coefficients in eq.
(5), we have
[Equation]
+ + and similarly,
[Equation]
For the simplicity of notation: let A p z q
[Equation]
ř
[Equation]
C
[Equation]
qj
[Equation]
| b | j z j ´ 1 .
Note | B
pq p z q| ď C p| z |q and | B 1 pq p z q| ď C 1 p| z |q .
Using eq.
(13), we can further write that
[Equation]
Further let d and n θ denote the dimensions of x and θ respectively: x, y, u, v P R d and θ : “ θ L P R n θ .
For the operation x ÞÑ y , we denote y “ F p x ; θ L , W, b q : “ r p { q s e L p x q .
Then y “ A p L q v , v “ σ p Wu ` b q , u “ B p L q x , and [Equation] B y
[Equation]
All matrix norms are spectral (i.e., the largest singular value), unless otherwise indicated as in equation (25), whereas all vectors norms are Euclidean (i.e., the L2-norm).
Denote by B g,θ “ ř n θ j “ 1 g j B B θ j , the differential operator induced by g .
The quantity of interest is |B g,θ x y, h y| .
Next, by chain rule
[Equation]
[Equation]
where D g is a diagonal matrix of 1’s and 0’s depending upon the signatures of entries in Wu ` b (see for instance the computation of Lipschitz constants in Zou et al.
(2020a)).
Hence } D g } “ 1 .
Since σ (ReLU) is contractive with Lipschitz constant 1, } v } 2 ď } W }} u } 2 `} b } 2 ď } W }}
B p L q}} x } 2 `} b } 2 .
Substituting back in eq.
(17) we obtain
[Equation]
Using eq.
, the spectral norms are bounded further by
[Equation]
[Figure] Next, by using the product rule, we obtain
[Equation]
Similarly,
[Equation]
Using eq.
[Equation]
[Equation]
and (23)
[Equation]
we obtain the bound for eq.
(16) as
[Equation] [Equation]
which is used to write the eq.
(7) in Theorem 1.
Similarly, the analysis is extended as
[Equation]
and
[Equation]
(26) where, D Φ and D h are diagonal matrices with 1’s and 0’s.
Using eq.
(25) and (26) [Equation] we obtain the eq.
(8) and (9) in the Theorem 1, respectively
[Equation]