F OURIER N EURAL O PERATOR FOR P ARAMETRIC P ARTIAL D IFFERENTIAL E QUATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
The classical development of neural networks has primarily focused on learning mappings between ﬁnite-dimensional Euclidean spaces.
Recently, this has been generalized to neural operators that learn mappings between function spaces.
For partial differential equations (PDEs), neural operators directly learn the mapping from any functional parametric dependence to the solution.
Thus, they learn an entire family of PDEs, in contrast to classical methods which solve one instance of the equation.
In this work, we formulate a new neural operator by parameterizing the integral kernel directly in Fourier space, allowing for an expressive and efﬁcient architecture.
We perform experiments on Burgers’ equation, Darcy ﬂow, and Navier-Stokes equation.
The Fourier neural operator is the ﬁrst ML-based method to successfully model turbulent ﬂows with zero-shot super-resolution.
It is up to three orders of magnitude faster compared to traditional PDE solvers. Additionally, it achieves superior accuracy compared to previous learning-based solvers under ﬁxed resolution.
I NTRODUCTION
Many problems in science and engineering involve solving complex partial differential equation (PDE) systems repeatedly for different values of some parameters.
Examples arise in molecular dynamics, micro-mechanics, and turbulent ﬂows.
Often such systems require ﬁne discretization in order to capture the phenomenon being modeled.
As a consequence, traditional numerical solvers are slow and sometimes inefﬁcient.
For example, when designing materials such as airfoils, one needs to solve the associated inverse problem where thousands of evaluations of the forward model are needed.
A fast method can make such problems feasible.
Conventional solvers vs. Data-driven methods.
Traditional solvers such as ﬁnite element methods (FEM) and ﬁnite difference methods (FDM) solve the equation by discretizing the space.
Therefore, they impose a trade-off on the resolution: coarse grids are fast but less accurate; ﬁne grids are accurate but slow.
Complex PDE systems, as described above, usually require a very ﬁne discretization, and therefore very challenging and time-consuming for traditional solvers.
On the other hand, data-driven methods can directly learn the trajectory of the family of equations from the data.
As a result, the learning-based method can be orders of magnitude faster than the conventional solvers.
Machine learning methods may hold the key to revolutionizing scientiﬁc disciplines by providing fast solvers that approximate or enhance traditional ones (Raissi et al., 2019;
Li et al., 2020b;
Jiang et al., 2020; Greenfeld et al., 2019).
However, classical neural networks map between ﬁnitedimensional spaces and can therefore only learn solutions tied to a speciﬁc discretization.
This is often a limitation for practical applications and therefore the development of mesh-invariant neural networks is required.
We ﬁrst outline two mainstream neural network-based approaches for PDEs – the ﬁnite-dimensional operators and Neural-FEM.
Finite-dimensional operators.
These approaches parameterize the solution operator as a deep convolutional neural network between ﬁnite-dimensional Euclidean spaces Guo et al.
; Zhu & Zabaras (2018); Adler & Oktem (2017); Bhatnagar et al.
; Khoo et al.
Such approaches are, by deﬁnition, mesh-dependent and will need modiﬁcations and tuning for different resolutions and discretizations in order to achieve consistent error (if at all possible).
Furthermore,
[Figure]
these approaches are limited to the discretization size and geometry of the training data and hence, it is not possible to query solutions at new points in the domain.
In contrast, we show, for our method, both invariance of the error to grid resolution, and the ability to transfer the solution between meshes.
Neural-FEM.
The second approach directly parameterizes the solution function as a neural network (E & Yu, 2018; Raissi et al., 2019; Bar & Sochen, 2019; Smith et al., 2020).
This approach is designed to model one speciﬁc instance of the PDE, not the solution operator.
It is mesh-independent and accurate, but for any given new instance of the functional parameter/coefﬁcient, it requires training a new neural network.
The approach closely resembles classical methods such as ﬁnite elements, replacing the linear span of a ﬁnite set of local basis functions with the space of neural networks.
The Neural-FEM approach suffers from the same computational issue as classical methods: the optimization problem needs to be solved for every new instance.
Furthermore, the approach is limited to a setting in which the underlying PDE is known.
Neural Operators.
Recently, a new line of work proposed learning mesh-free, inﬁnitedimensional operators with neural networks (Lu et al., 2019; Bhattacharya et al., 2020; Nelsen &
Stuart, 2020; Li et al., 2020b;a).
The neural operator remedies the mesh-dependent nature of the ﬁnite-dimensional operator methods discussed above by producing a single set of network parameters that may be used with different discretizations.
It has the ability to transfer solutions between meshes.
Furthermore, the neural operator needs to be trained only once.
Obtaining a solution for a new instance of the parameter requires only a forward pass of the network, alleviating the major computational issues incurred in Neural-FEM methods.
Lastly, the neural operator requires no knowledge of the underlying PDE, only data.
Thus far, neural operators have not yielded efﬁcient numerical algorithms that can parallel the success of convolutional or recurrent neural networks in the ﬁnite-dimensional setting due to the cost of evaluating integral operators.
Through the fast
Fourier transform, our work alleviates this issue.
Fourier Transform.
The Fourier transform is frequently used in spectral methods for solving differential equations, since differentiation is equivalent to multiplication in the Fourier domain.
Fourier transforms have also played an important role in the development of deep learning.
In theory, they appear in the proof of the universal approximation theorem (Hornik et al., 1989) and, empirically, they have been used to speed up convolutional neural networks (Mathieu et al., 2013).
Neural network architectures involving the Fourier transform or the use of sinusoidal activation functions have also been proposed and studied (Bengio et al., 2007; Mingo et al., 2004; Sitzmann et al., 2020).
Recently, some spectral methods for PDEs have been extended to neural networks (Fan et al., 2019a;b; Kashinath et al., 2020).
We build on these works by proposing a neural operator architecture deﬁned directly in Fourier space with quasi-linear time complexity and state-of-the-art approximation capabilities.
Our Contributions.
We introduce the Fourier neural operator, a novel deep learning architecture able to learn mappings between inﬁnite-dimensional spaces of functions; the integral operator is restricted to a convolution, and instantiated through a linear transformation in the Fourier domain.
• The Fourier neural operator is the ﬁrst work that learns the resolution-invariant solution operator for the family of Navier-Stokes equation in the turbulent regime, where previous graph-based neural operators do not converge.
• By construction, the method shares the same learned network parameters irrespective of the discretization used on the input and output spaces.
It can do zero-shot super-resolution: trained on a lower resolution directly evaluated on a higher resolution, as shown in Figure 1.
• The proposed method consistently outperforms all existing deep learning methods even when ﬁxing the resolution to be 64 × 64 .
It achieves error rates that are 30% lower on Burgers’ Equation, 60% lower on Darcy Flow, and 30% lower on Navier Stokes (turbulent regime with Reynolds number 10000 ).
When learning the mapping for the entire time series, the method achieves < 1% error with Reynolds number 1000 and 8% error with Reynolds number 10000 .
• On a 256 × 256 grid, the Fourier neural operator has an inference time of only 0 .
005 s
compared to the 2 .
2 s of the pseudo-spectral method used to solve Navier-Stokes.
Despite its tremendous speed advantage, the method does not suffer from accuracy degradation when used in downstream applications such as solving the Bayesian inverse problem, as shown in Figure 5.
We observed that the proposed framework can approximate complex operators raising in PDEs that are highly non-linear, with high frequency modes and slow energy decay.
The power of neural operators comes from combining linear, global integral operators (via the Fourier transform) and nonlinear, local activation functions.
Similar to the way standard neural networks approximate highly non-linear functions by combining linear multiplications with non-linear activations, the proposed neural operators can approximate highly non-linear operators.
L EARNING O PERATORS
Our methodology learns a mapping between two inﬁnite dimensional spaces from a ﬁnite collection of observed input-output pairs.
Let D ⊂ R d be a bounded, open set and
A =
A ( D ; R d a ) and
U =
U ( D ; R d u ) be separable Banach spaces of function taking values in R d a and R d u respectively.
Furthermore let G † : A → U be a (typically) non-linear map.
We study maps G † which arise as the solution operators of parametric PDEs – see Section 5 for examples.
Suppose we have observations { a j , u j } Nj =1 where a j ∼ µ is an i.i.d.
sequence from the probability measure µ supported on A and u j = G † ( a j ) is possibly corrupted with noise.
We aim to build an approximation of G † by constructing a parametric map
[Equation]
for some ﬁnite-dimensional parameter space Θ by choosing θ † ∈ Θ so that G ( · , θ † ) =
G θ † ≈ G † .
This is a natural framework for learning in inﬁnite-dimensions as one could deﬁne a cost functional
C : U × U → R and seek a minimizer of the problem
[Equation]
which directly parallels the classical ﬁnite-dimensional setting (Vapnik, 1998).
Showing the existence of minimizers, in the inﬁnite-dimensional setting, remains a challenging open problem.
We will approach this problem in the test-train setting by using a data-driven empirical approximation to the cost used to determine θ and to test the accuracy of the approximation.
Because we conceptualize our methodology in the inﬁnite-dimensional setting, all ﬁnite-dimensional approximations share a common set of parameters which are consistent in inﬁnite dimensions.
A table of notation is shown in Appendix 3.
Learning the Operator.
Approximating the operator G † is a different and typically much more challenging task than ﬁnding the solution u ∈ U of a PDE for a single instance of the parameter a ∈ A .
Most existing methods, ranging from classical ﬁnite elements, ﬁnite differences, and ﬁnite volumes to modern machine learning approaches such as physics-informed neural networks
[Figure]
(PINNs) (Raissi et al., 2019) aim at the latter and can therefore be computationally expensive.
This makes them impractical for applications where a solution to the PDE is required for many different instances of the parameter.
On the other hand, our approach directly approximates the operator and is therefore much cheaper and faster, offering tremendous computational savings when compared to traditional solvers.
For an example application to Bayesian inverse problems, see Section 5.5.
Discretization.
Since our data a j and u j are, in general, functions, to work with them numerically, we assume access only to point-wise evaluations.
Let D j = { x 1 , . . .
, x n } ⊂ D be a n -point discretization of the domain D and assume we have observations a j | D j ∈ R n × d a , u j | D j ∈ R n × d v , for a ﬁnite collection of input-output pairs indexed by j .
To be discretization-invariant, the neural operator can produce an answer u ( x ) for any x ∈ D , potentially x / ∈ D j .
Such a property is highly desirable as it allows a transfer of solutions between different grid geometries and discretizations.
N EURAL O PERATOR
The neural operator, proposed in (Li et al., 2020b), is formulated as an iterative architecture v 0 (cid:55)→ v 1 (cid:55)→ . . .
(cid:55)→ v T where v j for j = 0 , 1 , . . .
, T − 1 is a sequence of functions each taking values in
R d v .
As shown in Figure 2 (a), the input a ∈ A is ﬁrst lifted to a higher dimensional representation v 0 ( x ) =
P ( a ( x )) by the local transformation P which is usually parameterized by a shallow fullyconnected neural network.
Then we apply several iterations of updates v t (cid:55)→ v t +1 (deﬁned below).
The output u ( x ) = Q ( v T ( x )) is the projection of v T by the local transformation Q : R d v → R d u .
In each iteration, the update v t (cid:55)→ v t +1 is deﬁned as the composition of a non-local integral operator
K and a local, nonlinear activation function σ .
Deﬁnition 1 (Iterative updates) Deﬁne the update to the representation v t (cid:55)→ v t +1 by
[Equation]
where K : A × Θ K → L ( U ( D ; R d v ) , U ( D ; R d v )) maps to bounded linear operators on U ( D ; R d v ) and is parameterized by φ ∈ Θ K , W : R d v → R d v is a linear transformation, and σ : R → R is a non-linear activation function whose action is deﬁned component-wise.
We choose K ( a ; φ ) to be a kernel integral transformation parameterized by a neural network.
Deﬁnition 2 (Kernel integral operator K ) Deﬁne the kernel integral operator mapping in (2) by
[Equation]
where κ φ : R 2( d + d a ) → R d v × d v is a neural network parameterized by φ ∈ Θ K .
Here κ φ plays the role of a kernel function which we learn from data.
Together deﬁnitions 1 andconstitute a generalization of neural networks to inﬁnite-dimensional spaces as ﬁrst proposed in Li et al. (2020b).
Notice even the integral operator is linear, the neural operator can learn highly nonlinear operator by composing linear integral operator with non-linear activation functions, analogous to standard neural networks.
If we remove the dependence on the function a and impose κ φ ( x, y )
=
κ φ ( x − y ) , we obtain that (3) is a convolution operator.
We exploit this fact in the following section by parameterizing κ φ directly in Fourier space and using the Fast Fourier Transform (FFT) to efﬁciently compute (3).
This leads to a fast architecture which obtains state-of-the-art results for PDE problems.
F OURIER N EURAL O PERATOR
We propose replacing the kernel integral operator in (3), by a convolution operator deﬁned in Fourier space.
Let F denote the Fourier transform of a function f : D → R d v and F − 1 its inverse then
[Equation]
for j = 1 , . . .
, d v where i =
√− 1 is the imaginary unit.
By letting κ φ ( x, y, a ( x ) , a ( y ))
= κ φ ( x − y ) in (3) and applying the convolution theorem, we ﬁnd that
[Equation]
∀ x ∈ D.
We therefore propose to directly parameterize κ φ in Fourier space.
Deﬁnition 3 (Fourier integral operator K ) Deﬁne the Fourier integral operator
[Equation]
where R φ is the Fourier transform of a periodic function κ : ¯ D → R d v × d v parameterized by φ ∈ Θ K .
An illustration is given in Figure 2 (b).
For frequency mode k ∈ D , we have ( F v t )
( k ) ∈
C d v and R φ ( k ) ∈
C d v × d v .
Notice that since we assume κ is periodic, it admits a Fourier series expansion, so we may work with the discrete modes k ∈ Z d .
We pick a ﬁnite-dimensional parameterization by truncating the Fourier series at a maximal number of modes k max = | Z k max | = |{ k ∈ Z d : |
k j | ≤ k max ,j , for j = 1 , . . .
, d }| .
We thus parameterize R φ directly as complex-valued ( k max × d v × d v ) -tensor comprising a collection of truncated Fourier modes and therefore drop φ from our notation.
Since κ is real-valued, we impose conjugate symmetry.
We note that the set Z k max is not the canonical choice for the low frequency modes of v t .
Indeed, the low frequency modes are usually deﬁned by placing an upper-bound on the (cid:96) 1 -norm of k ∈ Z d .
We choose Z k max as above since it allows for an efﬁcient implementation.
The discrete case and the FFT.
Assuming the domain D is discretized with n ∈ N points, we have that v t ∈ R n × d v and F ( v t ) ∈ C n × d v .
Since we convolve v t with a function which only has k max Fourier modes, we may simply truncate the higher modes to obtain
F ( v t ) ∈
C k max × d v .
Multiplication by the weight tensor
R ∈ C k max × d v × d v is then (cid:0)
[Equation]
When the discretization is uniform with resolution s 1 × · · · × s d = n , F can be replaced by the Fast
Fourier Transform.
For f ∈ R n × d v , k = ( k 1 , . . .
, k d ) ∈
Z s 1 ×· · ·× Z s d , and x = ( x 1 , . . .
, x d ) ∈ D , the FFT ˆ F and its inverse ˆ F − 1 are deﬁned as
[Equation]
for l = 1 , . . .
, d v .
In this case, the set of truncated modes becomes
Z k max = { ( k 1 , . . . , k d ) ∈
Z s 1 × · · · × Z s d | k j ≤ k max ,j or s j − k j ≤ k max ,j , for j = 1 , . . .
, d } .
When implemented, R is treated as a ( s 1 × · · · × s d × d v × d v ) -tensor and the above deﬁnition of
Z k max corresponds to the “corners” of R , which allows for a straight-forward parallel implementation of (5) via matrix-vector multiplication.
In practice, we have found that choosing k max ,j = 12 which yields k max = 12 d parameters per channel to be sufﬁcient for all the tasks that we consider.
Parameterizations of R .
In general, R can be deﬁned to depend on ( F a ) to parallel (3).
Indeed, we can deﬁne R φ : Z d × R d v → R d v × d v as a parametric function which maps (cid:0) k, ( F a )( k )) to the values of the appropriate Fourier modes.
We have experimented with linear as well as neural network parameterizations of R φ .
We ﬁnd that the linear parameterization has a similar performance to the previously described direct parameterization, while neural networks have worse performance.
This is likely due to the discrete structure of the space Z d .
Our experiments in this work focus on the direct parameterization presented above.
Invariance to discretization.
The Fourier layers are discretization-invariant, because they can learn from and evaluate functions which are discretized in an arbitrary way.
Since parameters are learned directly in Fourier space, resolving the functions in physical space simply amounts to projecting on the basis e 2 πi (cid:104) x,k (cid:105) which are well-deﬁned everywhere on R d .
This allows us to achieve zero-shot super-resolution as shown in Section 5.4.
Furthermore, our architecture has a consistent error at any resolution of the inputs and outputs.
On the other hand, notice that, in Figure 3, the standard CNN methods we compare against have an error which grows with the resolution.
Quasi-linear complexity.
The weight tensor R contains k max < n modes, so the inner multiplication has complexity O ( k max ) .
Therefore, the majority of the computational cost lies in computing the Fourier transform F ( v t ) and its inverse.
General Fourier transforms have complexity O ( n 2 ) , however, since we truncate the series the complexity is in fact O ( nk max ) , while the FFT has complexity O ( n log n ) .
Generally, we have found using FFTs to be very efﬁcient. However a uniform discretization is required.
N UMERICAL EXPERIMENTS
In this section, we compare the proposed Fourier neural operator with multiple ﬁnite-dimensional architectures as well as operator-based approximation methods on the 1-d Burgers’ equation, the 2-d
Darcy Flow problem, and 2-d Navier-Stokes equation.
The data generation processes are discussed in Appendices A.3.1, A.3.2, and A.3.3 respectively.
We do not compare against traditional solvers (FEM/FDM) or neural-FEM type methods since our goal is to produce an efﬁcient operator approximation that can be used for downstream applications.
We demonstrate one such application to the
Bayesian inverse problem in Section 5.5.
We construct our Fourier neural operator by stacking four Fourier integral operator layers as speciﬁed in (2) and (4) with the ReLU activation as well as batch normalization.
Unless otherwise speciﬁed, we use N = 1000 training instances and 200 testing instances.
We use Adam optimizer to train for 500 epochs with an initial learning rate of 0 .
001 that is halved every 100 epochs.
We set k max ,j = 16 , d v = 64 for the 1-d problem and k max ,j = 12 , d v = 32 for the 2-d problems.
Lower resolution data are downsampled from higher resolution.
All the computation is carried on a single
Nvidia V100 GPU with 16GB memory.
Remark on Resolution.
Traditional PDE solvers such as FEM and FDM approximate a single function and therefore their error to the continuum decreases as the resolution is increased.
On the other hand, operator approximation is independent of the ways its data is discretized as long as all relevant information is resolved.
Resolution-invariant operators has consistent error rates among different resolutions as shown in Figure 3. Further, resolution-invariant operators can do zero-shot super-resolution, as shown in Section 5.4.
Benchmarks for time-independent problems (Burgers and Darcy): NN: a simple point-wise feedforward neural network.
RBM: the classical Reduced Basis Method (using a POD basis) (De-
FCN: a the-state-of-the-art neural network architecture based on Fully Convolution
Networks (Zhu & Zabaras, 2018).
PCANN: an operator method using PCA as an autoencoder on both the input and output data and interpolating the latent spaces with a neural network (Bhattacharya et al., 2020). GNO: the original graph neural operator (Li et al., 2020b). MGNO: the multipole graph neural operator (Li et al., 2020a). LNO: a neural operator method based on the low-rank decomposition of the kernel κ ( x, y ) :
= (cid:80) rj = φ j ( x ) ψ j ( y ) , similar to the unstacked DeepONet proposed in (Lu et al., 2019) FNO the newly purposed Fourier neural operator. time
Navier 18 layers with residual (He et al A for regression tasks consisting of four blocks with 2-d convolutions and deconvolutions (Ronneberger et al., 2015).
TF-Net: A network designed for learning turbulent ﬂows based on a combination of spatial and temporal convolutions (Wang et al., 2020).
FNO-2d 2-d Fourier neural operator with a
RNN structure in time.
FNO-3d 3-d Fourier neural operator that directly convolves in space-time.
5. B URGERS ’ E QUATION
The 1-d Burgers’ equation is a non-linear PDE with various applications including modeling the one dimensional ﬂow of a viscous ﬂuid.
It takes the form
[Equation]
with periodic boundary conditions where u 0 ∈ L 2 per ((0 , 1); R ) is the initial condition and ν ∈ R + is the viscosity coefﬁcient.
We aim to learn the operator mapping the initial condition to the solution at time one, G † : L 2 per ((0 , 1); R )
→ H r per ((0 , 1); R ) deﬁned by u 0 (cid:55)→ u ( · , 1) for any r > 0 .
The results of our experiments are shown in Figure 3 (a) and Table 3 (Appendix A.3.1).
Our proposed method obtains the lowest relative error compared to any of the benchmarks.
Further, the error is invariant with the resolution, while the error of convolution neural network based methods (FCN) grows with the resolution.
Compared to other neural operator methods such as GNO and MGNO that use Nystr¨om sampling in physical space, the Fourier neural operator is both more accurate and more computationally efﬁcient.
5. D ARCY F LOW
We consider the steady-state of the 2-d Darcy Flow equation on the unit box which is the second order, linear, elliptic PDE −∇ · [Equation] u
[Equation]
f
[Equation]
[Equation]
[Equation]
u (
[Equation]
[Equation] [Equation] [Equation] with a Dirichlet boundary where a ∈ L ∞ ((0 , 1) 2 ; R + ) is the diffusion coefﬁcient and f ∈
L 2 ((0 , 1) 2 ; R ) is the forcing function.
This PDE has numerous applications including modeling the pressure of subsurface ﬂow, the deformation of linearly elastic materials, and the electric potential in conductive materials.
We are interested in learning the operator mapping the diffusion coefﬁcient to the solution, G † : L ∞ ((0 , 1) 2 ; R + )
→ H 10 ((0 , 1) 2 ; R + ) deﬁned by a (cid:55)→ u .
Note that although the PDE is linear, the operator G † is not.
The results of our experiments are shown in Figure 3 (b) and Table 4 (Appendix A.3.2).
The proposed Fourier neural operator obtains nearly one order of magnitude lower relative error compared to any benchmarks.
We again observe the invariance of the error with respect to the resolution.
5. N AVIER -S TOKES E QUATION
We consider the 2-d Navier-Stokes equation for a viscous, incompressible ﬂuid in vorticity form on the unit torus:
[Equation]
where u ∈ C ([0 , T ]; H r per ((0 , 1) 2 ; R 2 )) for any r > 0 is the velocity ﬁeld, w = ∇ × u is the vorticity, w 0 ∈
L 2 per ((0 , 1) 2 ; R ) is the initial vorticity, ν ∈ R + is the viscosity coefﬁcient, and f ∈
L 2 per ((0 , 1) 2 ; R ) is the forcing function.
We are interested in learning the operator mapping the vorticity up to time 10 to the vorticity up to some later time T > 10 , G † : C ([0 , 10]; H r per ((0 , 1) 2 ; R ))
→
C ((10 , T ]; H r per ((0 , 1) 2 ; R )) deﬁned by w | (0 , 1) 2 × [0 , 10] (cid:55)→ w | (0 , 1) 2 × (10 ,T ] .
We experiment with the viscosities ν = 1e − 3 , 1e − 4 , 1e − 5 , decreasing the ﬁnal time T as the dynamic becomes chaotic.
Since the baseline methods are not resolution-invariant, we ﬁx the resolution to be 64 × 64 for both training and testing.
[Table]
As shown in Table 1, the FNO-3D has the best performance when there is sufﬁcient data ( ν = 1e − 3 , N = 1000 and ν = 1e − 4 , N = 10000 ).
For the conﬁgurations where the amount of data is insufﬁcient ( ν = 1e − 4 , N = 1000 and ν = 1e − 5 , N = 1000 ), all methods have > 15% error with FNO-2D achieving the lowest.
Note that we only present results for spatial resolution 64 ×since all benchmarks we compare against are designed for this resolution.
Increasing it degrades their performance while FNO achieves the same errors.
2D and 3D Convolutions.
FNO-2D, U-Net, TF-Net, and ResNet all do 2D-convolution in the spatial domain and recurrently propagate in the time domain (2D+RNN).
The operator maps the solution at previous time steps to the next time step (2D functions to 2D functions).
On the other hand, FNO-3D performs convolution in space-time.
It maps the initial time steps directly to the full trajectory (3D functions to 3D functions).
The 2D+RNN structure can propagate the solution to any arbitrary time T in increments of a ﬁxed interval length ∆ t , while the Conv3D structure is ﬁxed to the interval [0 , T ] but can transfer the solution to an arbitrary time-discretization.
We ﬁnd the 3-d method to be more expressive and easier to train compared to its RNN-structured counterpart.
5. Z ERO - SHOT SUPER - RESOLUTION .
The neural operator is mesh-invariant, so it can be trained on a lower resolution and evaluated at a higher resolution, without seeing any higher resolution data (zero-shot super-resolution).
Figureshows an example where we train the FNO-3D model on 64 × 64 × 20 resolution data in the setting above with ( ν = 1e − 4 , N = 10000 ) and transfer to 256 × 256 × 80 resolution, demonstrating super-resolution in space-time.
Fourier neural operator is the only model among the benchmarks (FNO-2D, U-Net, TF-Net, and ResNet) that can do zero-shot super-resolution.
And surprisingly, it can do super-resolution not only in the spatial domain, but also in the temporal domain.
5. B AYESIAN I NVERSE P ROBLEM
In this experiment, we use a function space Markov chain Monte Carlo (MCMC) method (Cotter et al., 2013) to draw samples from the posterior distribution of the initial vorticity in Navier-Stokes given sparse, noisy observations at time T = 50 .
We compare the Fourier neural operator acting as a surrogate model with the traditional solvers used to generate our train-test data (both run on GPU).
We generate 25,000 samples from the posterior (with a 5,000 sample burn-in period), requiring 30,000 evaluations of the forward operator.
As shown in Figure 5 (Appendix A.5), FNO and the traditional solver recover almost the same posterior mean which, when pushed forward, recovers well the late-time dynamic of Navier Stokes.
In sharp contrast, FNO takes 0 .
005 s to evaluate a single instance while the traditional solver, after being optimized to use the largest possible internal time-step which does not lead to blow-up, takes 2 .
2 s .
This amounts to 2 .
5 minutes for the MCMC using FNO and over 18 hours for the traditional solver.
Even if we account for data generation and training time (ofﬂine steps) which take 12 hours, using FNO is still faster!
Once trained, FNO can be used to quickly perform multiple MCMC runs for different initial conditions and observations, while the traditional solver will take 18 hours for every instance.
Furthermore, since FNO is differentiable, it can easily be applied to PDE-constrained optimization problems without the need for the adjoint method.
Spectral analysis.
Due to the way we parameterize R φ , the function output by (4) has at most k max ,j Fourier modes per channel.
This, however, does not mean that the Fourier neural operator can only approximate functions up to k max ,j modes.
Indeed, the activation functions which occur between integral operators and the ﬁnal decoder network Q recover the high frequency modes.
As an example, consider a solution to the Navier-Stokes equation with viscosity ν = 1e − 3 .
Truncating this function at 20 Fourier modes yields an error around 2% as shown in Figure 4 (Appendix A.2), while our Fourier neural operator learns the parametric dependence and produces approximations to an error of ≤ 1% with only k max ,j = 12 parameterized modes.
Non-periodic boundary condition.
Traditional Fourier methods work only with periodic boundary conditions. However, the Fourier neural operator does not have this limitation.
This is due to the linear transform W (the bias term) which keeps the track of non-periodic boundary.
As an example, the Darcy Flow and the time domain of Navier-Stokes have non-periodic boundary conditions, and the Fourier neural operator still learns the solution operator with excellent accuracy.
D ISCUSSION AND C ONCLUSION
Requirements on Data.
Data-driven methods rely on the quality and quantity of data.
To learn
Navier-Stokes equation with Reynolds number Re = 1e+
, we need to generate N =training pairs { a j , u j } with the numerical solver.
However, for more challenging PDEs, generating a few training samples can be already very expensive.
A future direction is to combine neural operator with numerical solvers to levitate the requirements on data.
Recurrent structure.
The neural operator has an iterative structure that can naturally be formulated as a recurrent network where all layers share the same parameters without sacriﬁcing performance.
(We did not impose this restriction in the experiments.)
Computer vision.
Operator learning is not restricted to PDEs.
Images can naturally be viewed as real-valued functions on 2-d domains and videos simply add a temporal structure.
Our approach is therefore a natural choice for problems in computer vision where invariance to discretization crucial.
We leave this as an interesting and exciting future direction.
A A PPENDIX A. T ABLE OF NOTATIONS
A table of notations is given in Table 2.
[Table]
The spectral decay of Burgers’ equation, Darcy Flow, and Navier Stokes equation are shown in
Figure 4.
We note the harder equations with lower viscosity decay slower than others..
A. D ATA GENERATION
In this section
[Table]
we provide the details of data generator for the three equation we used in Section 5.
A.3. B URGERS E QUATION
Recall the 1-d Burger’s equation on the unit torus:
[Equation]
The initial condition u 0 ( x ) is generated according to u 0 ∼ µ where µ = N (0 , 625( − ∆ + 25 I ) − 2 ) with periodic boundary conditions.
We set the viscosity to ν = 0 .
1 and solve the equation using a split step method where the heat equation part is solved exactly in Fourier space then the non-linear part is advanced, again in Fourier space, using a very ﬁne forward Euler method.
We solve on a spatial mesh with resolution 2 13 = 8192 and use this dataset to subsample other resolutions.
A.3. D ARCY F LOW
The 2-d Darcy Flow is a second-order linear elliptic equation of the form
[Equation]
The coefﬁcients a ( x ) are generated according to a ∼ µ where µ = ψ # N (0 , ( − ∆ + 9 I ) − 2 ) with zero Neumann boundary conditions on the Laplacian.
The mapping ψ : R → R takes the valueon the positive part of the real line and 3 on the negative and the push-forward is deﬁned pointwise.
The forcing is kept ﬁxed f ( x ) = 1 .
Such constructions are prototypical models for many physical systems such as permeability in subsurface ﬂows and material microstructures in elasticity.
Solutions u are obtained by using a second-order ﬁnite difference scheme on a 421 × 421 grid.
Different resolutions are downsampled from this dataset.
A.3. N AVIER -S TOKES E QUATION
Recall the 2-d Navier-Stokes equation for a viscous, incompressible ﬂuid in vorticity form on the unit torus:
[Equation]
The initial condition w 0 ( x ) is generated according to w 0 ∼ µ where µ = N (0 , 7 3 / 2 ( − ∆+49 I ) − 2 . 5 ) with periodic boundary conditions.
The forcing is kept ﬁxed f ( x ) = 0 .
1(sin(2 π ( x 1 + x 2 ))
+ cos(2 π ( x 1 + x 2 ))) .
The equation is solved using the stream-function formulation with a pseudospectral method.
First a Poisson equation is solved in Fourier space to ﬁnd the velocity ﬁeld.
Then the vorticity is differentiated and the non-linear term is computed is physical space after which it is dealiased.
Time is advanced with a Crank–Nicolson update where the non-linear term does not enter the implicit part.
All data are generated on a 256 × 256 grid and are downsampled to 64 × 64 .
We use a time-step of 1e − 4 for the Crank–Nicolson scheme in the data-generated process where we record the solution every t = 1 time units.
The step is increased to 2e − 2 when used in MCMC for the Bayesian inverse problem.
A. R ESULTS OF B URGERS ’ EQUATION AND D ARCY F LOW
The details error rate on Burgers’ equation and Darcy Flow are listed in Table 3 and Table
[Table] [Table]
[Table]
A. B AYESIAN I NVERSE P ROBLEM
Results of the Bayesian inverse problem for the Navier-Stokes equation are shown in Figure 5.
It can be seen that the result using Fourier neural operator as a surrogate is as good as the result of the traditional solver.