Single Reference Frequency Loss for Multi-frequency Waveﬁeld Representation using Physics-Informed Neural Networks
Abstract
Physics-informed neural networks (PINNs) can offer approximate multidimensional functional solutions to the Helmholtz equation that are ﬂexible, require low memory, and have no limitations on the shape of the solution space.
However, the neural network (NN) training can be costly and the cost dramatically increases as we train for multi-frequency waveﬁelds by adding frequency to the NN multidimensional function, as the variation of the waveﬁeld with frequency adds more complexity to the NN training.
Thus, we propose a new loss function for the NN multidimensional input training that allows us to seamlessly include frequency as a dimension.
We speciﬁcally utilize the linear relation between frequency and wavenumber (the waveﬁeld space representation) to incorporate a reference frequency scaling to the loss function.
As a result, the effective wavenumber of the waveﬁeld solution as a function of frequency remains stationary reducing the learning burden on the NN function.
We demonstrate the effectiveness of this modiﬁed loss function on a layered model.
Introduction
Frequency-domain wave equation modeling based on the Helmholtz equation, is quite common and of great importance in modeling many physical phenomena, e g., electromagnetic and seismic wave propagation.
However, in inverting for the subsurface properties, we have to solve the Helmholtz equation for many frequencies to recover ﬁne-scale details, like in ultrasound medical imaging, ground penetrating radar, and seismic full waveform inversion.
Consequently, an accurate and efﬁcient multi-frequency solution is extremely important in many scientiﬁc and industrial applications.
However, when the size of the subsurface model is large and the frequency is high, the computational cost of classical methods such as ﬁnite difference, ﬁnite element, and spectral methods is high.
Besides, the complexity of wave equation in elastic or anisotropic media can considerably add to the computational burden.
The recently developed physics-informed neural network (PINN) for solving for the Helmholtz equation showed considerable potential in modeling because of its ﬂexibility, low memory requirement, and no limitations are imposed on the shape of the solution space.
However, it is hard to train, admitting less than optimal solutions for practical size neural network models.
A major challenge is that when we increase the dimension of the input (like including frequency), the complexity of the waveﬁeld increases, yielding poor convergence of PINN.
Reducing the complexity of the PINN optimization problem is an important objective.
Here we propose to use a reference frequency based loss function to train a neural network (NN) for multi-frequency waveﬁeld representation using PINN.
The reference frequency allows us to effectively mitigate the change in spatial wavenumber over frequency by adapting the spatial scale to frequency, thus, reducing the complexity of the waveﬁeld as if it was representing a single frequency.
This is important, as the PINN convergence depends highly on the complexity of the waveﬁeld.
We apply our method using the frequency-domain scattered wave equation to predict multi-frequency waveﬁelds.
Compared to the traditional PINN with multi-frequency loss, our approach yields more accurate and efﬁcient waveﬁeld solutions.
Related Work
Physics-informed neural networks play a vital role in surrogate modeling with potential applications in many ﬁelds [9, 13, 11, 3, 5].
For waveﬁelds, when the solution domain is large or the frequency is high, the complexity of the solution requires a large-size neural network, which is hard to train.
To address the limitations of PINN, e.g., convergence, especially its convergence in scenarios with a large solution domain and high-frequency, Wang et al.
[14] made use of Fourier input feature to solve the optimization using trigonometric functions.
Liu et al.
[7] proposed multi-scale deep neural networks with a PINN loss to improve the convergence.
Recently, domain decomposition has gained attention [6, 4, 8] in which we divide the problem into many sub-domains.
For inverse problems, we also need multi-frequency solutions and all these methods have not addressed this need.
Alkhalifah et al.
[1] demonstrated the multi-dimensional waveﬁeld solutions potential of PINNs, but the accuracy over the range of frequencies was not good.
Single Reference Frequency Loss 3.1 Helmholtz equation for scattered waveﬁeld
In this section, we brieﬂy revisit several key concepts of the frequency-domain scattered waveﬁeld representation using PINN.
To reduce the spatial samples needed for training the 2-D frequencydomain acoustic wave equation (Helmholtz equation), and mitigate the source singularity of the frequency-domain waveﬁeld, we use the scattered waveﬁeld instead, given by:
[Equation]
(1) where m is the squared slowness, ω is the angular frequency, U is the frequency-domain waveﬁeld as a function of ( x, z ) due to the source term s =
( s x , s z ) and ∇ is the gradient operator, U [-
 isthe-]0 is the background waveﬁeld, δ U
=
U − U
[2] is the scattered waveﬁeld, and δ m = m − m [-
 isthe-]0 is the squared slowness perturbation.
Considering the background velocity to be constant, U 0 can be directly calculated using the constant background squared slowness m 0 with an analytical relation [10]:
[Equation]
where H ( 2 )is the zero-order Hankel function of the second kind.
To ﬁnd a neural network representation Φ( θ, x ) satisfying the physical constraint (PINN), where θ represents the model parameters and x = ( x, z, s x , ω ) , we use the physical multi-frequency loss function deﬁned as:
[Equation]
3.2 Relationship between frequency upscaling and spatial rescaling
From equation 1, we observe that when we double the frequency, the ﬁrst and the third term will quadruple.
To keep equation 1 stationary, we need the second term (The Laplacian operator acting on δ U ) to quadruple, as well.
To do so, we rescale the spatial axes to maintain the effective wavenumber.
So when we double the frequency, we rescale the spatial coordinates by a half.
For simplicity we use three frequency-domain waveﬁelds (Figure 1) to demonstrate the process.
We immediately arrive to the conclusion that the waveﬁeld by the frequency upscaling (8Hz waveﬁeld) and the waveﬁeld by spatial rescaling (4Hz waveﬁeld after spatial rescaling) share similar wavenumber content satisfying the Helmholtz equation, which means we just need one frequency here to describe two waveﬁelds with different frequencies.
δ U 1 ( ω 1 =2 π ×4 Hz ) ω
21 m δ U 1 +∇ 2 δ U 1 + ω 21 δ mU 0(4 Hz ) =0, δ U 2 ( ω 2 =2 π ×8 Hz ) ω
22 m δ U 2 +∇ 2 δ U 2 + ω 22 δ mU 0(8 Hz ) =0,
Spatial rescaling δ U 1( SR ) ( ω 1 =2 π ×4 Hz ) ω
22 m δ U 1( SR ) +∇ 2 δ U 1( SR ) + ω 22 δ mU 0(4 Hz ) =0,
[Figure]
3.3 Dynamic frequency weighting derivation
The relationship between a frequency upscaling and spatial rescaling can be established to maintain a stationary wavenumber of the waveﬁeld over the frequency range.
However, we do so by literally rescaling the spatial coordinates as a function of frequency will admit nonuniform space dimension range for the various frequencies, in which the higher frequency component space dimension will be larger than the low frequency one.
This dimension will non uniformity add complexity in the implementation of PINN.
To implement this stationary wavenumber concept, we introduce a frequency weighting approach which up-weights the derivation of low-frequency waveﬁelds.
The frequency weighting is dynamically determined by the ratio of current frequency of the training sample to the reference frequency, and as a result, the gradient of the scattered waveﬁeld is given by:
[Equation]
where α is the scaling factor, equal to the ratio of the current frequency to a reference frequency.
Inserting equation 4 into 1, we have the new loss function:
[Equation]
where ω ref is the reference angular frequency.
In practice, we implement equation 5 by utilizing a computational graph (shown in Algorithm 1).
Algorithm 1: Training with the single reference frequency loss function.
Draw N points { x i } N i =1 sampled from the 4-D model region, and no boundary points.
Initiate: model parameters θ for each epoch do
Compute the x ref =
( x, z, s x × ωω ref )
Then the input of the NN: x = ( x ref × ω ref ω , ω ) normalize x and feed them into the network to get Φ( θ, x ) automatic differentiation: ∂ 2 Φ( θ, x ) ∂ 2 x ref calculate the loss function of equation
Update: model parameters θ
Experiments
In this section, we show the effectiveness of our proposed loss function for the scattered wave equation, and compare the accuracy of waveﬁeld and the corresponding velocity.
We compare the performances of conventional PINN using the multi-frequency loss function and PINN utilizing our single reference frequency loss function.
Based on a simple layered model covering an area of 2 . 5 × 2 .
5 km 2 , we generate 1280000 random samples from our four dimensional waveﬁeld, with f , the frequency, ranging from 3.0 Hz to [-8.
-]8.
Hz, along with δm for squared slowness perturbation, m 0 for background squared slowness at these points.
The depth of sources s z is set to 0.025 km.
The background waveﬁeld is calculated analytically for a background velocity of 1.5km/s. The reference frequency here is 8.0 Hz.
The basic network architecture for both methods is a Multi-Layer Perception with three hidden layers, as well as positional encoding [5].
The inputs include { x, z, s x , ω } and the hidden layers are of the size { 512 , 512 , 512 } from shallow to deep.
We use an Adam optimizer to train our networks.
During training, we set the batch size to 40000.
The initial learning rate is chosen to be 1 e − 3 and it is gradually decreased to 5 e − 5 .
We have trained our model with these settings for 15000 epochs.
To evaluate the results, we solve the Helmholtz equation numerically for speciﬁc frequencies and source locations and use the solution as reference, and these solutions are provided in the 2 . 5 × 2 .
5 km 2 area using 100 samples in both the x and z directions.
For Figure 2, the source is located at a depth of 0.025 km and lateral distance of 1.0 km .
Figure 2 shows
[Figure] [Figure]
n a l P I NN
O u r s the predicted waveﬁeld for various frequencies.
It is obvious that with one more input dimension, the representation of NNs for the waveﬁeld becomes harder to obtain via conventional training.
On the other hand, our proposed loss function provided reasonable results considering the larger (four) dimensional space the PINN is expected to predict.
We also calculated the velocity models from the PINN predictions using equation 1[12], as shown in Figure 3.
We can observe that the PINN with our proposed loss function reconstructs the details of the velocity model much better than the conventional method.
Conclusion
We proposed using a reference frequency based loss function to train the NN for multi-frequency waveﬁeld representation and we demonstrated that this approach admitted superior performance and waveﬁeld accuracy.
The reference frequency loss function implicitly embeds the relationship between frequency upscaling and spatial rescaling into the network, which makes the network easier to train and improves its representation.
The method has the potential to solve multi-frequency Helmholtz equation even in a large and complex model.
The method can be generalized to other physical problems, which may include a multi-frequency or multi-scale component.