P HYSICS - AWARE D IFFERENCE G RAPH N ETWORKS FOR S PARSELY -O BSERVED D YNAMICS
A BSTRACT
Sparsely available data points cause numerical error on ﬁnite differences which hinders us from modeling the dynamics of physical systems.
The discretization error becomes even larger when the sparse data are irregularly distributed or deﬁned on an unstructured grid, making it hard to build deep learning models to handle physics-governing observations on the unstructured grid.
In this paper, we propose a novel architecture, Physics-aware Difference Graph Networks (PA-DGN), which exploits neighboring information to learn ﬁnite differences inspired by physics equations.
PA-DGN leverages data-driven end-to-end learning to discover underlying dynamical relations between the spatial and temporal differences in given sequential observations.
We demonstrate the superiority of PA-DGN in the approximation of directional derivatives and the prediction of graph signals on the synthetic data and the real-world climate observations from weather stations.
I NTRODUCTION
Modeling real world phenomena, such as climate observations, trafﬁc ﬂow, physics and chemistry simulation (Li et al., 2018;
Geng et al., 2019; Long et al., 2018; de Bezenac et al., 2018; SanchezGonzalez et al., 2018; Gilmer et al., 2017), is important but extremely challenging.
While deep learning has achieved remarkable successes in prediction tasks by learning latent representations from data-rich applications such as image recognition (Krizhevsky et al., 2012), text understanding (Wu et al., 2016), and speech recognition (Hinton et al., 2012), we confront many challenging scenarios in modeling natural phenomena with deep neural networks when only a limited number of observations are available.
Particularly, the sparsely available data points cause substantial numerical error when we utilize existing ﬁnite difference operators and the limitation requires a more principled way to redesign deep learning models.
While many methods have been proposed to model physics-simulated observations using deep learning, many of them are designed under the assumption that input is on a continuous domain.
For example, Raissi et al.
(2017a;b) proposed physics-informed neural networks (PINNs) to learn nonlinear relations between input (spatial- and temporal-coordinates ( x , t )) and output simulated with a given partial differential equation (PDE).
Since Raissi et al.
(2017a;b) use the coordinates as input and compute derivatives based on the coordinates to represent the equation, the setting is only valid when the data are densely observed over spatial and temporal space.
Prior knowledge related to physics equations has been combined with data-driven models for various purposes. Chen et al.
(2015) proposed a nonlinear diffusion process for image restoration and de Bezenac et al.
(2018) incorporated the transport physics (advection-diffusion equation) with deep neural networks for forecasting sea surface temperature by extracting the motion ﬁeld.
Lutter et al.
(2019) introduced deep Lagrangian networks specialized to learn Lagrangian mechanics with learnable parameters.
Seo & Liu (2019) proposed a physics-informed regularizer to impose dataspeciﬁc physics equations.
In common, the methods in Chen et al.
(2015); de Bezenac et al.
Lutter et al.
(2019) are not efﬁciently applicable to sparsely discretized input as only a small number of data points are available and continuous properties on given space are not easily recovered.
It is unsuitable to directly use continuous differential operators to provide local behaviors because it is hard to approximate the continuous derivatives precisely with the sparse points (Shewchuk, 2002;
Amenta & Kil, 2004; Luo et al., 2009).
Furthermore, they are only applicable when the speciﬁc physics equations are explicitly given and still hard to be generalized to incorporate other types of equations.
As another direction to modeling physics-simulated data, Long et al.
(2018) proposed PDE-Net which uncovers the underlying hidden PDEs and predicts the dynamics of complex systems.
Ruthotto & Haber (2018) derived new CNNs: parabolic and hyperbolic CNNs based on ResNet (He et al., 2016) architecture motivated by PDE theory.
While Long et al.
(2018); Ruthotto & Haber (2018) are ﬂexible to uncover hidden physics from the constrained kernels, it is still restrictive to a regular grid where the proposed constraints on the learnable ﬁlters are easily deﬁned.
The topic of reasoning physical dynamics of discrete objects has been actively studied (SanchezGonzalez et al., 2018; Battaglia et al., 2016; Chang et al., 2016) as the appearance of graph-based neural networks (Kipf & Welling, 2017; Santoro et al., 2017; Gilmer et al., 2017).
Although these models can handle sparsely located data points without explicitly given physics equations, they are purely data-driven so that the physics-inspired inductive bias for exploiting ﬁnite differences is not considered at all.
In contrast, our method consists of physics-aware modules allowing efﬁciently leveraging the inductive bias to learn spatiotemporal data from the physics system.
In this paper, we propose physics-aware difference graph networks (PA-DGN) whose architecture is motivated to leverage differences of sparsely available data from the physical systems.
The differences are particularly important since most of the physics-related dynamic equations (e.g., Navier–Stokes equations) handle differences of physical quantities in spatial and temporal space instead of using the quantities directly.
Inspired by the property, we ﬁrst propose spatial difference layer (SDL) to efﬁciently learn the local representations by aggregating neighboring information in the sparse data points.
The layer is based on graph networks (GN) as it easily leverages structural features to learn the localized representations and share the parameters for computing the localized features.
Then, the layer is followed by recurrent graph networks (RGN) to predict the temporal difference which is another core component of physics-related dynamic equations.
PA-DGN is applicable to various tasks and we provide two representative tasks; the approximation of directional derivatives and the prediction of graph signals.
Our contributions are:
• We tackle a limitation of the sparsely discretized data which cause numerical error when we model the physical system by proposing spatial difference layer (SDL) for efﬁciently exploiting neighboring information under the limitation of sparsely observable points.
• We combine SDL with recurrent graph networks to build PA-DGN which automatically learns the underlying spatiotemporal dynamics in graph signals.
• We verify that PA-DGN is effective in approximating directional derivatives and predicting graph signals in synthetic data.
Then, we conduct exhaustive experiments to predict climate observations from land-based weather stations and demonstrate that PA-DGN outperforms other baselines.
P HYSICS - AWARE D IFFERENCE G RAPH N ETWORK
In this section, we introduce the building module used to learn spatial differences of graph signals and describe how the module is used to predict signals in the physics system.
2. D IFFERENCE O PERATORS ON G RAPH
As approximation of derivatives in continuous domain, difference operators have been used as a core role to compute numerical solutions of (continuous) differential equations.
Since it is hard to derive closed-form expressions of derivatives in real-world data, the difference operators have been considered as alternative tools to describe and solve PDEs in practice.
The operators are especially important for physics-related data (e.g., meteorological observations) because the governing rules behind the observations are mostly differential equations. [Figure] [Figure] [Figure]
Graph signals Given a graph G =
( V , E ) where V is a set of vertices
V = { 1 , . .
, N v } and E is a set of edges E ⊆ { ( i, j ) | i, j ∈ V } ( | E | = N e ), graph signals on all nodes at time t are f ( t ) =
{ f i ( t ) | i ∈ V } where f i : V → R .
Graph signals on edges can also be deﬁned similarly, F ( t ) =
{ F ij ( t ) | ( i, j ) ∈ E } where F ij : E → R .
Both signals can be multidimensional.
Gradient on graph The gradient ( ∇ ) of a function on nodes of a graph is represented by ﬁnite difference
[Equation]
0 otherwise, where L 2 ( V ) and L 2 ( E ) denote Hilbert spaces for node/edge functions, respectively.
The gradients on a graph provide ﬁnite differences of graph signals and they become edge ( i, j ) features.
Laplace-Beltrami operator Laplace-Beltrami operator (or Laplacian, ∆ ) in graph domain is deﬁned as
[Equation]
This operator is usually regarded as a matrix form in other literature, L = D − A where A is an adjacency matrix and D = diag ( (cid:80) j : j (cid:54) =
i A ij ) is a degree matrix.
2. D IFFERENCE O PERATORS ON T RIANGULATED M ESH
According to Crane (2018), the gradient and Laplacian operators on the triangulated mesh can be discretized by incorporating the coordinates of nodes.
To obtain the gradient operator, the per-face gradient of each triangular face is calculated ﬁrst.
Then, the gradient on each node is the area-weighted average of all its neighboring faces, and the gradient on edge ( i, j ) is deﬁned as the dot product between the per-node gradient and the direction vector e ij .
The Laplacian operator can be discretized with Finite Element Method (FEM):
[Equation]
where node j belongs to node i ’s immediate neighbors ( j ∈ N i ) and ( α j , β j ) are two opposing angles of the edge ( i, j ) .
2. S PATIAL D IFFERENCE L AYER
While the difference operators are generalized in Riemannian manifolds (Lai et al., 2013; Lim, 2015), there exist numerical error compared to those in continuous space and the error can be larger when the nodes are spatially far from neighboring nodes because the connected nodes ( j ∈ N i ) of i -th node fail to represent local features around the node.
Furthermore, the error is even larger if available data points are sparsely distributed (e.g., sensor-based observations).
In other words, the difference operators are unlikely to discover meaningful spatial variations behind the sparse observations since they are highly limited to immediate neighboring information only.
To mitigate the limitation, we propose spatial difference layer (SDL) which consists of a set of parameters to deﬁne learnable [Figure] difference operators as a form of gradient and Laplacian to fully utilize neighboring information:
[Equation]
where w ij are the parameters tuning the difference operators along with the corresponding edge direction e ij .
The two forms (Eq 1) are associated with edge and node features, respectively.
The superscript in w ∇ and w ∆ denotes that the difference operators are functions of the learnable parameters w .
w ( g ) ij and w ( l ) ij are obtained by integrating local information as follow:
[Equation]
While the standard difference operators consider two connected nodes only ( i and j ) for each edge ( i, j ) , Eq 2 uses a larger view ( h -hop) to represent the differences between i and j nodes.
Since graph networks (GN) (Battaglia et al., 2018) are efﬁcient networks to aggregate neighboring information, we use GN for g ( · ) function and w ij are edge features of output of GN.
Eq 2 can be viewed as a higher-order difference equation because nodes/edges which are multi-hop apart are considered.
w ij has a similar role of parameters in convolution kernels of CNNs.
For example, while the standard gradient operator can be regarded as an example of simple edge-detecting ﬁlters, the operator can be a sharpening ﬁlter if w ( g 1 ) ij = 1 and w ( g 2 ) ij =
| N i | +1 | N i | for i node and the operators over each edge are summed.
In other words, by modulating w ij , it is readily extended to conventional kernels including edge detection or sharpening ﬁlters and even further complicated kernels.
On top of w ij , the difference forms in Eq 1 make an optimizing process for learnable parameters based on the differences instead of the values intentionally.
Eq 1 thus naturally provides the physics-inspired inductive bias which is particularly effective for modeling physics-related observations.
Furthermore, it is easily possible to increase the number of channels for w ( g ) ij and w ( l ) ij to be more expressive.
Figure 1 illustrates how the exemplary ﬁlters convolve the given graph signals.
2. R ECURRENT G RAPH N ETWORKS
Difference graph Once the modulated spatial differences ( w ∇ f ( t ) , w ∆ f ( t ) ) are obtained, they are concatenated with the current signals f ( t ) to construct node-wise ( z i ) and edge-wise ( z ij ) features and the graph is called a difference graph .
The difference graph includes all information to describe spatial variations.
Recurrent graph networks Given a snapshot ( f ( t ) , F ( t ) ) of a sequence of graph signals, one difference graph is obtained and is used to predict next graph signals.
While a non-linear layer can be used to combine the learned spatial differences to predict the next signals, it is limited to discover spatial relations only among the features in the difference graph.
Since many equations describing physics-related phenomena are non-static (e.g., Navier–Stokes equations), we adopt recurrent graph networks (RGN)
(Sanchez-Gonzalez et al., 2018) with a graph state G h as input to combine the spatial differences with temporal variations.
RGN returns a graph state ( G ∗ h
= ( h ∗ ( v ) , h ∗ ( e ) ) ) and next graph signal z ∗ i and z ∗ ij .
The update rule is described as follow:
( z ∗ ij , h ∗ ( e ) ) ← φ e ( z ij , z i , z j , h ( e ) ) for all ( i,
j ) ∈
E pairs, 2.
( z ∗ i , h ∗ ( v ) ) ← φ v ( z i , ¯ z (cid:48)
i , h ( v ) ) for all i ∈ V , ¯ z (cid:48)
i is an aggregated edge attribute related to the node i ,
where φ e , φ v are edge and node update functions, respectively, and they can be any recurrent unit.
Finally, the prediction is made through a decoder by feeding the graph signal, z ∗ i and z ∗ ij .
Learning objective Let ˆ f and ˆ F denote predictions of the target node/edge signals.
PA-DGN is trained by minimizing the following objective:
[Equation]
For multistep predictions, L is summed over all predicting steps.
If only one type (node or edge) of signal is given, the corresponding term in Eq 3 is used to optimize the parameters in SDL and RGN simultaneously.
E FFECTIVENESS OF S PATIAL D IFFERENCE L AYER
To investigate if the proposed spatial difference forms (Eq 1) can be beneﬁcial to learning physicsrelated patterns, we use SDL on two different tasks: (1) approximate directional derivatives and (2) predict synthetic graph signals.
3. A PPROXIMATION OF D IRECTIONAL D ERIVATIVES
[Equation]
As we claimed in Section 2.3, the standard difference forms (gradient and Laplacian) on a graph can cause signiﬁcant numerical error easily because they are susceptible to a distance of two points and variations of a given function.
To evaluate the applicability of the proposed SDL, we train SDL to approximate directional derivatives on a graph.
First, we deﬁne a synthetic function and its gradients on 2D space and samplepoints ( x i , y i ) .
Then, we construct a graph on the sampled points by using k -NN algorithm ( k = 4 ).
With the known gradient (cid:16) ∇ f = ( ∂f∂x , ∂f∂y ) (cid:17) at each point (a node in the graph), we can compute directional derivatives by projecting ∇ f to a connected edge e ij (See Figure 3).
We compare against four baselines: (1) the ﬁnite gradient ( FinGrad ) (2) multilayer perceptron ( MLP ) (3) graph networks ( GN ) (4) a different form of Eq( One-w ).
For the ﬁnite gradient ( ( f j − f i ) /
||
x j − x i || ), there is no learnable parameter and it only uses two points.
For MLP, we feed ( f i , f j , x i , x j ) as input to see whether learnable parameters can beneﬁt the approximation or not.
For GN, we use distances of two connected points as edge features and function values on the points as node features.
The edge feature output of GN is used as a prediction for the directional derivative on the edge.
Finally, we modify the proposed form as ( w ∇ f ) ij = w ij f j − f i .
GN and the modiﬁed form are used to verify the effectiveness of Eq 1.
Note that we deﬁne two synthetic functions (Figure 4) which have different property; (1) monotonically increasing from a center and (2) periodically varying.
[Table]
Approximation accuracy As shown in Table 1, the proposed spatial difference layer outperforms others by a large margin.
As expected, FinGrad provides the largest error since it only considers two points without learnable parameters.
It is found that the learnable parameters can signiﬁcantly beneﬁt to approximate the directional derivatives even if input is the same (FinGrad vs. MLP).
Note that utilizing neighboring information (GN, One-w, SDL) is generally helpful to learn spatial variations properly.
However, simply training parameters in GN is not sufﬁcient and explicitly deﬁning difference , which is important to understand spatial variations, provides more robust inductive bias.
One important thing we found is that One-w is not effective as much as GN and it can be even worse than FinGrad.
It is because of its limited degree of freedom.
As implied in the form ( ∇ w f ) ij = w ij ∗ f j − f i , only one w ij adjusts the relative difference between f i and f j , and this is not enough to learn whole possible linear combinations of f i and f j .
The unstable performance supports that the form of SDL is not ad-hoc but more rigorously designed.
3. G RAPH S IGNAL P REDICTION
We evaluate PA-DGN on the synthetic data sampled from the simulation of speciﬁc convectiondiffusion equations, to provide if the proposed model can predict next signals of the simulated dynamics from observations on discrete nodes only.
For the simulated dynamics, we use an equation slightly modiﬁed based on the one in Long et al.
[Equation]
where the index i points the i -th node whose coordinate is ( x i , y i ) in the 2D space ( [0 , 2 π ] × [0 , 2 π ] ) and ˆ x and ˆ y indicate x - and y -direction in the space.
a ( i ) = 0 .
5( cos ( y i )
+ x i (2 π − x i ) sin ( x i ))
+0 .
6 , b ( i ) =
2( cos ( y i )
+ sin ( x i ))
+ 0 . 8 , and c ( i )
= 0 .(cid:18)
[Equation]
− √ ( [Equation] i −[-π ) 
-]
[Equation] +( y [Equation] − π ) 2 √ 2 π (cid:19) .
Then, we uniformly sample 250 points in the above 2D space.
The task is to predict signal values of all points in the future M steps given observed values of the ﬁrst N steps.
For our experiments, we choose N =and M = 15 .
Since there is no a priori graph structure on sampled points, we construct a graph with k -NN algorithm ( k = 4 ) using the Euclidean distance.
Figure 5 shows the dynamics and the graph structure of sampled points.
To evaluate the effect of the proposed SDL on the above prediction task, we cascade SDL and a linear regression model as our prediction model since the dynamics follows a linear partial differential equation.
We compare its performance with four baselines: (1) vector auto-regressor ( VAR ); (2) multilayer perceptron ( MLP ); (3) StandardOP : the standard approximation of differential operators in Section 2.1 followed by a linear regressor; (4) MeshOP : similar to StandardOP but use the discretization on triangulated mesh in Section 2.2 for differential operators.
[Table]
Prediction Performance Table 2 shows the prediction performance of different models measured with mean absolute error.
The prediction model with our proposed spatial differential layer outperforms other baselines.
All models incorporating any form of spatial difference operators (StandardOP,
[Figure] MeshOP and SDL) outperform those without spatial difference operators (VAR and MLP), showing that introducing spatial differences information inspired by the intrinsic dynamics helps prediction.
However, in cases where points with observable signal are sparse in the space, spatial difference operators derived with ﬁxed rules can be inaccurate and sub-optimal for prediction since the locally linear assumption which they are based on no longer holds.
Our proposed SDL, to the contrary, is capable of bridging the gap between approximated difference operators and accurate ones by introducing learnable coefﬁcients utilizing neighboring information, and thus improves the prediction performance.
P REDICTION : G RAPH S IGNALS ON L AND - BASED W EATHER S ENSORS
We evaluate the proposed model on the task of predicting climate observations ( Temperature ) from the land-based weather stations located in the United States.
4. E XPERIMENTAL S ET - UP
Data and task We sample the weather stations located in the United States from the Online Climate
Data Directory of the National Oceanic and Atmospheric Administration (NOAA) and choose the stations which have actively measured meteorological observations during 2015.
We choose two geographically close but meteorologically diverse groups of stations: the Western and Southeastern states.
We use k -Nearest Neighbor (NN) algorithm ( k = 4 ) to generate graph structures and the ﬁnal adjacency matrix is A
=
( A k + A (cid:62) k ) / 2 to make it symmetric where A k is the output adjacency matrix from k -NN algorithm.
Figure 6 shows the distributions of the land-based weather stations and their connectivity.
Since the stations are not synchronized and have different timestamps for the observations, we aggregate the time series hourly.
The 1-year sequential data are split into the train set (8 months), the validation set (2 months), and the test set (2 months), respectively.
Our main task is to predict the next graph signals based on the current and past graph signals.
All methods we evaluate are trained through the objective (Eq 3) with the
Adam optimizer and we use scheduled sampling (Bengio et al., 2015) for the models with recurrent modules.
We evaluate PA-DGN and other baselines on two prediction tasks, (1) 1-step and (2) multistep-ahead predictions.
Furthermore, we demonstrate the ablation study that provides how much the spatial derivatives from our proposed SDL are important signals to predict the graph dynamics.
4. G RAPH S IGNAL P REDICTIONS
We compare against the widely used baselines (VAR, MLP, and GRU) for 1-step and multistep prediction.
Then, we use RGN (Sanchez-Gonzalez et al., 2018) to examine how much the graph structure is beneﬁcial.
Finally, we evaluate PA-DGN to verify if the proposed architecture (Eq 1) is able to reduce prediction loss.
Experiment results for the prediction task are summarized in Table 3.
Overall, RGN and PA-DGN are better than other baselines and it implies that the graph structure provides useful inductive bias for the task.
It is intuitive as the meteorological observations are continuously changing over the space and time and thus, the observations at the i -th station are strongly related to those of its neighboring stations.
PA-DGN outperforms RGN and the discrepancy comes from the fact that the spatial derivatives (Eq 1) we feed in PA-DGN are beneﬁcial and this ﬁnding is expected because the meteorological signals at a certain point are a function of not only its previous signal but also the relative differences between neighbor signals and itself.
Knowing the relative differences among local observations is particularly essential to understand physics-related dynamics.
For example, Diffusion equation, which describes how physical quantities (e.g., heat) are transported through space over time [Table] is also a function of relative differences of the quantities ( dfdt =
D ∆ f ) rather than values of the neighbor signals.
In other words, spatial differences are physics-aware features and it is desired to leverage the features as input to learn dynamics related to physical phenomena
[Table]
[Table]
4. C ONTRIBUTION OF S PATIAL D ERIVATIVES
We further investigate if the modulated spatial derivatives (Eq 1) are effectively advantageous compared to the spatial derivatives deﬁned in Riemannian manifolds.
First, RGN without any spatial derivatives is assessed for the prediction tasks on Western and Southeastern states graph signals.
Note that this model does not use any extra features but the graph signal, f ( t ) .
Secondly, we add (1) StandardOP , the discrete spatial differences (Gradient and Laplacian) in Section 2.1 and (2)
MeshOP , the triangular mesh approximation of differential operators in Section 2.2 separately as additional signals to RGN.
Finally, we incorporate with RGN our proposed Spatial Difference Layer.
Table 3 shows the contribution of each component.
As expected, PA-DGN provides much higher drops in MAE (3.56%, 5.50%, 8.51% and 8.73%, 8.32%, 5.49% on two datasets, respectively) compared to
RGN without derivatives and the results demonstrate that the derivatives, namely, relative differences from neighbor signals are effectively useful.
However, neither RGN with StandardOP nor with
MeshOP can consistently outperform RGN.
We also found that PA-DGN consistently shows positive effects on the prediction error compared to the ﬁxed derivatives.
This ﬁnding is a piece of evidence to support that the parameters modulating spatial derivatives in our proposed Spacial Difference Layer are properly inferred to optimize the networks and to improve the prediction performance.
C ONCLUSION
In this paper, we introduce a novel architecture (PA-DGN) that approximates spatial derivatives to use them to represent PDEs which have a prominent role for physics-aware modeling.
PA-DGN effectively learns the modulated derivatives for predictions and the derivatives can be used to discover hidden physics describing interactions between temporal and spatial derivatives.
A CKNOWLEDGEMENTS
This work is supported in part by NSF Research Grant IIS-1254206 and MINERVA grant N00014-171-2281, granted to co-author Yan Liu in her academic role at the University of Southern California.
The views and conclusions are those of the authors and should not be interpreted as representing the ofﬁcial policies of the funding agency or the U
S. Government.
Last but not least, we appreciate anonymous reviewers for your thorough comments and suggestions.
A. E XPERIMENT S ETTINGS
Here we provide additional details for models we used in this work, including model architecture settings and hyper-parameter settings.
A.2. M ODEL S ETTINGS
Unless mentioned otherwise [Table] all models use a hidden dimension of size 64.
• VAR :
A vector autoregression model with 2 lags.
Input is the concatenated features of previous 2 frames.
The weights are shared among all nodes in the graph.
• MLP :
A multilayer perceptron model with 2 hidden layers.
Input is the concatenated features of previous 2 frames.
The weights are shared among all nodes in the graph.
• GRU : A Gated Recurrent Unit network with 2 hidden layers.
Input is the concatenated features of previous 2 frames.
The weights are shared among all nodes in the graph.
• RGN :
A recurrent graph neural network model with 2 GN blocks.
Each GN block has an edge update block and a node update block, both of which use a 2-layer GRU cell as the update function.
We set its hidden dimension to 73 so that it has the same number of learnable parameters as our proposed model PA-DGN.
• RGN(StandardOP) :
Similar to RGN, but use the output of difference operators in Section 2.1 as extra input features.
We set its hidden dimension to 73.
• RGN(MeshOP) : Similar to RGN(StandardOP), but the extra input features are calculated using opeartors in Section 2.2.
We set its hidden dimension to 73.
• PA-DGN :
Our proposed model.
The spatial derivative layer uses a message passing neural network (MPNN) with 2 GN blocks using 2-layer MLPs as update functions.
The forward network part uses a recurrent graph neural network with 2 recurrent GN blocks using 2-layer GRU cells as update functions.
The numbers of learnable parameters of all models are listed as follows:
[Table]
A.2. T RAINING S ETTINGS
The number of evaluation runs We performed 3 times for every experiment in this paper to report the mean and standard deviations.
Length of prediction For experiments on synthetic data, all models take ﬁrst 5 frames as input and predict the following 15 frames.
For experiments on NOAA datasets, all models take ﬁrst 12 frames as input and predict the following 12 frames.
Training hyper-parameters We use Adam optimizer with learning rate 1e-3, batch size 8, and weight decay of 5e-4.
All experiments are trained for a maximum of 2000 epochs with early stopping.
All experiments are trained using inverse sigmoid scheduled sampling with the coefﬁcient k = 107 .
Environments All experiments are implemented with Python3.6 and PyTorch 1.1.0, and are run with NVIDIA GTX 1080 Ti GPUs.
A. E FFECT OF DIFFERENT GRAPH STRUCTURES
In this section, we evaluate the effect of 2 different graph structures on baselines and our models:
(1) k -NN : a graph constructed with k -NN algorithm ( k = 4 ); (2) TriMesh : a graph generated with Delaunay Triangulation.
All graphs use the Euclidean distance.
[Table] [Table]
Table 5 and Table 6 show the effect of different graph structures on the synthetic dataset used in
Section 3.2 and the real-world dataset in Section 4.2 separately.
We ﬁnd that for different models the effect of graph structures is not homogeneous.
For RGN and PA-DGN, k -NN graph is more beneﬁcial to the prediction performance than TriMesh graph, because these two models rely more on neighboring information and a k -NN graph incorporates it better than a Delaunay Triangulation graph.
However, switching from TriMesh graph to k -NN graph is harmful to the prediction accuracy of RGN(MeshOP) since Delaunay Triangulation is a well-deﬁned method for generating triangulated mesh in contrast to k -NN graphs.
Given the various effect of graph structures on different models, our proposed PA-DGN under k -NN graphs always outperforms other baselines using any graph structure.
MAE distribution for 1-step prediction 124 122 120 118 116 114 32. MAE distribution for 6-step prediction 124 122 120 118 116 114 32.
MAE distribution for 12-step prediction 0.
A. T HE DISTRIBUTION OF PREDICTION ERROR ACROSS NODES
Figure 7 provides the distribution of MAEs across the nodes of PA-DGN applied to the graph signal prediction task of the west coast region of the real-world dataset in Section 4.2.
As shown in the ﬁgure, nodes with the highest prediction error for short-term prediction are gathered in the inner part where the observable nodes are sparse, while for long-term prediction nodes in the area with a limited number of observable points no longer have the largest MAE.
This implies that PA-DGN can utilize neighboring information efﬁciently even under the limitation of sparsely observable points.
A. E VALUATION ON NEMO SEA SURFACE TEMPERATURE (SST) DATASET
We tested our proposed method and baselines on the NEMO sea surface temperature (SST) dataset 1 .
We ﬁrst download the data in the area between 50N ◦ - 65N ◦ and 75W ◦ - 10W ◦ starting from 2016-0101 to 2017-12-31, then we crop the [0 , 550] ×
[100 , 650] square from the area and sample 250 points from the square as our chosen dataset.
We divide the data into 24 sequences, each lasting 30 days, and truncate the tail.
All models use the ﬁrst 5-day SST as input and predict the SST in the following 15 and 25 days.
We use the data in 2016 for training all models and the left for testing.
For StandardOP, MeshOP and SDL, we test both options using linear regression and using RGN for the prediction part and report the best result.
The results in Table 7 show that all methods incorporating spatial differences gain improvement on prediction and that our proposed learnable
SDL outperforms all other baselines.
[Table]
A. E VALUATION ON DATASETS WITH DIFFERENT SPARSITY
We changed the number of nodes to control the sparsity of data.
As shown in Table 8, our proposed model outperforms others under various settings of sparsity on the synthetic experiment in Section 3.2.
[Table]
Furthermore, we sampled 400 points and trained SDL as described in Section 3.
[Table]
and resampled fewer points (350,300,250,200) to evaluate if SDL generalizes less sparse setting.
As Table 9 shows,
MSE increases when fewer sample points are used.
However, SDL is able to provide much more accurate gradients even if it is trained under a new graph with different properties.
Thus, the results support that SDL is able to generalize the c setting.
[Table]