On the Robustness of Graph Neural Diffusion to Topology Perturbations
Abstract
Neural diffusion on graphs is a novel class of graph neural networks that has attracted increasing attention recently.
The capability of graph neural partial differential equations (PDEs) in addressing common hurdles of graph neural networks (GNNs), such as the problems of over-smoothing and bottlenecks, has been investigated but not their robustness to adversarial attacks.
In this work, we explore the robustness properties of graph neural PDEs.
We empirically demonstrate that graph neural PDEs are intrinsically more robust against topology perturbation as compared to other GNNs.
We provide insights into this phenomenon by exploiting the stability of the heat semigroup under graph topology perturbations.
We discuss various graph diffusion operators and relate them to existing graph neural PDEs.
Furthermore, we propose a general graph neural PDE framework based on which a new class of robust GNNs can be defined.
We verify that the new model achieves comparable state-of-the-art performance on several benchmark datasets.
Introduction
Deep learning on graphs and Graph Neural Networks (GNNs), in particular, have achieved remarkable success in a variety of application areas such as those related to chemistry (molecules), finance (trading networks) and the social media (the Facebook friend network)
GNNs have been applied to various tasks including node classification [3], link prediction [6] and recommender systems [7].
The key to the success of GNNs is the neural message passing scheme [8] where messages are propagated along edges and optimized toward a downstream task.
While aggregating neighboring nodes’ information is a powerful principle of representation learning, the way that GNNs exchange information between nodes makes them vulnerable to adversarial attacks [9].
Adversaries can perturb a graph’s topology by adding or removing edges [10–12] or by injecting malicious nodes to the original graph [13–15].
Another common attack is to perturb node attributes [9, 16–18].
Our paper will mainly tackle graph topology perturbation.
Adversaries who can inject nodes to the original graph while not modifying the original graph directly are called injection attacks [13–15].
Adversaries who can directly modify the original graph including edges and node features are called modification attacks [10–12, 9, 16–18].
To defend against adversarial attacks, several robust GNN models have been proposed.
Examples include RobustGCN [19], GRAND [20], and ProGNN [21].
In addition, pre-processing based defenders include GNN-SVD [22] and GNNGuard [23].
Recent studies [24–27] have applied neural Ordinary Differential Equations (ODEs) [28] to defend against adversarial attacks.
Some works like [24, 27] have revealed interesting intrinsic properties of
ODEs that make them more stable than conventional convolutional neural networks (CNNs).
Neural
Partial Differential Equations (PDEs) have also been applied to graph-structured data [29, 30].
Some papers like [29] (GRAND) and [30] (BLEND) approach deep learning on graphs as a continuous diffusion process and treat GNNs as spatial discretizations of an underlying PDE.
However, robustness to adversarial attack has not been studied on such graph neural PDEs.
In this work, we demonstrate that graph neural PDEs are intrinsically more robust against adversarial topological perturbations compared to other GNNs.
We investigate the heat diffusion on a general Riemannian manifold and show that the diffusion process is essentially stable under small perturbations of the manifold metric.
In doing so, we provide insights into the underlining reasons why graph neural PDEs are stable under graph topological perturbations.
Such insights indicate that further improvements to the design of graph neural PDEs are possible.
We develop several such improved models under a general graph neural PDE framework and show that these models are also robust to node attribute perturbations.
Main contributions .
In this paper, our objective is to develop a general diffusion framework on graphs and study the robustness properties of the induced graph neural PDEs.
Our main contributions are summarized as follows:
• We review the notion of heat diffusion on Riemannian manifolds and the stability of its semigroup.
We present analogous concepts of gradient, divergence, and Laplacian operators for heat diffusion on a graph. • We generalize heat flow to more general flow schemes, including mean curvature flow and Beltrami flow, which are able to preserve inter-class edges in diffusion.
A novel class of graph neural PDEs is thereby induced.
• We show that the proposed graph neural PDEs are intrinsically robust to graph topology perturbations.
We verify that the new model achieves comparable state-of-the-art performance on several benchmark datasets.
The rest of this paper is organized as follows.
In Section 2, we start with preliminaries on continuous diffusion over a Riemannian manifold and the discrete graph.
In Section 3, we present our main results on the stability properties of heat flow on graphs and generalize heat flow to edge-preserving flows from which a new class of graph neural PDEs is proposed.
Our proposed model architecture is detailed in Section 4.
We summarize experimental results in Section 5 and conclude the paper in
Section 6.
The proofs for all theoretical results in this paper are given in the supplementary material, where more experiments are also presented.
Preliminaries
Similar to [29, 30], we consider a graph as a discretization of a Riemannian manifold.
We introduce concepts and notations for flows diffused over a general manifold and a discrete graph.
Readers are referred to [31] for more details.
The stability of the heat kernel and heat semigroup of the heat diffusion equation under perturbations of the manifold metric is considered.
This discussion sheds light on the diffusion stability of graphs under perturbation of the graph topology in the next section.
2.1 Heat Equation and Solution Stability on Manifold
A Riemannian manifold ( M, g ) is a smooth manifold M endowed with a Riemannian metric g , where the norms of tangent vectors and the angles between them are defined by an inner product.
It is the natural generalization of Euclidean space and correspondingly the divergence, gradient, and
Laplace operators are well-defined, analogous to their corresponding concepts in Euclidean space.
Let C ∞ 0 ( M ) denote the space of smooth functions on M with compact support and ◦ the composition operation.
Definition 1.
The Laplace operator ∆ : C ∞ 0 ( M ) (cid:55)→ C ∞ 0 ( M ) on a d -dimensional Riemannian manifold ( M, g ) is defined as
[Equation]
where div is the divergence defined for the C ∞ 0 vector fields on M and ∇ is the gradient operator.
In a local chart U with coordinates ( x 1 , x 2 , . . . , x n ) , for function f and vector field ψ , we have
[Equation]
where ψ j is the j -th component of ψ , g ij the components of the inverse metric of g , and det is the determinant operator of a matrix.
We also have
[Equation]
The classical Cauchy problem associated with the heat diffusion equation is to find a function φ ( t, x ) ∈ C ∞
( R + × M ) such that
[Equation]
where we only consider f ∈ C ∞ 0 and R + is the space of positive real numbers.
We can extend the Laplace operator to the generalized Dirichlet Laplace operator , denoted as
L = − ∆ and defined on a larger domain 2 [31, section 4.2].
This operator is self-adjoint and nonnegative definite on L 2 ( M ) , and it can be shown that the above problem (3) is solved by means of the following family { P t } t ≥ 0 of operators:
[Equation]
where { E λ } λ ∈ spec L in L 2 ( M ) is the unique spectral resolution of the Dirichlet Laplace operator L and spec L is the spectrum set of L .
The family { P t } t ≥ 0 is called the heat semigroup associated with ∆ , and the solution is given by P t f .
It is also well-known that the solution of the above problem has an integration form via the (minimal) heat kernel function k t ( x, y ) :
R + × M × M
(cid:55)→ R [31,
Theorem 7.13]:
[Equation]
where d µ is the Riemannian measure [31, Section 3.4] on M .
For a Riemannian manifold M , different metrics g give rise to different structures.
Let (cid:101) ∆ be the
Laplacian associated with another metric ˜ g on M such that, for some α ≥ 1 ,
[Equation]
The new (cid:101) ∆ is said to be quasi-isometric [32] to g , and can be viewed as a perturbation of the metric g .
(cid:101) ∆ can be also viewed as a uniformly elliptic operator [32] with respect to (w.r.t.) to g .
The stability of the heat semigroup and the heat kernel under perturbations of the Laplace operator (i.e., changes of g ) is well studied in [33].
In the special case of uniformly elliptic operators on Euclidean manifold R d , the uniformly elliptic operator can be written as div( A ∇ ) with symmetric measurable coefficients
A ( x ) =
( a ij ( x )) such that (cid:80) i,j a ij ( x ) ξ i ξ j ≥ λ 0 | ξ | 2 for all ξ ∈ R d and for some constant λ >independent of x .
The reference [33] shows the following theorem:
Theorem 1.
On the Euclidean manifold R n , let { P t } t ≥ 0 and { (cid:101) P t } t ≥ 0 be two diffusion semigroups on R d ( d ≥ 2) associated with uniformly elliptic operators div( A ∇ ) and div( ˜ A ∇ ) with symmetric measurable coefficients
A ( x ) = ( a ij ( x )) and ˜ A ( x ) =
(˜ a ij ( x )) , respectively.
The corresponding heat kernels are denoted by p t ( x, y ) and ˜ p t ( x, y ) .
We then have that
i. There is a bounded, piecewise continuous function F 1 ( t, z ) on R + × R + with lim
z
→ 0 F 1 ( t, z ) =
for each t > 0 and a constant c > 0 , both of which depend only on d and α , such that
[Equation]
for any ( t, x, y ) ∈ R + × R d × R d , ∥ · ∥ 2 is the Euclidean norm, and ∥ · ∥ L 2 loc is the local L 2 -norm distance defined in [33]. ii.
Furthermore, we also have an L p -operator norm bound for P t − (cid:101) P t in terms of the local L 2 -norm distance between a ij and ˜ a ij : There is a bounded, piecewise continuous function F 2 ( t, z ) on R + × R + with lim
z → 0 F 2 ( t, z )
=
for each t > 0 that depends only on d and α , such that for any p ∈ [1 , ∞ ] , we have
[Equation]
The above theorem shows the pointwise convergence of the heat kernel under perturbations of the matrix A .
It proves the stability of the solution under small perturbations of the Laplace operator.
These results can also be extended to general Riemannian manifolds [34] since every Riemannian manifold can be isometrically embedded into some Euclidean space [35] and the kernel is isometrically invariant [31, theorem 9.12].
We therefore know that if the difference between ˜ A and A (or more generally ˜ g and g ) are small, the final solutions to (3) have small difference.
An intuitive explanation of the stability of the solution comes from the fact that the heat kernel k t ( x, y ) is the transition density function of a Brownian motion on the manifold [36] since Brownian motion can be viewed as a diffusion generated by half of the Laplace operator.
This means that k t ( x, y ) is a weighted average over all possible paths between x and y at time t , which does not change dramatically under small perturbations of g .
More specifically, for a Brownian motion on a domain M , if we perturb the g over a subset P ⊂ M then only the paths passing through P will be affected.
If we go one step further from the uniformly elliptic operators div( A ∇ ) defined in Theorem 1 by extending A ( x ) φ to A ( x, t, φ, ∇ φ ) , we get the following general quasilinear parabolic equation :
[Equation]
where the principle part A ( x, t, φ, ∇ φ ) is equipped with additional structure conditions as shown in [37].
When A ( x, t, φ, ∇ φ )
=
A ( x, t ) is a time-dependent diffusion process, analogous bounds to
Theorem 1 are shown in [38].
We leave the analysis of general A ( x, t, φ, ∇ φ ) for future work.
2.2 Gradient and Divergence Operators on Graphs
We consider a graph as a discretization of a Riemannian manifold.
More specifically, consider an undirected graph G =
( V , E ) consisting of a finite set V of vertices, together with a subset E ⊂ V × V of edges.
A graph is weighted when it is associated with a function w : E (cid:55)→ R + which is symmetric, i.e., w ([ u, v ])
=
w ([ v, u ]) , for all [ u, v ] ∈ E .
Let H ( V ) denote the Hilbert space on V of real-valued functions with the inner product defined as ⟨ a, b ⟩ H ( V )
= (cid:80) v ∈V a ( v ) b ( v ) , for all a, b ∈ H ( V ) .
Similarly, we define a Hilbert space H ( E ) with inner product ⟨ c, d ⟩ H ( E ) =
(cid:80) [ u,v ] ∈E c ([ u, v ])
d ([ u, v ]) , for all c, d ∈ H ( E ) .
We next define the gradient and divergence operators on graphs analogue to the general continuous manifold defined in
Definition 1 as follows [39].
Definition 2.
Given an undirected graph G =
( V , E ) , we define the following:
i. The graph gradient is an operator ∇ : H ( V ) (cid:55)→ H ( E ) defined by
[Equation]
where h ( v ) =
(cid:80) [ u,v ] ∈E w ([ u, v ]) is the degree of node v .
ii.
The graph divergence is an operator div : H ( E ) (cid:55)→ H ( V ) defined by
[Equation]
iii.
The graph Laplacian is an operator ∆ : H ( V ) (cid:55)→ H [Equation] V ) defined by
[Equation]
The graph Laplacian defined above is identical to the normalized Laplacian matrix, i.e.,
[Equation]
where D is a diagonal matrix with D ( v, v )
=
h ( v ) , and W is the adjacency matrix satisfying
W ( u, v ) =
w ([ u, v ]) if [ u, v ] ∈ E and W ( u, v ) = 0 otherwise.
Note that in (8) we have included the negative sign.
The analogue of the graph Laplacian ∆ in Section 2.1 is the Dirichlet Laplace operator L .
Note the use of the same notation ∆ for graphs.
The manifold Laplace operator ∆ and graph Laplacian ∆ will be apparent from the context.
Neural Diffusion and Stability on Graphs
We now consider neural diffusion on graphs by making use of concepts from Section 2.
Various parabolic-type equations on graphs are studied in this section.
The solution stability against adversarial attacks is linked to the stability of the solution for the heat diffusion equation under perturbation of the manifold metric, introduced in Section 2.1.
The general framework we consider is based on parabolic-type equations on graphs:
[Equation]
with φ ( u, 0) being the initial node attribute at node u and where A ( u, t, φ, ∇ φ ) can take different forms.
We next provide some examples.
3.1 Continuous Diffusion on Graphs
Definition 3 (Heat diffusion) .
The heat diffusion on graphs is defined by
[Equation]
Definition 4 (GRAND/BLEND [29, 30]) .
According to [29, eq (1)] and [30, eq (7) and (9)], the
GRAND/BLEND flow is defined by
[Equation]
where ∇ t is an adaptive Laplace operator (with possible graph rewiring) depending on the evolved node feature φ ( · , t ) .
The gradient operator defined in GRAND/BLEND assumes constant edge weight.
In Definition 4, the weight function w ([ u, v ]) defined in (6) is incorporated into the gradient definition, so it can be absorbed in the time-dependent term ∇ t φ .
Note that BLEND degenerates to GRAND [29] when there is no positional encoding.
In this paper, for a fair comparison, we do not use positional encoding for all GNNs.
Analogous to [40], we can define mean curvature flow and Beltrami flow as follows.
Definition 5.
The mean curvature diffusion on graphs is defined by
[Equation] [Equation]
where − 1 2div (cid:16) ∇ φ ∥∇ φ ∥ (cid:17) is a discrete analogue of the mean curvature operator, ∥∇ φ ∥ = ⟨∇ φ, ∇ φ ⟩ 1 / 2 H ( E ) and ∥∇ φ ( u, t ) ∥
= (cid:16)(cid:80)
[ v,u ] ∈E ( ∇ φ ([ u, v ] , t )) 2 (cid:17) 1 / 2 .
Definition 6.
The Beltrami diffusion on graphs is defined by
[Equation]
Intuitively, the term ∥∇ φ ( u, t ) ∥ , which appears in (13) and (14) but not in (11), measures the smoothness of the signals in the neighborhood around vertex u at time t .
The diffusion using (13) or (14) at vertex u is small when ∥∇ φ ( u, t ) ∥ is large, i.e., signals are less smooth around vertex u .
Hence, mean curvature flow and Beltrami flow are able to preserve the non-smooth graph signals.
This phenomenon is visualized in Fig.
1, where mean curvature flow and Beltrami flow are capable of preserving inter-class edges whose weights are much larger than those in heat flow.
3.2 Stability against Graph Topology Perturbation
Attackers can modify the original graph by adding or removing edges [10–12] and perturbing node attributes.
Adding or removing edges leads to a different graph Laplacian ∆ defined in (8).
Consequently, the solution to (11) is altered.
Using the language of Section 2.1, the edge perturbations correspond to the perturbation of the metric g (or equivalently A in Theorem 1).
According to
Theorem 1, the stability of the heat semigroup and the solutions under small perturbations of the
Laplace operator is guaranteed.
Consequently, the solution is not affected significantly by the edge perturbations.
This is different from the notion of stability in [20], where the output φ ( u, t ) is shown to be stable after perturbation of node attributes φ ( u, 0) with fixed graph edges.
The stability studied in [20] is called Lyapunov stability , which is addressed in Proposition 2.
These two notions of stability combined lead to a graph neural PDE in Section 4 that is robust against both edge and node attacks.
The following result shows the stability of the solution of (11), i.e., ∂φ ( u,t ) ∂t = − ∆ φ ( u, t ) , under graph topology or Laplacian perturbation.
If the change in the graph topology with respect to any matrix norm is small, the semigroup perturbation can be bounded similarly as the result in Theorem 1.
This result considers only the case where the Laplace operator is time-invariant.
The time-variant analogy of [38] as discussed after (5) is provided in the supplementary material.
Proposition 1. Consider ∆ =
D − 1 / 2 ( D − W ) D − 1 / 2 in (9) and suppose ˜∆ = ˜ D − 1 / 2 ( ˜ D − ˜ W ) ˜ D − 1 / 2 with ˜ D being the diagonal degree matrix defined analogously to (9) for ˜ W = W + E .
Suppose ε := ~ E ~ = o (1) for a matrix norm ~ · ~ , and D and ˜ D are non-singular.
Then, ∥ φ ( u, t ) − ˜ φ ( u, t ) ∥
=
O ( ε ) .
3.3 Neural Flows on Graphs
Substituting (6) and (7) into (14), we obtain the neural Beltrami flow as
[Equation]
(15) where ∥∇ φ ( u ) ∥
=
(cid:113)(cid:80) [ v,u ] ∈E ( ∇ φ ([ u, v ])) 2 .
Here, we further assume φ ( u ) , for all nodes u are time-independent for the sake of simplicity.
Suppose |V| = n .
The function φ : V (cid:55)→ R d maps each node to a feature vector.
Stacking all the feature vectors together, we obtain Z ∈ R n × d .
Let the weight function w ([ u, v ]) be the scaled dot product attention function [41] given by
[Equation]
where W K and W Q are the key and query learnable matrices, respectively, and d K denotes the number of rows W K has.
If multi-head attention is applied and denote A h as the attention matrix associated with head h , then A ( Z ) = 1 h (cid:80) h
A h ( Z ) with A h ( u, v ) =
w h ([ u, v ]) where w h is the weight function for head h .
The diffusion equation (15) can be compactly written in matrix form as
[Equation]
where ⊙ denotes element-wise multiplication, B ( u, v ) = softmax (cid:16)
[Equation]
and Ψ ( Z ( t )) is a diagonal matrix with Ψ( u, u )
=
(cid:80) v ( A ⊙ B ) ( u, v ) .
Similarly, the diffusion equations using mean curvature flow (13) and heat flow (11) can be written as:
[Equation]
(18) where B ( u, v ) = softmax
[Equation]
and ∂ Z (
[Equation]
(19) respectively.
Although the BLEND model in [30] is inspired from Beltrami flow, its final formulation (equation (8) in [30] where Z ( t ) contains positional and node feature embeddings) is indeed (19), i.e., heat flow using attention weight function.
Proposition 2.
Diffusion equations (17) , (18) and (19) are all Lyapunov stable [42].
Model Architecture with Lipschitz Constraint
In this section, we propose a general graph neural PDE network based on the discussion in Section 3.
A layer of graph neural PDE is illustrated in Fig.
Since we may stack up multiple such layers, the diffusion is performed in a hierarchical manner, where the high-dimensional features are diffused over the graph at the front layers and the low-dimensional features are diffused at the back layers.
One important ingredient in our model is that we perform spectral normalization on W K and W Q in (16).
This is motivated by the fact that attention models have poor performance when the depth increases.
In other words, attention weights tend to be uniformly distributed for excessive message exchanges.
This phenomenon becomes even more obvious when the nodes’ features are diffused according to (17), (18) or (19) as solving such a PDE normally requires many discrete steps.
To overcome this over-smoothing problem, we utilize the strategy proposed in [43] to enforce Lipschitz continuity by normalizing the attention scores.
Besides, enforcing Lipschitz continuity can also help improve the robustness of the model because the Lipschitz constant controls the perturbation of the output given a bounded input perturbation.
Fig.
3 illustrates that attention weights become overly smooth if no spectral normalization is applied. [Figure]
Experiments
In this section, we compare graph neural PDEs under different flows to popular GNN architectures:
GAT [41], GraphSAGE [45], GIN [46], APPNP [47], and the state-of-the-art GNN defenders: RobustGCN [
GNNGuard [23] GCNSVD [22], on standard node classification benchmarks.
In our ex- [48] and PubMed (biomedical literature) [49].
We use a refined version of these datasets provided by [50].
We refer the readers to the supplementary material for more details.
Our experiment codes are provided in https://github.com/zknus/Robustness-of-Graph-Neural-Diffusion .
[Table]
5.1 Attack Setup
We apply the setup introduced in the graph robustness benchmark (GRB) [50].
Based on the assumption that nodes with lower degrees are easier to attack, GRB constructs three test subsets of nodes with different degree distributions.
According to the average degrees, GRB defines these subsets of nodes as Easy, Medium, Hard, or Full.
In the Easy subset, attacks are easy to succeed and hence the worst performance is expected.
The remaining nodes are divided into a train set (60%) and val set (10%), for training and validation respectively.
In our experiments, we choose the Easy subset, i [Table] e., the most challenging mode.
Following GRB, in this paper, we mainly consider the following real-world adversarial attack settings:
Black -box : The attacker does not know the defender’s method and vice versa.
Attackers firstly attack a pre-trained GCN [3] and then transfer the perturbed graphs to the target model.
Evasion : Attacks will only happen during the inference phase.
Inductive : GNNs are used to classify unseen data (e.g., new users), i.e., validation or test data are unseen during training.
Injection : Attackers can only inject new nodes but not modify the target nodes directly.
This reflects applications like in online social networks where it is usually hard to hack into users’ accounts and modify their profiles.
However, it is easier to create fake accounts and connect them to existing users.
For other attack settings such as white-box attacks and modification attacks, relevant experiments and discussions are provided in the supplementary material.
[Table]
We apply two state-of-the-art injection attack methods: SPEIT [14] and Topological Defective Graph
Injection Attack (TDGIA) [15].
These two attacks are the two strongest attacks reported in the GRB.
For both attacks, both the maximum allowable injected nodes and the maximum allowable injected edges are set to 50 for Cora and Citeseer, and 300 for PubMed.
We provide more details about how these two attacks work under evasion, black-box, and injection setting in the supplemental material.
Table 1 indicates that graph neural PDE induced from Beltrami flow is more robust than all other GNNs except for GNNGuard, which is specifically designed to remove malicious edges and is thus robust against injection attacks.
This suggests that the output features from a graph PDE are stable under topology perturbations, as suggested by Proposition 1.
Furthermore, the heat diffusion process on a manifold tends to diffuse differently under different geometries.
For example, it diffuses slower at points with positive curvature, and faster at points with negative curvature [34, 51].
In [52] and [53], the authors show different datasets have different geometric properties like hyperbolicity distribution or Balanced Forman curvature.
Our theoretical analysis only shows a loose uniform bound in terms of the adjacency matrix, while the performance is very much dataset-dependent.
More advanced theoretical analysis for different datasets is highly non-trivial and needs further investigations.
Our graph neural PDE can be combined with GNNGuard.
We denote this model as BeltramiGuard.
Table 2 demonstrates that BeltramiGuard renders attacks in vain and exceeds GNNGuard on the
Citeseer and PubMed datasets.
Note from Table 1 that the vanilla Beltrami model already surpasses GNNGuard on the large PubMed dataset.
More details are provided in the supplementary material.
5.2 Ablation Studies Diffusion Schemes
We compare different diffusion equations: heat flow (19) where w ([ u, v ])
= 1 for all [ u, v ] ∈ E ,
GRAND/BLEND [29, 30] which is another heat flow where w ([ u, v ]) in (19) is the attention function defined in (16), mean curvature flow (18), and Beltrami flow (17).
For GRAND/BLEND, we stack three neural PDE layers using the architecture proposed in Fig.
2 since the original GRAND/BLEND in [29, 30], which has only one PDE layer, does not perform well.
[Table]
From Table [Table] we observe that even the vanilla time-invariant heat flow preserves some robustness as compared to non-PDE GNNs in Table 1.
This further validates our theoretical analysis in Proposition [Table] which suggests that if the topology perturbation is bounded, the learned representations are close to those under the “clean” scenario.
We can observe that 1) the flows which are capable of preserving non-smooth features are generally more robust than heat flow, 2) the proposed mean curvature flow and Beltrami flow are more robust than GRAND/BLEND, and 3) these two flows generally suffer less performance variance than the other methods.
More Experiments
Due to space constraint, we refer the reader to the supplementary material for more ablation studies including the impact of the Lipschitz constraint, PDE solvers, the number of layers, and time complexity.
The model performance under other attacks is also presented in the supplementary material.
Conclusion
In this paper we have introduced a general graph neural PDE framework from which several graph neural PDEs are proposed.
We analyzed the robustness of the graph neural PDEs and showed that graph neural PDEs are inherently robust against topology perturbations and are also Lyapunov stable.
We provided theoretical evidence showing that the robustness of graph neural PDEs stems from the stability of the heat kernel and semigroup during the diffusion process.
Moreover, we conducted extensive experiments that empirically verify the robustness of the proposed graph neural PDEs when compared with an existing graph neural PDE induced from approximated heat flow, popular GNNs and the state-of-the-art GNN defenders.  Acknowledgement This research is supported by the Singapore Ministry of Education Academic Research Fund Tier 2 grant MOE-T2EP20220-0002 and A*STAR under its RIE2020 Advanced Manufacturing and Engineering (AME) Industry Alignment Fund – Pre Positioning (IAF-PP) (Grant No. A19D6a0053)
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] .
We have left more complicated problems to future work.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] .
See the Broader Impact section in supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] .
See Sectionand Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] .
See supplementary material.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] .
See supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] .
See Section 5 and supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] .
We run each experiment for 10 times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] .
See Section 5. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5 for the open-source datasets and models we have used from GitHub.
(b) Did you mention the license of the assets?
[No] .
Please see the licenses given in the GitHub link or datasets websites.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] .
No new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] .
The datasets used in the paper are all open-source datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] .
There is no identifiable information or offensive content in the datasets.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
In Appendix A of this supplementary material, we provide more details about the datasets used in our main paper. We briefly describe the attack settings in Appendix B. More experiments and ablation studies that are not included in the main paper due to space constraints are now presented in Appendix C. We include supplemental experiments with more datasets in Appendix D. Further ablation studies of our model are also presented in Appendix E. In addition, we present an extension of mean curvature and Beltrami flows in Appendix F. Additional implementation details of the models are provided in Appendix G.
The proofs for all theoretical results in the main paper are given in Appendix H. A Datasets In our main paper
[Table]
we conduct experiments using the first three datasets in Table 4:
Cora (citation networks) [44], Citeseer (citation networks) [48] and PubMed (biomedical literature) [49].
In this supplementary material
[Table]
we conduct further experiments using the other three datasets in Table 4:
Flickr (social networks) [54], Coauthor (academic networks) [55] and Amazon Computer (recommendation networks) [56].
We use a refined version of these datasets provided by [50]
[Table]
where the main statistics are summarized in Table 4.
The features in these datasets are normalized by an arctan (bijective) transformation [50]
[Table]
which permits attackers to restore the original features and thus allows real-world adversarial attacks
[Table]
[Table]
B Attack Settings
There are three common categories of adversarial attacks studied in the literature [50]:
• Poison (attack occurs in training) or evasion (attack occurs in testing).
• White-box (attackers knows target model/method) or black-box (attackers do not know target model/method and thus need to attack a surrogate model and then transfer to target model).
• Injection (attackers inject nodes/edges to the original graph and generate attributes for the injected nodes) or modification (attackers modify the original graph including its topology and node features directly).
Our experiments in the main paper mainly focus on the evasion, black-box, and injection attack setting, which we believe are the most realistic attack settings.
The two selected attack methods SPEIT and TDGIA are tailored for these attack settings.
To be more specific, we carry out the following:
• Evasion: SPEIT and TDGIA are performed on a trained model during testing time.
• Black-box: SPEIT and TDGIA are used to attack a trained GCN,
i
e., a surrogate model, to generate graph perturbations, and then the target model is tested on this perturbed graph.
Injection: when perturbing the graph based on a trained GCN, SPEIT and TDGIA first inject new nodes into the original graph and then generate the injected nodes’ features.
In this supplementary material, we include more attack settings such as white-box attacks and modification attacks.
Further experiments to test graph PDE robustness against node attribute perturbation are also included.
C More Experiments and Ablation Studies
We include more experiments and ablation studies that are not included in the main paper due to space constraints.
More specially, in this section, we conduct experiments to study the effect of the Lipschitz constraint and the number of layers, inference time complexity, more injection attacks including evasion white-box injection attacks and attacks with various attack strengths, and the modification attackers which modify the original graph including its topology and node features directly.
Our codes are developed based on the following two repositories:
• https://github.com/twitter-research/graph-neural-pde and • https://github.com/THUDM/grb ,
where the new diffusion schemes and their induced neural PDEs are developed based on the first repository and we follow the second repository to set up the robustness evaluation benchmark.
C.1 PDE Solvers
The impact of PDE solvers on the performance can be observed in Table 5.
The adaptive step-size solver Dopri5 performs better than the fixed-step solvers (Implicit/Explicit Adam) at the cost of higher computational complexity.
For fixed-step solvers, increasing the step size τ reduces the variance.
For sufficiently large step sizes, the implicit method converges faster than the explicit method.
[Table]
C.2 Lipschitz Constraint and Number of Layers
On one hand, keeping as few layers as possible can mitigate the over-smoothing problem.
On the other hand, too few layers result in the underfitting problem, i.e., the test clean accuracy is low.
To understand this better, we performed experiments using a different number of layers.
The results are summarized in Table 6.
The number of layers is a hyperparameter we tune during training.
We have tried using the same number of layers for GRAND/BLEND as in their paper.
However, the test clean accuracy is low.
This is mainly due to two reasons: 1) we are using the inductive setting whereas the paper of GRAND/BLEND uses transductive training; 2) the datasets we are using have been calibrated for the robustness evaluation, which is different from the original datasets used by the paper of GRAND/BLEND.
For example, grb-cora [50] has node feature size of 302 while the original
Cora dataset has node feature size of 1433.
Table 6 provides empirical evidence for GRAND/BLEND needing more layers in our setting.
C.3 Time Complexity
We have summarized the time complexity in Table 7.
The time is computed by averaging 500 diffusion operations.
We can see that Beltrami and GRAND/BLEND have similar time complexity to the two
[Table]
defenders GNNGuard
GCNSVD small step sizes but incur more computation time than the other
GNNs to complete a diffusion process
Heat, unlike Beltrami and does not use
[Table]
Dopri
[Table]
C.4 White-Box Attacks
We now include the white-box attacks for our model and the baselines.
The results are shown in
Table 8.
We observe that our model outperforms the baselines under white-box attacks.
The results further validate that neural PDEs are intrinsically robust to graph topology perturbations, as indicated by Proposition 1 and Lemma 1.
C.5 Injection Attacks with Various Attack Strengths
Under injection attacks with a different number of injection nodes and edges, we have performed more experiments with various attack strengths.
Table 9 shows that the robustness is not significantly affected by the number of nodes and edges injected by attackers as the original is not perturbed in this setting.
C.6 Modification Attacks
In the modification attack, attackers can directly flip the original graph’s edges and perturb the features of the nodes.
We apply the PGD method to randomly flip edges and then perturb node features.
In
Table 10, we observe that
[Table]
[Table]
1) the robustness performance starts to break down when the 60% of nodes have their features perturbed by ϵ = 0 .
1 (the value of features is in [-1,1]) and 60% of edges are flipped, and 2) as long as feature perturbation is small, the robustness can still be retained even if 80% of nodes have their features perturbed and
80% of edges are flipped.
In summary, from those extensive experiment results, we observe that our models are more robust against topology perturbation than feature perturbation
[Table]
[Table]
C.7 More Discussion about Neural Heat Diffusion
“Heat” in Table 3 of the main paper serves as a baseline.
Recall that both “Heat” and GRAND/BLEND are derived from heat flow, where GRAND/BLEND uses the attention function to weigh edges while “Heat” uses the constant function, which is not learned from the data and hence is not expected to perform well.
In a further experiment
[Table]
we now treat the constant in “Heat” as a trainable variable that is shared by all edges.
We denote this new variant as Heat + and compare it with GAT and
APPNP (which were more robust than "Heat") under SPEIT attack in Table 11.
We now see that
Heat + is more robust.
This constant variable controls the diffusivity on the graph, analogous to the thermal diffusivity on a manifold.
It is interesting to observe such a phenomenon since the experiment indicates that heat diffusivity also affects robustness.
We believe further investigations in this direction can be performed in future work
[Table]
[Table]
D Further Experiments
We repeat the experiments in Table 1 in the main paper but using the three new datasets.
The inherent robustness of the proposed graph neural PDEs is again verified by Table 12, especially on the Flickr and Coauthor datasets
[Table]
[Table]
E Further Ablation Studies
We repeat the experiments in Table 3 of the main paper but using three new datasets.
Similar to what we have stated in the main paper, we observe the advantage of the proposed mean curvature flow and
Beltrami flow in terms of robustness against adversarial attacks.
F Extension
The graph Laplacian and curvature can be generalized to an operator that can be thought of as the discrete analog of the p -Laplacian in the continuous case [39, 37]:
[Equation]
where p = 1 , 2 correspond to the mean curvature and heat (GRAND/BLEND) equations, respectively.
Here, we consider the cases where p > 2 .
Note that when p > 2 , as opposed to the cases where p ≤ 1 , e.g., p = 1 for mean curvature and Beltrami flows, the edges that connect nodes with similar features are potentially preserved in the diffusion process.
Table 14 shows that the PDE’s robustness decreases as p increases.
[Table]
[Table]
G Implementation Details
By default, all the models are implemented using three layers, with layer normalization, 50% dropout at the end of each layer, and hidden feature dimensions 64 − 64 .
Here are some notes that should be taken:
• When dealing with the Cora dataset, PDEs induced by Beltrami and mean curvature flows tend to underfit.
Thus, we implement them using four layers with hidden feature dimensions 128 − 128 − 128 .
• For all neural PDEs, node features are diffused independently over layers, i.e., each layer solves its own PDE.
At each layer, once a diffusion process is over, the parameters in the associated PDE are reset.
This operation is effective to alleviate the overfitting problem.
• By default, the integral period for all PDEs is set to [0 , 1] .
All experiments are repeated 5 to 10 times with different random seeds.
For all the attacks, the maximum number of nodes and edges that are allowed to be perturbed are summarized in Table 15.
[Table]
[Figure]
[Table]
[Figure]
GNNGuard implemented in Table 1 of the main paper is based on GCN, where each layer in
GNNGuard contains two operations: 1) adjusting attention coefficients by pruning likely fake edges and assigning less weight to suspicious edges; and 2) a normal GCN layer.
Regarding BeltramiGuard implemented in Table 2 of the main paper, we replace the second component as mentioned above, i.e., a GCN layer, with a PDE layer induced by Beltrami flow.
H Proofs of Results
In this section, we provide detailed proof of the results stated in the main paper.
The time-variant analogy of [38] as discussed after (5) in the main paper is also presented.
H.1 Proof of Proposition
For the system ∂φ ( u,t ) ∂t = − ∆ φ ( u, t ) , the solution is given by φ ( u, t ) =
e − t ∆ φ ( u, 0) .
We therefore need to derive a bound for ~
e − t ∆ − e − t ˜∆ ~ .
Note for matrix exponent, in general e X + Y ̸ =
e X e Y unless X and Y commute (i.e. XY = Y X ).
However, from [57, eq (3.5) ], we have
[Equation]
where the positive definiteness of ∆ is used.
We further analyze ∆ − ˜ ∆ .
Since ˜ W = W + E with ~ E ~ = ε , we have ~ D − ˜ D
~ = O ( ε ) because norms for a finite dimensional space are equivalent [58, Theorem 5.4.4.].
Let E ′ =
˜ D − 1 / 2 − D − 1 / 2 .
We also have ~ E ′ ~
= O ( ε ) .
It follows that
[Equation]
( ε ) .
We therefore obtain the conclusion that ~ e − t ∆ − e − t ˜∆ ~ = O ( εte − ρt ) for some constant ρ >since ~ e − t ∆ ~
= O ( e − tρ ) according to [57] and the fact that ∆ is positive definite according to our assumption.
The conclusion ∥ φ ( u, t ) − ˜ φ ( u, t ) ∥
=
O ( εte − ρt ) then follows.
H.2 Proof of Proposition
Proof.
Since A is right stochastic, all eigenvalues of A − I have non-positive real parts.
By the
Lyapunov Stability Theorem [42], the stability of (19) is ensured.
Since Ψ − 1 A ⊙ B is right stochastic, all eigenvalues of Ψ − 1 A ⊙ B − I have non-positive parts and thus (18) and (17) are stable.
H.3 The Time-Variant Case
In this section, we provide an analysis of the time-variant system as alluded to before Proposition 1.
In the time-variant system, we have
[Equation]
where ∆( t ) is a time-variant Laplacian operator.
Let the state transition matrix be Φ ∆ ( t ; 0) so that the solution of (21) is given by φ ( u, t )
=
Φ ∆ ( t ; 0) φ ( u, 0) [42].
In [59], the authors discuss the stability radius of time-variant systems.
The system still preserves stability if the perturbation of ∆( t ) is smaller than a stability radius.
In this discussion, we assume the perturbation is small, i.e., it is inside the stability radius so that both ∆( t ) and its perturbed version ˜∆( t ) generate exponentially stable solutions.
In other words, for a given matrix norm ~ · ~ , there exist positive constants M , ˜ M , ω and ˜ ω such that
[Equation]
To simplify the analysis, we further assume the perturbation is small enough such that the exponent difference is also small, i.e.,
[Equation]
We now consider the solution difference after perturbation.
Let D ( t ) and W ( t ) be the time-variant version of D and W , respectively.
They are assumed to satisfy (22) and (23).
Lemma 1.
Let ∆( t ) =
D − 1 / 2 ( t )( D ( t ) − W ( t ))
D − 1 / 2 ( t ) and ˜∆( t ) =
˜ D − 1 / 2 ( t )( ˜ D ( t ) − ˜ W ( t ))
˜ D − 1 / 2 ( t ) with ˜ D ( t ) being the diagonal degree matrix for ˜ W ( t )
=
W ( t ) + E ( t ) , where { ˜ W ( t ) : t ≥ 0 } satisfies (22) and (23) .
Denote ε ( t ) =
~ E ( t ) ~ .
We have ∥ φ ( u, t ) − ˜ φ ( u, t ) ∥ =
O (cid:16) e − ρt (cid:82) t 0 ε ( τ ) d τ (cid:17) for some constant ρ > 0 .
Proof.
For the perturbed system, we have
[Equation]
[Equation]
Formally, (24) can be interpreted as a closed loop system obtained by applying the dynamical feedback (cid:16) ∆( t ) − ˜∆( t ) (cid:17) φ ( u, t ) .
According to [42], the solution of the above system, denoted by ˜ φ ( u, t ) , satisfies
[Equation]
Similar to the proof in Appendix H.1 for the time-invariant case, we have ~ ∆( τ ) − ˜∆( τ ) ~
= O ( ε ( t )) .
It follows that
[Equation]
for some constant ρ > 0 .
In the second equality, we have used the fact that the initial point for both the unperturbed and perturbed system is φ ( u, 0) , and the last equality follows from (22) and (23).
Remark H.1.
For the sake of simplicity, in this work, we only present theoretical analysis for the time-invariant or time-variant graph Laplacian in Proposition 1 and Lemma 1 for heat diffusion (11) .
The more complicated case where the graph Laplacian ∆ depends on node features u as implemented in (16) is left for future work.
From experiment results, for example, the results shown in Table 3 in the main paper and Table 13 in this supplementary material, the time-invariant case already preserve some robustness as compared to non-PDE GNNs in Table 1 and Table 12, which validates our theoretical analysis.
The more robust mean curvature flow and Beltrami flow proposed in the paper are highly non-linear, making a theoretical analysis difficult.
However, our analysis for the time-variant case provides some insights as to why robustness is present in these cases, as demonstrated in our experiments.
Broader Impact
Our work develops robust GNNs to mitigate the threat of adversarial attacks, which can lead to reliable deployment of automation in various applications like sensor networks, transportation networks, and manufacturing.
This may potentially lead to the replacement of repetitive tasks or jobs that are traditionally performed by humans.
However, automation and artificial intelligence (AI) can lead to better productivity, efficiency, and cost-effectiveness with an overall increase in societal living standards.
By incorporating the ability to defend against adversarial attacks, our research can lead to more secure and robust adoption of AI technologies.
However, potential failures and engineering issues remain challenging open problems.