On the Robustness of Graph Neural Diffusion to Topology Perturbations
Anonymous Author(s)
AffiliationAddress email
Abstract
Neural diffusion on graphs is a novel class of graph neural networks that hasattracted increasing attention recently.
The capability of graph neural partial differential equations (PDEs) in addressing common hurdles of graph neural networks(GNNs), such as the problems of over-smoothing and bottlenecks, has been investigated but not their robustness to adversarial attacks.
In this work, we explorethe robustness properties of graph neural PDEs.
We empirically demonstrate thatgraph neural PDEs are intrinsically more robust against topology perturbation ascompared to other GNNs.
We provide insights into this phenomenon by exploitingthe stability of the heat semigroup under graph topology perturbations.
We discussvarious graph diffusion operators and relate them to existing graph neural PDEs.
Furthermore, we propose a general graph neural PDE framework based on which anew class of robust GNNs can be defined.
We verify that the new model achievescomparable state-of-the-art performance on several benchmark datasets.
Introduction
Deep learning on graphs and Graph Neural Networks (GNNs), in particular, have achieved remarkablesuccess in a variety of application areas such as those related to chemistry (molecules), finance (tradingnetworks) and the social media (the Facebook friend network)
GNNs have been applied tovarious tasks including node classification [3], link prediction [6] and recommender systems [7].
Thekey to the success of GNNs is the neural message passing scheme [8] where messages are propagatedalong edges and optimized towards a downstream task.
While the aggregation of neighboring nodes’ information is a powerful principle of representationlearning, the way that GNNs exchange the information between nodes makes them vulnerable toadversarial attacks [9].
Adversaries can perturb a graph’s topology by adding or removing edges[10–12] or by injecting malicious nodes to the original graph [13–15]. Another common attack isto perturb node attributes [9, 16–18].
Our paper will mainly tackle graph topology perturbation. Adversaries who can inject nodes to the original graph while not modifying the original graphdirectly are called injection attacks [13–15]. Adversaries who can directly modify the original graphincluding edges and node features are called modification attacks [10–12, 9, 16–18].
To defendagainst adversarial attacks, several robust GNN models have been proposed.
Examples include RobustGCN [19], GRAND [20], and ProGNN [21].
In addition, pre-processing based defendersinclude GNN-SVD [22] and GNNGuard [23].
Recent studies [24–27] have applied neural Ordinary Differential Equations (ODEs) [28] to defendagainst adversarial attacks.
Some works like [24, 27] have revealed interesting intrinsic properties of
ODEs that make them more stable than conventional convolutional neural networks (CNNs).
Neural
Partial Differential Equations (PDEs) have also been applied to graph-structured data [29, 30].
Somepapers like [29] (GRAND) and [30] (BLEND) approach deep learning on graphs as a continuous diffusion process and treat GNNs as spatial discretizations of an underlying PDE.
However, robustnessto adversarial attack has not been studied on such graph neural PDEs.
In this work, we demonstratethat graph neural PDEs are intrinsically more robust against adversarial topological perturbationscompared to other GNNs.
We investigate the heat diffusion on a general Riemannian manifold andshow that the diffusion process is essentially stable under small perturbations of the manifold metric.
In doing so, we provide insights into the underlining reasons why graph neural PDEs are stable undergraph topological perturbations.
Such insights indicate that further improvements to the design ofgraph neural PDEs are possible.
We develop several such improved models under a general graphneural PDE framework and show that these models are also robust to node attribute perturbations.
Main contributions .
In this paper, our objective is to develop a general diffusion framework ongraphs and study the robustness properties of the induced graph neural PDEs.
Our main contributionsare summarized as follows:
• We review the notion of heat diffusion on Riemannian manifolds and the stability of its semigroup.
We present analogous concepts of gradient, divergence and Laplacian operatorsfor heat diffusion on a graph.• We generalize heat flow to more general flow schemes, including mean curvature flow andBeltrami flow, which are able to preserve inter-class edges in diffusion.
A novel class ofgraph neural PDEs is thereby induced.•
We show that the proposed graph neural PDEs are intrinsically robust to graph topology per- turbations.
We verify that the new model achieves comparable state-of-the-art performanceon several benchmark datasets.
The rest of this paper is organized as follows.
In Section 2, we start with preliminaries on continuousdiffusion over a Riemannian manifold and the discrete graph.
In Section 3, we present our mainresults on the stability properties of heat flow on graphs and generalize heat flow to edge-preservingflows from which a new class of graph neural PDEs is proposed.
Our proposed model architectureis detailed in Section 4.
We summarize experimental results in Section 5 and conclude the paper in
Section 6.
The proofs for all theoretical results in this paper are given in the supplementary material,where more experiments are also presented.
Preliminaries
Similar to [29, 30], we consider a graph as a discretization of a Riemannian manifold.
We introduceconcepts and notations for flows diffused over a general manifold and a discrete graph.
Readers arereferred to [31] for more details.
The stability of the heat kernel and heat semigroup of the heatdiffusion equation under perturbations of the manifold metric is considered.
This discussion shedslight on the diffusion stability on graphs under perturbation of the graph topology in the next section.
2.1 Heat Equation and Solution Stability on Manifold
A Riemannian manifold ( M, g ) is a smooth manifold M endowed with a Riemannian metric g , wherethe norms of tangent vectors and the angles between them are defined by an inner product.
It isthe natural generalization of the Euclidean space and correspondingly the divergence, gradient and
Laplace operators are well-defined, analogous to their corresponding concepts in Euclidean space.
Let C ∞ 0 ( M ) denote the space of smooth functions on M with compact support and ◦ the compositionoperation.
Definition 1.
The Laplace operator ∆ : C ∞ 0 ( M ) (cid:55)→ C ∞ 0 ( M ) on a d -dimensional Riemannianmanifold ( M, g ) is defined as
[Equation]
where div is the divergence defined for the C ∞ 0 vector fields on M and ∇ is the gradient operator.
Ina local chart U with coordinates ( x 1 , x 2 , . .
, x n ) , for function f and vector field ψ , we have( ∇ f ) i = d (cid:88) j = g ij ∂f ∂x j , div ψ = d (cid:88) j =
[Equation]
√ det
[Equation] (cid:16)(cid:112) det [Equation] (cid:17) , (1) where ψ j is the j -th component of ψ , g ij the components of the inverse metric of g , and det is thedeterminant operator of a matrix.
We also have
[Equation]
The classical Cauchy problem associated with the heat diffusion equation is to find a functionφ ( t, x ) ∈ C ∞
( R + × M ) such that
[Equation]
where we only consider f ∈ C ∞ 0 and R + is the space of positive real numbers.
We can extend the Laplace operator to the generalized Dirichlet Laplace operator , denoted as
L = − ∆ and defined on a larger domain 1 [31, section 4.2].
This operator is self-adjoint and nonnegative definite on L 2 ( M ) , and it can be shown that the above problem (3) is solved by means ofthe following family { P t } t ≥ 0 of operators:
[Equation]
where { E λ } λ ∈ spec L in L 2 ( M ) is the unique spectral resolution of the Dirichlet Laplace operator Land spec L is the spectrum set of L .
The family { P t } t ≥ 0 is called the heat semigroup associated with∆ , and the solution is given by P t f .
It is also well-known that the solution of the above problemhas an integration form via the (minimal) heat kernel function k t ( x, y ) :
R + × M × M
(cid:55)→ R [31, Theorem 7.13]:
[Equation]
where d µ is the Riemannian measure [31, Section 3.4] on M .
For a Riemannian manifold M , different metrics g give rise to different structures.
Let (cid:101) ∆ be the
Laplacian associated with another metric ˜ g on M such that, for some α ≥ 1 ,
[Equation]
The new (cid:101) ∆ is said to be quasi-isometric [32] to g , and can be viewed as a perturbation of the metricg .
(cid:101) ∆ can be also viewed as a uniformly elliptic operator [32] with respect to (w.r.t.) to g .
The stabilityof the heat semigroup and the heat kernel under perturbations of the Laplace operator (i.e., changesof g ) is well studied in [33].
In the special case of uniformly elliptic operators on Euclidean manifold 102 R d , the uniformly elliptic operator can be writen as div( A ∇ ) with symmetric measurable coefficients A ( x ) = ( a ij ( x )) such that (cid:80) i,j a ij ( x ) ξ i ξ j ≥ λ 0 | ξ | 2 for all ξ ∈ R d and for some constant λ > 0independent of x . The reference [33] shows the following theorem:
Theorem 1.
On the Euclidean manifold R n , let { P t } t ≥ 0 and { (cid:101) P t } t ≥ 0 be two diffusion semigroupson R d ( d ≥ 2) associated with uniformly elliptic operators div( A ∇ ) and div( ˜ A ∇ ) with symmetricmeasurable coefficients A ( x ) =
( a ij ( x )) and ˜ A ( x ) =
(˜ a ij ( x )) , respectively.
The corresponding heatkernels are denoted by p t ( x, y ) and ˜ p t ( x, y ) .
We then have thati.
There is a bounded, piecewise continuous function F 1 ( t, z ) on R + × R + withlim
z
→ 0 F 1 ( t, z ) =
for each t > 0 and a constant c > 0 , both of which depend onlyon d and α , such that
[Equation]
for any ( t, x, y ) ∈ R + × R d × R d , ∥ · ∥ 2 is the Euclidean norm, and ∥ · ∥ L 2 loc is the local
L 2 -norm distance defined in [33]. ii.
Furthermore, we also have an L p -operator norm bound for P t − (cid:101) P t in terms of the local
L 2 -norm distance between a ij and ˜ a ij : There is a bounded, piecewise continuous function
F 2 ( t, z ) on R + × R + with lim z → 0 F 2 ( t, z )
=
for each t > 0 that depends only on d andα , such that for any p ∈ [1 , ∞ ] , we have [Equation]
The above theorem shows the pointwise convergence of the heat kernel under perturbations of thematrix A .
It proves the stability of the solution under small perturbations of the Laplace operator.
Theseresults can also be extended to general Riemannian manifolds [34] since every Riemannian manifoldcan be isometrically embedded into some Euclidean space [35] and the kernel is isometricallyinvariant [31, theorem 9.12].
We therefore know that if the difference between ˜ A and A (or moregenerally ˜ g and g ) are small, the final solutions to (3) have small difference.
An intuitive explanationof the stability of the solution comes from the fact that the heat kernel k t ( x, y ) is the transition densityfunction of a Brownian motion on the manifold [36] since Brownian motion can be viewed as adiffusion generated by half of the Laplace operator.
This means that k t ( x, y ) is a weighted averageover all possible paths between x and y at time t , which does not change dramatically under smallperturbations of g .
More specifically, for a Brownian motion on a domain M , if we perturb the g overa subset P ⊂ M then only the paths passing through P will be affected.
If we go one step further from the uniformly elliptic operators div( A ∇ ) defined in Theorem 1 byextending A ( x ) φ to A ( x, t, φ, ∇ φ ) , we get the following general quasilinear parabolic equation :
[Equation]
where the principle part A ( x, t, φ, ∇ φ ) is equipped with additional structure conditions as shownin [37].
When A ( x, t, φ, ∇ φ )
=
A ( x, t ) is a time-dependent diffusion process, analogous bounds to
Theorem 1 are shown in [38].
We leave the analysis of general A ( x, t, φ, ∇ φ ) for future work.
2.2 Gradient and Divergence Operators on Graphs
We consider a graph as a discretization of a Riemannian manifold.
More specifically, consider anundirected graph G =
( V , E ) consisting of a finite set V of vertices, together with a subset E ⊂ V × Vof edges.
A graph is weighted when it is associated with a function w : E (cid:55)→ R + which is symmetric,i.e., w ([ u, v ])
=
w ([ v, u ]) , for all [ u, v ] ∈ E .
Let H ( V ) denote the Hilbert space on V of real-valued functions with the inner product definedas ⟨ a, b ⟩ H ( V )
=
(cid:80) v ∈V a ( v ) b ( v ) , for all a, b ∈ H ( V ) .
Similarly, we define a Hilbert space H ( E )with inner product ⟨ c, d ⟩ H ( E ) =
(cid:80) [ u,v ] ∈E c ([ u, v ])
d ([ u, v ]) , for all c, d ∈ H ( E ) .
We next define thegradient and divergence operators on graphs analogue to the general continuous manifold defined in
Definition 1 as follows [39].
Definition 2.
Given an undirected graph G =
( V , E ) , we define the following:
i. The graph gradient is an operator ∇ : H ( V ) (cid:55)→ H ( E ) defined by [Equation]
[Equation]
where h ( v ) =
(cid:80) [ u,v ] ∈E w ([ u, v ]) is the degree of node v . ii
The graph divergence is an operator div : H ( E ) (cid:55)→ H ( V ) defined by
[Equation]
iii.
The graph Laplacian is an operator ∆ : H ( V ) (cid:55)→ H ( V ) defined by
[Equation]
The graph Laplacian defined above is identical to the normalized Laplacian matrix, i.e., [Equation]
[Equation]
(9) where D is a diagonal matrix with D ( v, v )
=
h ( v ) , and W is the adjacency matrix satisfying
W ( u, v ) =
w ([ u, v ]) if [ u, v ] ∈ E and W ( u, v ) = 0 otherwise.
Note that in (8) we have includedthe negative sign.
The analogue of the graph Laplacian ∆ in Section 2.1 is the Dirichlet Laplaceoperator L .
Note the use of the same notation ∆ for graphs.
The manifold Laplace operator ∆ andgraph Laplacian ∆ will be apparent from the context.
Neural Diffusion and Stability on Graphs
We now consider neural diffusion on graphs by making use of concepts from Section 2.
Variousparabolic-type equations on graphs are studied in this section.
The solution stability against adversarialattacks is linked to the stability of the solution for the heat diffusion equation under perturbationof the manifold metric, introduced in Section 2.1.
The general framework we consider is based onparabolic-type equations on graphs:
[Equation]
with φ ( u, 0) being the initial node attribute at node u and where A ( u, t, φ, ∇ φ ) can take differentforms.
We next provide some examples.
3.1 Continuous Diffusion on Graphs
Definition 3 (Heat diffusion) .
The heat diffusion on graphs is defined by
[Equation]
Definition 4 (GRAND/BLEND [29, 30]) .
According to [29, eq (1)] and [30, eq (7) and (9)], the
GRAND/BLEND flow is defined by
[Equation]
(12) where ∇ t is an adaptive Laplace operator (with possible graph rewiring) depending on the evolvednode feature φ ( · , t ) .
The gradient operator defined in GRAND/BLEND assumes constant edge weight.
In Definition 4,the weight function w ([ u, v ]) defined in (6) is incorporated into the gradient definition, so it can beabsorbed in the time-dependent term ∇ t φ .
Note that BLEND degenerates to GRAND [29] whenthere is no positional encoding.
In this paper, for fair comparison, we do not use positional encodingfor all GNNs.
Analogous to [40], we can define mean curvature flow and Beltrami flow as follows.
Definition 5.
The mean curvature diffusion on graphs is defined by
[Equation]
where − 1 2div (cid:16) ∇ φ ∥∇ φ ∥ (cid:17) is a discrete analogue of the mean curvature operator, ∥∇ φ ∥ =⟨∇ φ, ∇ φ ⟩ 1 / 2 H ( E ) and ∥∇ φ ( u, t ) ∥
= (cid:16)(cid:80)
[ v,u ] ∈E ( ∇ φ ([ u, v ] , t )) 2 (cid:17) 1 / [Equation]
Definition 6.
The Beltrami diffusion on graphs is defined by
[Equation]
[Equation] Intuitively, the term ∥∇ φ ( u, t ) ∥ , which appears in (13) and (14) but not in (11), measures thesmoothness of the signals in the neighborhood around vertex u at time t .
The diffusion using (13)or (14) at vertex u is small when ∥∇ φ ( u, t ) ∥ is large, i.e., signals are less smooth around vertex u .
Hence, mean curvature flow and Beltrami flow are able to preserve the non-smooth graph signals.
This phenomenon is visualized in Fig.
1, where mean curvature flow and Beltrami flow are capableof preserving inter-class edges whose weights are much larger than those in heat flow.
3.2 Stability against Graph Topology Perturbation
Attackers can modify the original graph by adding or removing edges [10–12] and perturbingnode attributes.
Adding or removing edges leads to a different graph Laplacian ∆ defined in (8).
Consequently, the solution to (11) is altered.
Using the language of Section 2.1, the edge perturbationscorrespond to the perturbation of the metric g (or equivalently A in Theorem 1).
According to
Theorem 1, the stability of the heat semigroup and the solutions under small perturbations of the
Laplace operator is guaranteed.
Consequently, the solution is not affected significantly by the edgeperturbations.
This is different from the notion of stability in [20], where the output φ ( u, t ) is shownto be stable after perturbation of node attributes φ ( u, 0) with fixed graph edges.
The stability studiedin [20] is called Lyapunov stability , which is addressed in Proposition 2.
These two notions of stabilitycombined lead to a graph neural PDE in Section 4 that is robust against both edge and node attacks.
The following result shows the stability of the solution of (11), i.e., ∂φ ( u,t ) ∂t = − ∆ φ ( u, t ) , undergraph topology or Laplacian perturbation.
If the change in the graph topology with respect to anymatrix norm is small, the semigroup perturbation can be bounded similarly as the result in Theorem 1.
This result considers only the case where the Laplace operator is time-invariant.
The time-variantanalogy of [38] as discussed after (5) is provided in the supplementary material.
Proposition 1. Consider ∆ =
D − 1 / 2 ( D − W ) D − 1 / 2 in (9) and suppose ˜∆ = ˜ D − 1 / 2 ( ˜ D −˜ W ) ˜ D − 1 / 2 with ˜ D being the diagonal degree matrix defined analogously to (9) for ˜ W = W + E .
Suppose ε := ~ E ~ = o (1) for a matrix norm ~ · ~ , and D and ˜ D are non-singular.
Then, ∥ φ ( u, t ) −˜ φ ( u, t ) ∥
=
O ( ε ) .
3.3 Neural Flows on Graphs
Substituting (6) and (7) into (14), we obtain the neural Beltrami flow as
[Equation]
u
[Equation]
u
[Equation] [Equation] [Equation] (15) where ∥∇ φ ( u ) ∥
= (cid:113)(cid:80) [ v,u ] ∈E ( ∇ φ ([ u, v ])) 2 .
Here, we further assume φ ( u ) , for all nodes u aretime-independent for the sake of simplicity.
Suppose |V| = n .
The function φ : V (cid:55)→ R d maps eachnode to a feature vector.
Stacking all the feature vectors together, we obtain Z ∈ R n × d .
Let theweight function w ([ u, v ]) be the scaled dot product attention function [41] given by
[Equation]
where W K and W Q are the key and query learnable matrices, respectively, and d K denotes thenumber of rows W K has.
If multi-head attention is applied and denote A h as the attention matrixassociated with head h , then A ( Z ) = 1 h (cid:80)
h
A h ( Z ) with A h ( u, v ) =
w h ([ u, v ]) where w h is theweight function for head h .
The diffusion equation (15) can be compactly written in matrix form as
[Equation]
where ⊙ denotes element-wise multiplication, B ( u, v ) = softmax (cid:16)
[Equation]
(cid:17)and Ψ ( Z ( t )) is a diagonal matrix with Ψ( u, u )
=
(cid:80) v ( A ⊙ B ) ( u, v ) . Similarly, the diffusion equations using mean curvature flow (13) and heat flow (11) can be written as:
[Equation]
where B ( u, v ) = softmax (cid:16)
[Equation]
and
[Equation]
respectively.
Although the BLEND model in [30] is inspired from Beltrami flow, its final formulation(equation (8) in [30] where Z ( t ) contains positional and node feature embeddings) is indeed (19),i.e., heat flow using attention weight function.
Proposition 2.
Diffusion equations (17) , (18) and (19) are all Lyapunov stable [42].
Model Architecture with Lipschitz Constraint
In this section, we propose a general graph neural PDE network based on the discussion in Section 3.
A layer of graph neural PDE is illustrated in Fig.
Since we may stack up multiple such layers, thediffusion is performed in a hierarchical manner, where the high-dimensional features are diffusedover the graph at the front layers and the low-dimensional features are diffused at the back layers.
One important ingredient in our model is that we perform spectral normalization on W K and W Q
[Equation]
, where each node’s feature vector are linearly transformed before being diffused over the graph. We have Q = A ( Z ( t ))
⊙ B ( Z ( t )) − Ψ ( Z ( t )) for mean curvature flow and Beltrami flow, while Q = A ( Z ( t )) − I for heat flow.
230 in (16).
This is motivated by the fact that attention models have poor performance when the depthincreases.
In other words, attention weights tend to be uniformly distributed for excessive messageexchanges.
This phenomenon becomes even more obvious when the nodes’ features are diffusedaccording to (17), (18) or (19) as solving such a PDE normally requires many discrete steps.
Toovercome this over-smoothing problem, we utilize the strategy proposed in [43] to enforce Lipschitzcontinuity by normalizing the attention scores.
Besides, enforcing Lipschitz continuity can also helpimprove the robustness of the model because the Lipschitz constant controls the perturbation of theoutput given a bounded input perturbation.
Fig.
3 shows that attention weights become overly smoothif no spectral normalization is applied.
Experiments
In this section, we compare graph neural PDEs under different flows to popular GNN architectures:
GAT [41], GraphSAGE [45], GIN [46], APPNP [47], and the state-of-the-art GNN defenders:
RobustGCN [19], GNNGuard [23], GCNSVD [22], on standard node classification benchmarks.
Inour experiments 2 , we use the following datasets: Cora (citation networks) [44], Citeseer (citationnetworks) [48] and PubMed (biomedical literature) [49].
We use a refined version of these datasetsprovided by [50].
We refer the readers to the supplementary material for more details. [Table]
5.1 Attack Setup
We apply the setup introduced in the graph robustness benchmark (GRB) [50].
Based on the assumption that nodes with lower degrees are easier to attack, GRB constructs three test subsets of nodeswith different degree distributions.
According to the average degrees, GRB defines these subsets ofnodes as Easy, Medium, Hard, or Full.
In the Easy subset, attacks are easy to succeed and hence theworst performance is expected.
For the rest of the nodes, they are divided into a train set (60%) andval set (10%), for training and validation respectively.
In our experiments, we choose the Easy subset,i.e., the most challenging mode.
Following GRB, in this paper, we mainly consider the following real-world adversarial attack settings:
Black-box : The attacker does not know the defender’s method and vice versa.
Attackers firstly
attacka pre-trained GCN [3] and then transfer the perturbed graphs to the target model.
Evasion : Attackswill only happen during the inference phase.
Inductive : GNNs are used to classify unseen data (e.g.,new users), i.e., validation or test data are unseen during training.
Injection : Attackers can onlyinject new nodes but not modify the target nodes directly.
This reflects application scenarios like inonline social networks where it is usually hard to hack into users’ accounts and modify their profiles.
However, it is easier to create fake accounts and connect them to existing users.
For other attacksettings such as white-box attacks and modification attacks, we have included relevant experimentsand discussions in the supplementary material.
[Table]
We apply two state-of-the-art injection attack methods: SPEIT [14] and Topological Defective Graph
Injection Attack (TDGIA) [15].
These two attacks are the two strongest attacks reported in the GRB.
For both attacks, both the maximum allowable injected nodes and the maximum allowable injectededges are set to 50 for Cora and Citeseer, and 300 for PubMed.
We provide more details about howthese two attacks work under evasion, black-box and injection setting in the supplemental material.
Table 1 shows that graph neural PDE induced from Beltrami flow is more robust than all other GNNsexcept for GNNGuard, which is specifically designed to remove malicious edges and is thus robustagainst topology perturbation. This shows that the output features from a graph PDE are stable undertopology perturbation, as suggested by Proposition 1.
Furthermore, the heat diffusion process on a manifold tends to diffuse differently under differentgeometries.
For example, it diffuses slower at points with positive curvature, and faster at points with negative curvature [34, 51].
In [52] and [53], the authors show different datasets have differentgeometric properties like hyperbolicity distribution or Balanced Forman curvature. Our theoreticalanalysis only shows a loose uniform bound in terms of the adjacency matrix, while the performanceis very much dataset-dependent. More advanced theoretical analysis for different datasets is highlynon-trivial and needs further extensive investigations. Our graph neural PDE can be combined with GNNGuard.
We denote this model as BeltramiGuard. Table 2 shows that BeltramiGuard render attacks in vain and exceeds GNNGuard on the Citeseer and PubMed datasets.
Note from Table 1 that the vanilla Beltrami model already surpasses GNNGuardon the large PubMed.
More details are provided in the supplementary material.
5.2 Ablation StudiesDiffusion Schemes
We compare different diffusion equations: heat flow (19) where w ([ u, v ])
= 1 for all [ u, v ] ∈ E ,
GRAND/BLEND [29, 30] which is another heat flow where w ([ u, v ]) in (19) is the attention functiondefined in (16), mean curvature flow (18), and Beltrami flow (17).
For GRAND/BLEND, we stackthree neural PDE layers using the architecture proposed in Fig.
2 since the original GRAND/BLENDin [29, 30], which has only one PDE layer, does not perform well.
[Table]
From Table 3, we observe that even the vanilla time-invariant heat flow preserves some robustness ascompared to non-PDE GNNs in Table 1.
This further validates our theoretical analysis in Proposition 1,which suggests that if the topology perturbation is bounded, the learned representations are close tothose under the “clean” scenario.
We can observe that 1) the flows which are capable of preservingnon-smooth features are generally more robust than heat flow, 2) the proposed mean curvature flowand Beltrami flow are more robust than GRAND/BLEND, and 3) these two flows generally sufferless performance variance than the other methods. More Experiments
Due to space constraint, we refer the reader to the supplementary material for more ablation studies including the impact of the Lipschitz constraint, PDE solvers, the number of layers, and time complexity.
The model performance under other attacks is also presented in the supplementary material.
Conclusion
In this paper, we have introduced a general graph neural PDE framework from which several graphneural PDEs are proposed.
We analyzed the robustness of the graph neural PDEs and showed thatgraph neural PDEs are inherently robust against topology perturbations and are also Lyapunov stable.
We provided theoretical evidence showing that the robustness of graph neural PDEs stems from thestability of the heat kernel and semigroup during the diffusion process.
Moreover, we conductedextensive experiments that empirically verify the robustness of the proposed graph neural PDEs whencompared with an existing graph neural PDE induced from approximated heat flow, popular GNNsand the state-of-the-art GNN defenders.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](
b ) Did you describe the limitations of your work
[Yes
We have
left more complicatedproblems to future work.
(c) Did you discuss any
potential negative societal impacts
of your work?
[Yes] .
See the
Broader Impact
section in supplementary material.
(d) Have you read the ethics review guidelines and ensured that
your
paper
conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] .
See Section 2and Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] .
See supplementarymaterial.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] .
Seesupplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] .
See Section 5 and supplementary material.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)? [Yes] .
We run each experiment for 10 times.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] .
See Section 5.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5 for theopen-source datasets and models we have used from GitHub.
(b) Did you mention the license of the assets?
[No] .
Please see the licenses given in the
GitHub link or datasets websites.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] .
No new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] .
The datasets used in the paper are all open-source datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content?
[No] .
There is no identifiable information or offensive content in the datasets.
If you used crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]