O N THE S TABILITY OF G RAPH C ONVOLUTIONAL N EURAL N ETWORKS UNDER E DGE R EWIRING
A BSTRACT
Graph neural networks are experiencing a surge of popularity within the machine learning community due to their ability to adapt to non-Euclidean domains and instil inductive biases.
Despite this, their stability, i.e., their robustness to small perturbations in the input, is not yet well understood.
Although there exists some results showing the stability of graph neural networks, most take the form of an upper bound on the magnitude of change due to a perturbation in the graph topology.
However, the change in the graph topology captured in existing bounds tend not to be expressed in terms of structural properties, limiting our understanding of the model robustness properties.
In this work , we develop an interpretable upper bound elucidating that graph neural networks are stable to rewiring between high degree nodes.
This bound and further research in bounds of similar type provide further understanding of the stability properties of graph neural networks.
I NTRODUCTION
Recently, there has been an increasing amount of research on the use of machine learning models which operate on graph-structured data (Bronstein et al., 2017; Wu et al., 2020).
Graphs encode pairwise interactions and can help model non-Euclidean data such as social networks and molecules as well as impart inductive biases (Battaglia et al., 2018).
Graph signal processing (GSP), for example, generalises traditional signal processing to network domains allowing us to transfer many of the existing tools such as ﬁltering and sampling (Shuman et al., 2013; Dong et al., 2020).
GSP also allows us to generalise convolutions providing a framework to design graph convolutional neural networks (GCNNs) where the convolutional layers are ﬁlter banks of spectral graph ﬁlters (Defferrard et al., 2016; Levie et al., 2019; Bianchi et al., 2020).
To date, much of the research has focused on the predictive performance of these models with far fewer studies looking at their theoretical properties.
Like their Euclidean counterparts, GCNNs are susceptible to adversarial attacks (Z ¨ ugner et al., 2018;
Dai et al., 2018; Jin et al., 2020).
An adversarial attack is a small but targeted perturbation of the input which causes large changes in the output (Szegedy et al., 2013).
In the case of GCNNs, the modiﬁcation of a few edges in the input graph can change signiﬁcantly the learned node representations so that the prediction in the downstream task, typically node or graph classiﬁcation, switches from a correct to incorrect prediction.
One approach to understanding the vulnerability of these models to adversarial attacks is to consider their robustness against perturbation.
Robustness is an important property for reliable deployment of machine learning models in the real world, especially in domains where adversaries are common (e.g., on the web).
Existing literature has shown that large functional classes of spectral graph ﬁlters, a key component of GCNNs, are bounded by the magnitude of the change in the input graph.
For example Levie et al.
(2019) give an upper bound on the change in output for ﬁlters in the Cayley smoothness space which depends on the the distance of the GSO under the operator norm.
Later, Kenlay et al.
(2020) provided a similar bound for polynomial graph ﬁlters.
Gama et al.
(2020) provide a bound for graph neural networks whose convolutions are made up of ﬁlter banks of spectral graph ﬁlters.
The main drawback of these approaches is that the bounds involved in quantifying stability to speciﬁc changes in the topology do not have natural structural interpretations.
For example, if we delete just a single edge, it is unclear how loose the bound on the output change will be even if we know the statistical properties about the edge and endpoint nodes.
Our main goal is to extend bounds found in the existing literature so that they have an interpretation in terms of the structural properties of the existing and perturbed graphs.
To do this, we focus on polynomial graph ﬁlters, and build on the work of Kenlay et al.
(2020), by providing a new bound that is tighter and generalises to the family of normalised augmented adjacency matrices.
We then bound the change in normalised augmented adjacency matrix by considering the largest change around each node where the change admits a structural interpretation.
As a speciﬁc example, we consider perturbations that do not modify the degree distribution of the graph (i.e., double edge rewiring, illustrated in Fig. S1).
We then discuss under which scenarios, deleting and adding edges between nodes, guarantee the ﬁlter to be robust to perturbations.
To demonstrate how these theoretical results can be combined, we bound the change in the representations learned by two well-known
GCNN architectures.
Speciﬁcally, we consider simple graph convolutional networks (Wu et al., 2019) and multilayered graph convolutional network (Kipf & Welling, 2017) for scenarios where the graph topology is perturbed using edge rewiring.
P ROBLEM FORMULATION
We consider unweighted and undirected graphs G =
( V , E ) where V is a ﬁnite set of nodes and E is the edge set.
The adjacency matrix A encodes connections between nodes with A uv = 1 if there is an edge between nodes u and v and zero otherwise.
The degree d u of a node u is the number of nodes that u is connected to.
A graph signal is a function x : V → R that assigns a scalar value to each node.
By ﬁxing a labelling of the nodes we can represent this as a vector x ∈ R n where x i is the function value of node i and n = |V| is the number of nodes in the graph.
We will be concerned with the normalised augmented adjacency matrix ∆ γ
=
D − 1 / 2 γ
[Equation]
where A γ =
A + γ I and D γ = D + γ I with γ ≥ 0 and D = diag( d 1 , . . . d n ) .
When γ = 0 this matrix is the normalised adjacency matrix.
For simplicity, we drop the γ subscript when the context is clear or the speciﬁc value of γ is unimportant.
We can write ∆ in terms of its entries as
[Equation]
= v
[Equation]
if A uv = 1 and u (cid:54) [Equation] otherwise .
Normalised augmented adjacency matrices admit an eigendecomposition ∆ =
UΛU (cid:124) where the columns of U are the orthonormal eigenvectors, and Λ = diag( λ 1 , . . .
λ n ) where λ 1 ≤ , . .
. , ≤ λ n is the diagonal matrix of eigenvalues.
The graph Fourier transform of a signal x is given by ˆ x = U (cid:124) x , with the inverse graph Fourier transform deﬁned as x = U
ˆ x .
A spectral graph ﬁlter is deﬁned as a function g : R → R which ampliﬁes and attenuates speciﬁc frequencies in the graph Fourier domain.
We can ﬁlter a signal by applying the ﬁlter to the graph shift operator (in this case ∆ ) directly: U diag( g ( λ 1 ) , . . .
g ( λ n ))
U (cid:124) x
=
U g
( Λ ) U
(cid:124) x
= g ( ∆ ) x .
In this work (cid:107)·(cid:107) 2 represents the Euclidean norm when applied to vectors and the operator norm when applied to matrices.
We will also consider the Frobenius norm (cid:107) A (cid:107) 2 F = (cid:80) i,j A 2 i,j , the matrix one norm (cid:107) A (cid:107) 1 = max i (cid:80) j | A ij | and the matrix inﬁnity norm (cid:107) A (cid:107) ∞ =
max j (cid:80) i | A ij | .
The eigenvalues of the normalised augmented adjacency matrix lie in the interval [ − 1 , 1] .
The goals of this paper are twofold.
First, we want to understand the stability of spectral graph ﬁlters in terms of structural perturbation, by providing bounds that quantify the change in the output.
Second, we want to ﬁnd sufﬁcient conditions for this change to be small, by using interpretable structural properties of the graphs and the perturbation.
We address the ﬁrst goal in Sec. 3 and the second in Sec. 4.
Furthermore, we demonstrate how these theoretical contributions can be combined to give insight into the stability of GCNNs.
In Sec. 5, we combine the results of Sec. 3 and Sec. 4 to show how we can provide sufﬁcient conditions for the stability of two popular GCNN architectures.
S TABILITY OF POLYNOMIAL FILTERS
Our notion of stability is based on relative output distance which is bounded by the ﬁlter distance
[Equation]
Kenlay et al.
(2020) bounded the ﬁlter distance of polynomial ﬁlters by some constant times the error (cid:107) E (cid:107) 2 , where the constant depends on the ﬁlter and the error is the magnitude of the difference between normalised Laplacian matrices.
We say that by satisfying this condition the ﬁlters are stable.
Although the focus of this paper is not on tight bounds, we improve this bound by ﬁnding a smaller constant for polynomial ﬁlters.
We will write E = ∆ p − ∆ to be the difference of normalised augmented adjacency matrices between a graph G and a perturbed version of the graph G p .
The smaller constant is given by the following proposition.
Proposition 1.
Let ∆ and ∆ p be the normalised augmented adjacency matrix for G and G p .
Consider a polynomial graph ﬁlter g θ ( λ ) =
(cid:80) Kk =
θ k λ k .
Then
[Equation]
Proofs for all results are given in the supplementary.
We assume that the parameters of the model are ﬁxed before and after perturbation.
In the adversarial learning literature, an attack that modiﬁes the input to cause large changes in the output whilst the model parameters are ﬁxed is known as an evasion attack (Jin et al., 2020).
Robust models in the context of our work are those that are robust to evasion attacks with respect to the graph structure.
R OBUSTNESS TO EDGE REWIRING PERTURBATIONS
In this section we bound the error term (cid:107) E (cid:107) 2 by interpretable properties relating to the structural change.
Consider a graph G which we perturb to arrive at G p .
Our approach to upper bounding (cid:107) E (cid:107) 2 relies on the inequality (cid:107) E (cid:107) 22 ≤ (cid:107) E (cid:107) 1 (cid:107) E (cid:107) ∞ (Higham, 2002, Section 6.3).
As E is symmetric (cid:107) E (cid:107) 1 = (cid:107) E (cid:107) ∞ giving (cid:107) E (cid:107) 2 ≤ (cid:107) E (cid:107) 1 .
There may exist strategies which give tighter bounds, but the beneﬁt of this approach to bounding the error term is that (cid:107) E (cid:107) 1 leads to an interpretation in the structural domain.
For a matrix E we write E u as the u th column of E so E (cid:124) u is the u th row.
The row E (cid:124) u corresponds to the node u in the graph.
By deﬁnition (cid:107) E (cid:107) 1 = max u ∈V (cid:107) E (cid:124) u (cid:107) 1 where (cid:107) E (cid:124) u (cid:107) 1 = (cid:80) v | E uv | is the Manhattan norm of the row.
Perturbations which cause small changes to (cid:107) E (cid:124) u (cid:107) 1 over all nodes u guarantee small change in terms of (cid:107) E (cid:107) 2 .
We aim to understand what type of perturbations spectral graph ﬁlters may be robust to.
As a speciﬁc example consider perturbations that preserve the degree of the nodes.
For this scenario we can consider how the entries change from ∆ to ∆ p by considering Eq.
(1) to write a closed-form for (cid:107) E (cid:124) u (cid:107) 1 .
We will write A u to be the set of edges added around a node u and D u to be the set of edges deleted around a node u .
The diagonal entries of ∆ and ∆ p remain unchanged, and each edge deletion ﬂips the entry from zero to the second case of Eq.
(1), whilst edge addition ﬂips the entry the other way.
This insight lets us write (cid:107) E (cid:124) u (cid:107) 1 in closed-form as (cid:107)
[Equation]
(cid:124) u (cid:107) 1 = (cid:88) v ∈D u (cid:112)
[Equation]
(cid:88) v ∈A u (cid:112)
[Equation]
=
[Equation]
(cid:32) (cid:88)
[Equation]
u
[Equation]
(cid:88)
v
[Equation]
u 1 √
[Equation]
(cid:33) (2)
One such perturbation is edge rewiring that preserves degree distribution.
We deﬁne double edge rewiring as a function of two edges ( u, v ) and ( u (cid:48) , v (cid:48) )
such that u is not connected to u (cid:48) or v (cid:48) and similarly v is not connected to u (cid:48) and v (cid:48) .
The operation consists of deleting edges ( u, v ) and ( u (cid:48) , v (cid:48) ) and adding edges ( u, u (cid:48) ) and ( v, v (cid:48) ) .
This operation is depicted graphically in Fig.
S1.
Although the precise deﬁnition of rewiring in this work is slightly different, the idea of rewiring has been proposed as a strategy to make modiﬁcations imperceptible in the context of topological adversarial attacks (Ma et al., 2019).
Approximately preserving the degree distribution has also been a criterion used to deﬁne imperceptibility (Z¨ugner et al., 2018).
We will write R u to be the number of rewiring operations involving u and write δ u to be the smallest degree amongst either the nodes u disconnects with or is now connected to.
Each rewiring causes a single edge deletion and edge addition for each node involved so that the number of terms in each sum, the cardinality of sets D u and A u , is R u .
Using this we can bound Eq.
(2) to get that (cid:107)
[Equation]
The largest possible value for the right hand side of the above equation over all nodes u provides an upper bound for (cid:107) E (cid:107) 2 .
From this, we can draw some conclusions as to when the ﬁlter will be robust to rewiring operations.
The ﬁrst is that one should not rewire around one node signiﬁcantly as this will increase R u .
To keep (cid:107) E (cid:107) 1 small, and thus (cid:107) E (cid:107) 2 small, we must keep (cid:107) E (cid:124)
u (cid:107) 1 small for all nodes
u suggesting the perturbation should be distributed across the graph.
Related to this observation, in
Kenlay et al.
it was numerically demonstrated that the locality of perturbations can play a role in the magnitude of the error.
The second strategy to ensure robustness is to rewire between high degree nodes.
This will cause d u to be large and therefore (cid:107) E (cid:124) u (cid:107) 1 will be small.
Finally, using a normalised augmented adjacency matrix with larger values of γ can cause smaller changes.
S TABILITY OF GCNN M ODELS
We make use of results in previous sections to analyse the stability of two popular GCNN models.
We consider the multilayered graph convolutional network (GCN) (Kipf & Welling, 2017) in this section.
An analogous result for the simple convolutional networks (SGCN) (Wu et al., 2019) is given in the supplementary.
We consider the logits of a model consisting of multiple GCN layers with pointwise non-linearities giving the l th layer representation as
[Equation]
where ˜ ∆ def = ∆ 1 is the normalised augmented adjacency matrix with γ = 1 , σ is the ReLU activation function and Θ ( l ) are the layer parameters.
The input features are given by the matrix X (
) ∈ R n × d where d is the dimension of the features associated with each node.
We can consider X (0) as d stacked graph signals which we will call feature maps.
We will assume that each column (feature map) of X (0) has unit norm and that the number of feature maps are the same throughout the model.
We provide a bound on how much the logits can change under the Frobenius norm.
To motivate why we are interested in the Frobenius norm here, consider taking the Euclidean norm of each node output to measure the amount of change in node representation.
Taking the mean squared error of these distances amounts to taking the Frobenius norm of the logits matrix.
Proposition 2. Consider the outputs X
( L ) and X ( L ) p of an GCN with L layers for a graph G and a perturbed graph G p where the perturbed graph is a result of double edge rewiring.
Let ˜ ∆ and ˜ ∆ p be the corresponding normalised augmented adjacency matrices.
Deﬁne d u , δ u and R u as in Sec. 4, then the following holds
[Equation]
We suspect this bound, and the bound in Proposition 3, will likely be loose in practice; nevertheless, they provide conceptional insight into the factors that may be related to the robustness of the GCN and SGCN models.
In particular, we can reason when this bound will be small as in the ﬁnal paragraph of Sec. 4, which relates interpretable structural perturbation to the robustness of these models.
D ISCUSSION
In this work, we bound the change in output of spectral graph ﬁlters under a speciﬁc form of topological perturbation, i.e., edge rewiring, where the bound involves terms which have a structural interpretation.
We then demonstrate a practical application of this bound by applying it to the SGCN and GCN models.
Future directions include proving that the change in output of other models scales proportional to (cid:107) E (cid:107) 2 and providing further interpretable bounds to (cid:107) E (cid:107) 2 to provide an understanding of stability for graph-based models.
Furthermore, the SGCN and GCN models we consider both apply low-pass ﬁlters, and exploring models with other ﬁltering characteristics is an interesting future direction.
Finally, extensions of the framework presented in this work such as considering a more general perturbation model will beneﬁt its utility for practical applications.
A S TABILITY OF SGCN
The SGCN model is motivated by considering a multilayered GCN with the activation functions removed.
By removing the activation functions the model boils down to a ﬁxed monomial ﬁlter of order K followed by applying a fully connected layer and a softmax layer to the node features.
Let the input data be given by the matrix X ∈ R n × d where d is the dimension of the features associated with each node.
We can consider X as d stacked graph signals which we will call feature maps.
We will assume that each column (feature map) of X has unit norm.
The output is a matrix
Y ∈ R n × c , representing class probabilities for each node.
The SGCN model is deﬁned as
[Equation]
where softmax( x ) i = exp( x i ) / (cid:80) i exp( x i ) normalises the rows to be probability distributions.
We write ˜ ∆
def = ∆ 1 as the normalised augmented adjacency matrix with γ = 1 .
We will analyse how the logits, the node representations before the softmax is applied, change.
The softmax function has a Lipschitz constant of 1 so any bound on the logits can trivially be applied to the model outputs (Gao & Pavel, 2017)[Proposition 4].
We provide a bound on how much the logits can change under the Frobenius norm.
Proposition 3.
The distance of the logits is bounded like so
[Equation]
B P ROOFS
We state the following Lemma which we will use to prove Proposition 1.
Lemma 1 (Lemma 3, Levie et al.
Suppose B , D , E ∈
C
N
×
N are Hermitian matrices satisfying B
= D + E , and (cid:107) B (cid:107) 2 , (cid:107) D (cid:107) 2 ≤ C for some C > 0 .
Then for every l ≥(cid:13)(cid:13) B l − D l (cid:13)(cid:13)
[Equation]
l
[Equation]
Proposition 1.
Let ∆ and ∆ p be the normalised augmented adjacency matrix for G and G p .
Consider a polynomial graph ﬁlter g θ ( λ ) =
(cid:80) Kk =
θ k λ k .
Then
[Equation]
Proof.
Using the triangle inequality followed by an application of 1 with constant C
=
we get
[Equation]
The following lemma is used in the proofs for Proposition 2 and Proposition 3.
Lemma 2.
Let B ∈ R m × r and C ∈ R r × n then (cid:107) BC (cid:107) F ≤ (cid:107) B (cid:107) F (cid:107) C (cid:107) 2 .
Proof.
By decomposing BC in terms of the rows b (cid:124) k of B we get
[Equation]
The inequality follows from the observation that
[Equation]
Taking the square root of both sides of Eq.
(4) gives the result.
We will also use the following bound on (cid:107) E (cid:107) 2 which is derived in Section
[Equation]
Proposition 2. Consider the outputs X
( L ) and X ( L ) p of an GCN with L layers for a graph G and a perturbed graph G p where the perturbed graph is a result of double edge rewiring.
Let ˜ ∆ and ˜ ∆ p be the corresponding normalised augmented adjacency matrices.
Deﬁne d u , δ u and R u as in Sec. 4, then the following holds
[Equation]
Proof.
We prove this by induction.
The base case L = 1 follows immediately from Proposition
[Equation]
Consider a more general L .
Then
[Equation]
where the ﬁrst inequality comes from ReLU having unit Lipschitz constant and the second being an application of Lemma 2.
By using triangle inequality we bound the following term
[Equation]
Note that
[Equation]
so by recursivity we get that
[Equation]
Recall that (cid:107) X (0) (cid:107) F = √ d .
Using this observation and the inductive assumption we get that
[Equation]
X
[Equation]
[Table]
[Equation]
[Table]
[Equation]
X
[Equation]
[Table]
[Equation]
p (cid:13)(cid:13)(cid:13)
[Equation]
[Table]
[Equation]
[Table]
l ) (cid:13)(cid:13)(cid:13)
[Equation]
The ﬁnal result follows from an application of Equation 5.
Proposition 3.
The distance of the logits is bounded like so
[Equation]
Proof.
We ﬁrst note that (cid:107) X (cid:107) F = √ d .
Using this, two applications of Lemma 2, and an application of Proposition 1 we get
[Equation]
The ﬁnal result follows from an application of Equation
[Equation]