P OWER UP ! R OBUST G RAPH C ONVOLUTIONAL N ETWORK BASED ON G RAPH P OWERING
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph convolutional networks (GCNs) are powerful tools for graph-structured data.
However, they have been recently shown to be vulnerable to topological attacks.
To enhance adversarial robustness, we go beyond spectral graph theory to robust graph theory.
By challenging the classical graph Laplacian we propose a new convolution operator that is provably robust in the spectral domain and is incorporated in the GCN architecture to improve expressivity and interpretability.
By extending the original graph to a sequence of graphs, we also propose a robust training paradigm that encourages transferability across graphs that span a range of spatial and spectral characteristics.
The proposed approaches are demonstrated in extensive experiments to simultaneously improve performance in both benign and adversarial situations.
I NTRODUCTION
Graph convolutional networks (GCNs) are powerful extensions of convolutional neural networks (CNN) to graph-structured data.
Recently, GCNs and variants have been applied to a wide range of domains, achieving state-of-the-art performances in social networks (Kipf & Welling, 2017), trafﬁc prediction (Rahimi et al., 2018), recommendation systems (Ying et al., 2018), applied chemistry and biology (Kearnes et al., 2016; Fout et al., 2017), and natural language processing (Atwood &
Towsley, 2016; Hamilton et al., 2017; Bastings et al., 2017; Marcheggiani & Titov, 2017), just to name a few (Zhou et al., 2018; Wu et al., 2019).
GCNs belong to a family of spectral methods that deal with spectral representations of graphs (Zhou et al., 2018; Wu et al., 2019).
A fundamental ingredient of GCNs is the graph convolution operation deﬁned by the graph Laplacian in the Fourier domain:
[Equation]
(1) where x ∈ R n is the graph signal on the set of vertices V
and ˆ g θ is a spectral function applied to the graph Laplacian L := D − A (where D and A are the degree matrix and the adjacency matrix, respectively).
Because this operation is computational intensive for large graphs and non-spatially localized (Bruna et al., 2014), early attempts relied on a parameterization with smooth coefﬁcients (Henaff et al., 2015) or a truncated expansion in terms of of Chebyshev polynomials (Hammond et al., 2011).
By further restricting the Chebyshev polynomial order by 2, the approach in (Kipf &
Welling, 2017) referred henceforth as the vanilla GCN pushed the state-of-the-art performance of semi-supervised learning.
The network has the following layer-wise update rule:
[Equation]
where H ( l ) is the l -th layer hidden state (with H (1) :
= X as nodal features), W ( l ) is the l -th layer weight matrix, ψ is the usual point-wise activation function, and A is the convolution operator chosen to be the degree weighted Laplacian with some slight modiﬁcations (Kipf & Welling, 2017).
Subsequent GCN variants have different architectures, but they all share the use of the Laplacian matrix as the convolution operator (Zhou et al., 2018; Wu et al., 2019).
1. W HY NOT GRAPH L APLACIAN ?
Undoubtedly, the Laplacian operator (and its variants, e.g., normalized/powered Laplacian) plays a central role in spectral theory, and is a natural choice for a variety of spectral algorithms such as principal component analysis, clustering and linear embeddings (Chung & Graham, 1997; Belkin &
Niyogi, 2002).
So what can be problematic?
From a spatial perspective, GCNs with d layers cannot acquire nodal information beyond its ddistance neighbors; hence, it severely limits its scope of data fusion. Recent works (Lee et al., 2018; Abu-El-Haija et al., 2018; 2019; Wu et al., 2019) alleviated this issue by directly powering the graph
Laplacian. From a spectral perspective, one could demand better spectral properties , given that GCN is fundamentally a particular (yet effective) approximation of the spectral convolution (1).
A key desirable property for generic spectral methods is known as “spectral separation,” namely the spectrum should comprise a few dominant eigenvalues whose associated eigenvectors reveal the sought structure in the graph.
A well-known prototype is the Ramanujan property, for which the second leading eigenvalue of a r -regular graph is no larger than 2 √ r − 1 , which is also enjoyed asymptotically by random r -regular graphs (Friedman, 2004) and Erd˝os-Rényi graphs that are not too sparse (Feige & Ofek, 2005).
In a more realistic scenario, consider the stochastic block model (SBM), which attempts to capture the essence of many networks, including social graphs, citation graphs, and even brain networks (Holland et al., 1983).
Deﬁnition 1 (Simpliﬁed stochastic block model) .
The graph G with n nodes is drawn under
SBM ( n, k, a intra , a inter ) if the nodes are evenly and randomly partitioned into k communities, and nodes i and j are connected with probability a intra /n ∈ [0 , 1] if they belong to the same community, and a inter /n ∈ [0 , 1] if they are from different communities.
It turns out that for community detection, the top k leading eigenvectors of the adjacency matrix
A play an important role.
In particular, for the case of 2 communities, spectral bisection algorithms simply take the second eigenvector to reveal the community structure.
This can be also seen from the expected adjacency matrix E
[ A ] under SBM ( n, 2 , a intra , a inter ) , which is a rank-2 matrix with the top eigenvalue 12 ( a intra + a inter ) and eigenvector 1 , and the second eigenvalue 12 ( a intra − a inter ) and eigenvector σ such that σ i = 1 if i is in community 1 and σ i = − 1 otherwise.
More generally, the second eigenvalue is of particular theoretical interests because it controls at the ﬁrst order how fast heat diffuses through graph, as depicted by the discrete Cheeger inequality (Lee et al., 2014).
While one would expect taking the second eigenvector of the adjacency matrix sufﬁces, it often fails in practice (even when it is theoretically possible to recover the clusters given the signal-to-noise ratio).
This is especially true for sparse networks, whose average nodal degrees is a constant that does not grow with the network size.
This is because the spectrum of the Laplacian or adjacency matrix is blurred by “outlier” eigenvalues in the sparse regime, which is often caused by high degree nodes (Kaufmann et al., 2016).
Unsurprisingly, powering the Laplacian would be of no avail, because it does not change the eigenvectors or the ordering of eigenvalues.
In fact, those outliers can become more salient after powering, thereby weakening the useful spectral signal even further.
Besides, pruning the largest degree nodes in the adjacency matrix or normalizing the Laplacian cannot solve the issue.
To date, the best results for pruning does not apply down to the theoretical recovery threshold (Coja-Oghlan, 2010; Mossel et al., 2012; Le et al., 2015); either outliers would persist or one could prune too much that the graph is destroyed.
As for normalized Laplacian, it may overcorrect the large degree nodes, such that the leading eigenvectors would catch the “tails” of the graph, i.e., components weakly connected to the main graph.
See Figure A.3 in the appendix for an experimental illustration.
In summary, graph Laplacian may not be the ideal choice due to its limited spatial scope of information fusion, and its undesirable artefacts in the spectral domain.
1. I F NOT L APLACIAN , THEN WHAT ?
In searching for alternatives, potential choices are many, so it is necessary to clarify the goals.
In view of the aforementioned pitfalls of the graph Laplacian, one would naturally ask the question:
Can we ﬁnd an operator that has wider spatial scope, more robust spectral properties, and is meanwhile interpretable and can increase the expressive power of GCNs?
From a perspective of graph data analytics , this question gauges how information is propagated and fused on a graph, and how we should interpret “adjacency” in a much broader sense.
An image can be viewed as a regular grid, yet the operation of a CNN ﬁlter goes beyond the nearest pixel to a local neighborhood to extract useful features.
How to draw an analogy to graphs?
From a perspective of robust learning , this question sheds light on the basic observation that realworld graphs are often noisy and even adversarial.
The nice spectral properties of a graph topology can be lost with the presence or absence of edges.
What are some principled ways to robustify the convolution operator and graph embeddings?
In this paper, we propose a graph learning paradigm that aims at achieving this goal, as illustrated in
Figure 1.
The key idea is to generate a sequence of graphs from the given graph that capture a wide range of spectral and spatial behaviors.
We propose a new operator based on this derived sequence.
Deﬁnition 2 (Variable power operator) .
Consider an unweighted and undirected graph G .
Let A [ k ] denote the k -distance adjacency matrix, i.e., (cid:2)
A [ k ] (cid:3) ij = 1 if and only if the shortest distance (in the original graph) between nodes i and j is k .
The variable power operator of order r is deﬁned as:
[Equation]
where θ : = ( θ 0 , . .
. , θ r ) is a set of parameters.
Clearly, A ( r ) θ is a natural extension of the classical adjacency matrix (i.e., r = 1 and θ
= θ 1 = 1 ).
With power order r > 1 , one can increase the spatial scope of information fusion on the graph when applying the convolution operation.
The parameters θ k also has a natural explanation—the magnitude and the sign of θ k can be viewed as “global inﬂuence strength” and “global inﬂuence propensity” at distance k , respectively, which also determines the participation factor of each graph in the sequence in the aggregated operator.
Furthermore, we provide some theoretical justiﬁcation of the proposed operator by establishing the following asymptotic property of spectral separation under the important SBM setting, which is, nevertheless, not enjoyed by the classical Laplacian operator or its normalized or powered versions.
(All proofs are given in the appendix.)
Theorem 3 (Asymptotic spectral separation of variable power operator) .
Consider a graph G drawn from SBM ( n, 2 , a intra , a inter ) .
Assume that the signal-to-noise ratio ξ 22 /ξ 1 > 1 , where ξ 1 = 12 ( a intra + a inter ) and ξ 2 = 12 ( a intra − a inter ) (c.f., (Decelle et al., 2011)).
Suppose r is on the order of c log( n ) for a constant c , such that c log( ξ 1 ) < 1 / 4 .
Given nonvanishing θ k for k > r/ 2 , the variable power operator A ( r ) θ has the following spectral properties: (i) the leading eigenvalue is on the order of Θ ( (cid:107) θ (cid:107) 1 ξ r 1 ) , the second leading eigenvalue is on the order of Θ ( (cid:107) θ (cid:107)
ξ r 2 ) , and the rest are bounded by (cid:107) θ (cid:107) 1 n (cid:15)
ξ r/ 2 1 O (log( n )) for any ﬁxed (cid:15) > 0 ; and (ii) the two leading eigenvectors are sufﬁcient to recover the two communities asymptotically (i.e., as n goes to inﬁnity).
Intuitively, the above theoretical result suggests that the variable power operator is able to “magnify” benign signals from the latent graph structure while “suppressing” noises due to random artefacts.
[Figure]
This is expected to improve spectral methods in general, especially when the benign signals tend to be overwhelmed by noises.
For the rest of the paper, we will apply this insight to propose a robust graph learning paradigm in Section 2, as well as a new GCN architecture in Section 3.
We also provide empirical evidence of the gain from this theory in Section 4 and conclude in Section 5.
1. R ELATED WORK
Beyond nearest neighbors.
Several works have been proposed to address the issue of limited spatial scope by powering the adjacency matrix (Lee et al., 2018; Wu et al., 2019; Li et al., 2019).
However, simply powering the adjacency does not extract spectral gap and may even make the eigenspectrum more sensitive to perturbations.
Abu-El-Haija et al.
(2018; 2019) also introduced weight matrices for neighbors at different distances.
But this could substantially increase the risk of overﬁtting in the low-data regime and make the network vulnerable to adversarial attacks.
Robust spectral theory.
The robustness of spectral methods has been extensively studied for graph partitioning/clustering (Li et al., 2007; Balakrishnan et al., 2011; Chaudhuri et al.;
Amini et al., 2013;
Joseph et al., 2016; Diakonikolas et al., 2019).
Most recently, operators based on self-avoiding or nonbacktracking walks have become popular for SBM (Massoulié, 2014; Mossel et al., 2013;
Bordenave et al., 2015), which provably achieve the detection threshold conjectured by Decelle et al.
Our work is partly motivated by the graph powering approach by Abbe et al.
(2018), which leveraged the result of Massoulié (2014); Bordenave et al.
(2015) to prove the spectral gap.
The main difference with this line of work is that these operators are studied only for spectral clustering without incorporating nodal features.
Our proposed variable power operator can be viewed as a kernel version of the graph powering operator (Abbe et al., 2018), thus substantially increasing the capability of learning complex nodal interactions while maintaining the spectral property.
Robust graph neural network.
While there is a surge of adversarial attacks on graph neural networks (GNNs) (Dai et al., 2018; Zügner & Günnemann, 2019; Bojchevski & Günnemann, 2019), very few methods have been proposed for defense (Sun et al., 2018).
Existing works employed known techniques from computer vision (Szegedy et al., 2014; Goodfellow et al., 2015; Szegedy et al., 2016), such as adversarial training with “soft labels” (Chen et al., 2019) or outlier detection in the hidden layers (Zhu et al., 2019), but they do not exploit the unique characteristics of graphstructured data.
Importantly, our approach simultaneously improves performance in both the benign and adversarial tests, as shown in Figure 2 (details are presented in Section 4).
[Figure]
G RAPH AUGMENTATION : R OBUST TRAINING PARADIGM
Exploration of the spectrum of spectral and spatial behaviors.
Given a graph G , consider a family of its powered graphs, (cid:8) G (2) , . .
, G ( r ) (cid:9) , where G ( k ) is obtained by connecting nodes with distance less than or equal to k .
This “graph augmentation” procedure is similar to “data augmentation”, because instead of limiting the learning on a single graph that is given, we artiﬁcially create a series of graphs that are closely related to each other in the spatial and spectral domains.
As we increase the power order, the graph also becomes more homogenized (Figure A.1).
In particular, it can help near-isolated nodes (i.e., low-degree vertices), since they become connected beyond their nearest neighbors.
By comparison, simply raising the adjacency matrix to its r -th power will make them appear even more isolated, because it inadvertently promotes nodes with high degrees or nearby loops much more substantially as a result of feedback.
Furthermore, the powered graphs can extract spectral gaps in the original graph despite local irregularities, thus boosting the signalto-noise ratio in the spectral domain.
Transfer of knowledge from the powered graph sequence.
Consider a generic learning task on a graph G with data D .
The loss function is denoted by (cid:96) ( W ; G , D ) for a particular GCN architecture parametrized by W .
For instance, in semi-supervised learning, D consists of features and labels on a small set of nodes (see Table A.1), and (cid:96) is the cross-entropy loss over all labeled examples.
Instead of minimizing over (cid:96) ( W ; G , D ) alone, we use all the powered graphs:
[Equation]
where α k ≥ 0 gauges how much information one desires to transfer from powered graph G ( k ) to the learning process.
By minimizing the (r-GCN) objective, one seeks to optimize the network parameter W on multiple graphs simultaneously, which is beneﬁcial in two ways: (i) in the low-data regime , like semi-supervised learning, it helps to reduce the variance to improve generalization and transferability; (ii) in the adversarial setting , it robustiﬁes the network since it is more likely that the perturbed network is contained in the wider spectrum during training.
F ROM FIXED TO VARIABLE POWER NETWORK
By using the variable power operator illustrated in Figure 1, we substantially increase the search space of graph operators.
The proposed operator also leads to a new family of graph algorithms with broader scope of spatial fusion and enhanced spectral robustness.
As the power grows, the network eventually becomes dense.
To manage this increased complexity and make the network more robust against adversarial attacks in the feature domain, we propose a pruning mechanism.
Graph sparsiﬁcation.
Given a graph G :=
( V , E [1] ) , consider its powered version G ( r ) :
= ( V , E ( r ) ) and a sequence of intermediate graphs G [2] , . . .
, G [ r ] , where G [ k ] :
= ( V , E [ k ] ) is constructed by connecting two vertices if and only if the shortest distance is k in G .
Clearly, {E [ k ] } rk =1 forms a partition of E ( r ) .
For each node i ∈ V , denote its r -neighborhood by N r ( i ) :
= { j ∈ V | d G ( i, j ) ≤ r } , which is identical to the set of nodes adjacent to i in G [ r ] .
Next, for each edge within this neighborhood, we associate a value using some suitable distance metric φ to measure “aloofness.”
For instance, it can be the usual Euclidean distance or correlation distance in the feature space.
Based on this formulation, we prune an edge e :=
( i, j ) in E ( r ) if the value is larger than a threshold τ ( i, j ) , and denote the edge set after pruning ¯ E ( r ) .
Then, we can construct a new sequence of sparsiﬁed graphs, G [ k ] with edge sets ¯ E [ k ] =
E [ k ] ∩ ¯ E ( r ) and adjacency matrix
¯ A [ k ] .
Hence, the variable power operator is given by ¯ A ( r ) θ = (cid:80) rk =
θ k ¯
A [ k ] .
Due to the inﬂuence of high-degree nodes in the spectral domain, one can adaptively choose the thresholds τ
( i, j ) to mitigate their effects.
Speciﬁcally, we choose τ to be a small number if either i or j are nodes with high degrees, thereby making the sparsiﬁcation more inﬂuential in densely connected neighborhoods than weakly connected parts.
Layer-wise update rule.
To demonstrate the effectiveness of the proposed operator, we adopt the vanilla GCN strategy (2).
Importantly, we replace the graph convolutional operator A with the variable power operator to obtain the variable power network (VPN):
[Equation]
where D ii = 1 + |{ j ∈ V | d G ( i, j ) = 1 }| .
The introduction of I is reminiscent of the “renormalization trick” (Kipf & Welling, 2017), but it can be also viewed as a regularization strategy in this context, which is well-known to improve the spectral robustness (Amini et al., 2013; Joseph et al., 2016).
This construction immediately increases the scope of data fusion by a factor of r .
Proposition 4.
By choosing A with (VPN) in the layer-wise update rule (2), the output at each node from a L -layer GCN depends on neighbors within L ∗ r hops.
Since we proved that the variable power operator has nice spectral separation in Theorem 3, VPN is expected to promote useful spectral signals from the graph topology (similar to the preservation of useful information in images (Jacobsen et al., 2018), our method preserves useful information in the graph topology).
This claim is substantiated with the following proposition.
Proposition 5.
Given a graph with two hidden communities.
Consider a 2-layer GCN architecture with layer-wise update rule (2).
Suppose that A has a spectral gap.
Further, assume that the leading two eigenvectors are asymptotically aligned with 1 and ν , i.e., the community membership vector, and that both are in the range of feature matrix X .
Then, there exists a conﬁguration of W (1) and
W (2) such that the GCN outputs can recover the community with high probability.
E XPERIMENTS
The proposed methods are evaluated against several recent models, including vanilla GCN (Kipf &
Welling, 2017) and its variant PowerLaplacian where we simply replace the adjacent matrix with its powered version, three baselines using powered Laplacian IGCN (Li et al., 2019), SGC (Wu et al., 2019) and LNet (Liao et al., 2019), the recent method MixHop (Abu-El-Haija et al., 2019) which attempts to increase spatial scope fusion, as well as a state-of-the-art baseline and RGCN (Zhu et al., 2019), which is also aimed at improving the robustness of Vanilla GCN.
All baseline methods on based on their public codes.
4. R EVISITING THE STOCHASTIC BLOCK MODEL
SBM dataset.
We generated a set of networks under SBM with 4000 nodes and parameters such that the SNRs range from 0.58 to 0.91.
To disentangle the effects from nodal features with that from the spectral signal, we set the nodal features to be one-hot vectors.
The label rates are 0.1%, 0.5% and 1%, the validation rate is 1%, and the rest of the nodes are testing points.
Performance.
Since the nodal features do not contain any useful information, learning without topology such as multi-layer perceptron (MLP) is only as good as random guessing.
The incorporation of graph topology improves classiﬁcation performance—the higher the SNR (i.e., ξ 22 /ξ 1 , see Theorem 3), the higher the accuracy.
Overall, as shown in Figure 3, the performance of the proposed method (VPN) is superior than other baselines, which either use Laplacian (GCN, Chebyshev,
RGCN, LNet) or its powered variants (IGCN, PowerLaplacian).
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
[Figure]
● [Figure] LanczosNet
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
●
[Figure]
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
[Figure]
● ●
[Figure]
Spectral separation and Fourier modes.
From the eigenspectrum of the convolution operators (Figure A.3), we see that the spectral separation property is uniquely possessed by VPN, whose ﬁrst two leading eigenvectors carry useful information about the underlying communities: without the help of nodal features, the accuracy is 87% even with label rate of 0.5%.
Let Φ denote the Fourier
[Figure]
[Table] modes of the the adjacency matrix or VPN, and X be the nodal features (i.e., identity matrix).
We analyze the information from spectral signals (e.g., the k -th and k +1 -th eigenvectors) by estimating the accuracy of an MLP with ﬁltered nodal features, namely Φ : ,k :( k +1) Φ (cid:62) : ,k :( k +1)
X , as shown in
Figure 4.
The accuracy reﬂects the information content in the frequency components.
We see that the two leading eigenvectors of VPN are sufﬁcient to perform classiﬁcation, whereas those of the adjacency matrix cannot make accurate inferences.
4. S EMI - SUPERVISED NODE CLASSIFICATION
Experimental setup.
We followed the setup of (Yang et al., 2016; Kipf & Welling, 2017) for citation networks Citeseer, Cora and Pubmed (please refer to the Appendix for more details).
[Figure]
Graph powering order can inﬂuence spatial and spectral behaviors.
Our theory suggests powering to the order of log( n ) ; in practice, orders of 2 to 4 sufﬁce (Figure 5).
Here, we chose the power order to be 4 for r-GCN on
Citeseer and Cora, and 3 for Pubmed, and reduced the order by 1 for VPN.
Performance.
By replacing Laplacian with VPN, we see an immediate improvement in performance (Table 1).
We also see that a succinct parametrization of the global inﬂuence relation in VPN is able to increase the expressivity of the network.
For instance, the learned θ at distances 2 and 3 for Citeseer are 3.15e-3 and 3.11e-3 with p -value less than 1e-5.
This implies that the network tends to put more weights in closer neighbors.
4. D EFENSE AGAINST EVASION ATTACKS
To evaluate the robustness of the learned network, we considered the setting of evasion attacks, where the model is trained on benign data but tested on adversarial data.
Adversarial methods.
Five strong global attack methods are considered, including DICE (Zügner & Günnemann, 2019), A abr and A DW 3 (Bojchevski & Günnemann, 2019), Meta-Train and MetaSelf (Zügner & Günnemann, 2019).
We further modulated the severity of attack methods by varying the attack rate, which corresponds to the percentage of edges to be attacked.
Robustness evaluation.
In general, both r-GCN and VPN are able to improve over baselines for the defense against evasion attacks, e.g., Figure 6 for the A DW 3 attack (detailed results for other attacks are listed in the Appendix).
It can be also observed that the proposed methods are more robust in
Citeseer and Cora than Pubmed.
In addition to the low label rates, we conjectured that topological
[Figure] [Figure] [Figure] attacks are more difﬁcult to defend for networks with prevalent high-degree nodes, because the attacker can bring in more irrelevant vertices by simply adding a link to the high-degree nodes.
Informative and robust low-frequency spectral signal.
It has been observed by Wu et al.
Maehara (2019) that GCNs share similar characteristics of low-pass ﬁlters, in the sense that nodal features ﬁltered by low-frequency Fourier modes lead to accurate classiﬁers (e.g., MLP).
However, one key question left unanswered is how to obtain the Fourier modes.
In their experiments, they derive it from the graph Laplacian.
By using VPN to construct the Fourier modes, we show that the information content in the low-frequency domain can be improved.
More speciﬁcally, we ﬁrst perform eigendecomposition of the graph convolutional operator (i.e., graph Laplacian or VPN) to obtain the Fourier modes Φ . We then reconstruct the nodal features X using only the k -th and the k + 1 -th eigenvectors, i.e., Φ : ,k :( k +1) Φ (cid:62) : ,k :( k +1)
X . We then use the reconstructed features in MLP to perform the classiﬁcation task in a supervised learning setting. As
Figure 7 shows, features ﬁltered by the leading eigenvectors of VPN lead to higher classiﬁcation accuracy compared to the classical adjacency matrix.
For the adversarial testing, we construct a new basis (cid:101) Φ based on the attacked graph, and then use
[Equation] : ,k :( k +1) X as test points for the MLP trained in the clean data setting. As can be seen in Figure 8, models trained based on VPN ﬁltered features also have better adversarial robustness in evasion attacks.
Since the eigenvalues of the corresponding operator exhibit low-pass ﬁltering characteristics (Figure A.2), the enhanced benign and adversarial accuracy of VPN is attributed to the increased signal-to-noise ratio in the low-frequency domain.
This is in alignment with the theoretical proof of spectral gap developed in this study.
[Figure]
4. D ISCUSSIONS
In this paper, we proposed a new graph operator to replace the graph Laplacian in the classic GCN.
It is worth mentioning that our approach is very different from the k -th order polynomials of the Laplacian developed in (Defferrard et al., 2016) or the high-order graph Laplacian employed in (Lee et al., 2018; Li et al., 2019; Abu-El-Haija et al., 2019; Wu et al., 2019).
If the graph Laplacian L is powered to the k -th order, the resulting matrix has the same eigenvectors as L , with only the eigenvalues
[Figure]
powered to the k -th order.
Thus, the corresponding k -th order polynomial has the same eigenvectors, with the polynomial function applied only to the eigenvalues.
However, the proposed variable power operator has radically different eigenvectors as the graph Laplacian or its powered variant.
This is an important difference because we observed empirically that the leading eigenvectors of a graph Laplacian are extremely sensitive to outliers under the SBM, and they often correspond to either tails or high-degree nodes (please see Figure A.3 for an illustration).
However, our result in
Theorem 3 suggests that the leading eigenvectors of the proposed operator asymptotically recovers the underlying community under SBM, and enjoys the “spectral gap” property
We also remark that the meaning of robustness in the machine learning literature (“adversarial robustness”) and in the spectral theory literature (“spectral robustness”) appear to be different.
However, in this work, we argue that spectral robustness is closely related to adversarial robustness.
This can be explained from a matrix perturbation point of view.
Let λ k and φ k be the eigenvalue and the corresponding eigenvector of the adjacency matrix A .
Then, the sensitivity of the eigenvector to the perturbation is given by (Trefethen & Bau III, 1997):
[Equation]
where A ( ij ) is the ( i, j ) -th entry of A , φ k ( j ) is the j -th entry of φ k , δ ( ij ) is the perturbation of the ( i, j ) -th entry of A , and n is the dimension of A .
It can be seen that the perturbation of the convolutional operator is controlled by the inverse of the spectral gap.
When the spectral gaps between the leading eigenvalues (i.e., λ 1 and λ 2 ) and the rest of the eigenvalues are large, the perturbations of the leading eigenvectors due to the adversarial attack are controlled.
Since the eigenvalues of the graph convolutional operator decays very quickly (see Figure A.2 as an illustration), the learned representation is heavily inﬂuenced by the leading eigenvectors.
Thus, by controlling the perturbations of the leading eigenvectors, one can expect to control the perturbations of the outputs.
Our theoretical result in Theorem 3 was based on the standard SBM, which is a classic model proposed in mathematical sociology and has been adopted to model and analyze real-world social and biological networks (Funke & Becker, 2019).
The main advantage of SBM is that it encodes the structural information that nodes belong to the same community tend to be more connected with each other.
Nevertheless, we recognize one limitation of our analysis is that SBM might not be a suitable model for some existing or future applications, in which case, models with more ﬂexibility in the degree distributions, such as the degree-corrected SBM proposed by Karrer & Newman (2011), labeled SBM by Heimlicher et al.
(2012), or hiearchical SBM proposed by Peixoto (2017) could be more applicable.
Also our analysis is limited to two communities, and the extension to more than two communities is a challenging open problem.
Nevertheless, through extensive experiments, we observed that our theory has strong implications for real-world graphs.
This is evident from the informative and robust low-frequency spectral signal results in Figures 7 and 8, which show that the proposed graph convolution operator has a clear advantage in the low-frequency regime over the classic graph Laplacian in terms of classiﬁcation accuracy.
C ONCLUSION
This study goes beyond classical spectral graph theory to defend GCNs against adversarial attacks.
We challenge the central building block of existing methods, namely the graph Laplacian, which is not robust to noisy links.
For adversarial robustness, spectral separation is a desirable property.
We propose a new operator that enjoys this property and can be incorporated in GCNs to improve expressivity and interpretability.
Furthermore, by generating a sequence of powered graphs based on the original graph, we can explore a spectrum of spectral and spatial behaviors and encourage transferability across graphs.
The proposed methods are shown to improve both benign and adversarial accuracy over various baselines evaluated against a comprehensive set of attack strategies
A A PPENDIX A. P ROOF OF P ROPOSITION
To validate the claim, we prove a more general result.
Consider a graph G and its powered graph
G ( r ) , whose adjacency matrices are A and A ( r ) , respectively.
Since the support of A ( r ) θ is identical to that of A ( r ) , we will focus on A ( r ) for notational simplicity.
For any signal f : V (cid:55)→ R |V| on a graph, denote the support of the signal f v on node v ∈ V by S fv , i.e., f v only depends on the signals on nodes in S fv .
Then, it is easy to see that the signal g (cid:44)
Af has support S gv ⊆ ∪ u ∼ r v S fu , where u ∼ r v denotes adjacency relation on G ( r ) deﬁned by A ( r ) .
Clearly, the set deﬁned by { u ∈ V | u ∼ r v } is identical to { u ∈ V | d G ( u, v ) ≤ r } , where d G ( u, v ) denotes the shortest distance from u to v on G .
Also, we see that element-wise operation does not expand the support, i.e., for any element-wise activation σ , the signal g σ (cid:44) σ ( f ) has support S g σ v ⊆ S fv .
In particular, we can prove the claim by iteratively applying the function composition layer by layer, since each layer is of the form σ ( A ( r ) θ H ( l ) W ( l ) ) , where H ( l ) is the hidden state at layer l and W ( l ) is the corresponding weight matrix.
Hence after L iterations of layer-wise updates, the support of the output on v is given by { [Equation]
A . P ROOF OF P ROPOSITION
Let λ
and λ 2 be the two leading eigenvalues of ¯ A with corresponding eigenvectors φ 1 and φ 2 , respectively.
Without loss of generality, assume that both eigenvalues are nonnegative.
Under the assumption that φ 1 and φ 2 lie in the range of X , there exists a W (1) such that XW (1) =
[ φφ 2 ] .
So the output of the ﬁrst layer with ReLU activation is given by:
[Equation]
where (cid:107) h 1 (cid:107) =
o (1) and (cid:107) h 2 (cid:107) =
o (1) (6) where (5) is due to the deﬁnition of eigenvectors and that σ is an element-wise operation, and (6) is due to the asymptotic alignment condition and the fact that σ ( ν ) = 12 ν + 12 1 .
Therefore, by choosing the last layer weight W (2) =
[Equation]
, the output of the last layer is given by:
[Equation]
where (cid:107) h (cid:48)
k (cid:107)
= o (1) , (cid:107) h (cid:48)(cid:48)
k (cid:107)
= o (1) for k = 1 , 2 .
Therefore, after the softmax operation, we can see that the 2-layer output recovers the membership of the elements exactly.
One general remark is that while the above argument is not limited to the leading two eigenvalues/eigenvectors, from the form of W (2) whose elements depend inversely on the corresponding eigenvalues, we can expect more numerical stability (and hence easier learning) when the corresponding eigenvalues are large.
A. P ROOF OF T HEOREM
Consider a stochastic block model with two communities, where the intra-community connection probability is a intra /n and the inter-community connection probability is a inter /n , and assume that a intra > a inter .
Let ξ 1 = a intra + a interand ξ 2 = a intra − a inter, which correspond to the ﬁrst and second eigenvalues of the expected adjacency matrix of the above model.
Let ν denote the community label vector, with ν i = ± 1 depending on the membership of node i .
For weak recovery, we assume the detectability condition (Decelle et al., 2011):
[Equation]
With the notations from the main paper, let A ( r ) θ = [Equation] rk =
θ k A [ k ] denote the θ -weighted variable power operator, where (cid:2)
A [ k ] (cid:3) ij = (cid:26) 1 if d G ( i, j ) =
k 0 o.w. is the k -distance adjacency matrix.
Let
A { k } denote the nonbacktracking path counting matrix, where the ( i, j ) -th component indicate the number of self-avoiding paths of graph edges of length k connecting i to j (Massoulié, 2014).
Our goal is to prove that the top eigenvectors of A ( r ) θ are asymptotically aligned with those of A { k } for k greater than log( n ) up to a constant multiplicative factor.
To do so, we ﬁrst recall the result of (Massoulié, 2014), which examines the behaviors of top eigenvectors of A { k } .
Lemma 6 ((Massoulié, 2014)) .
Assume that (7) holds.
Then, w.h.p., for all k ∈ { r/ 2 , ..., r } : (a) The operator norm (cid:107)
A { k } (cid:107) 2 is up to logarithmic factors Θ( ξ k 1 ) .
The second eigenvalue of A { k } is up to logarithmic factors Ω( ξ k 2 ) .
(b) The leading eigenvector is asymptotically aligned with A { k } 1 :
[Equation]
where (cid:107) h k (cid:107) =
o (1) .
The second eigenvector is asymptotically aligned with A { k } ν :
[Equation]
where (cid:107) h (cid:48)
k (cid:107) =
o (1) .
Now, we ﬁrst deﬁne Γ { r } θ
[Equation]
Also, by (Massoulié, 2014, Theorem 2.3), the local neighborhoods of vertices do not grow fast, and w.h.p., the graph of A ( r/ 2 − 1) has maximum degree O ( ξ r/ 2 1 (log n ) 2 ) .
Let φ be the leading eigenvector of A ( r/ 2 − 1) θ , and let v be the node where φ is maximum (i.e., φ ( v ) ≥ φ ( u ) ).
Without loss of generality, assume that φ ( v ) > 0 .
We have a good control over the ﬁrst eigenvalue λ 1 of
A ( r/ 2 − 1) θ as follows:
[Equation]
Due to (Abbe et al., 2018, Theorem 2), which proves an identical result of Lemma 6 but for A [ k ] , we know that w.h.p., for r = (cid:15) log( n ) , we can bound the last term by
[Equation]
Therefore
[Equation]
by triangle inequality
[Equation]
we can conclude that
[Equation]
Now [Equation] our goal is to show a similar result as Lemma 6 for Γ { r } θ .
In particular,
[Equation]
+ O   r (cid:88) k
= r/| θ k |(cid:107)
A { k } (cid:107) 2 (cid:13)(cid:13)(cid:13)(cid:13)
A { r } 1 (cid:107) A { r } 1 (cid:107) 2 −
A { k }(cid:107) A { k } 1 (cid:107)(cid:13)(cid:13)(cid:13)(cid:13) (10) = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) r (cid:88) k
= r/θ k A { k } A { k }(cid:107) A { k } 1 (cid:107) 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + O   r (cid:88) k
= r/| θ k | ξ k 1 n − δ 
 (11) =
(cid:13)(cid:13) (cid:13) (cid:13) (cid:13)(cid:13) r (cid:88) k
= r/θ k A { k } A { k }(cid:107) A { k } 1 (cid:107)(cid:13)(cid:13) (cid:13) (cid:13) (cid:13)(cid:13)
[-= r/θ k (cid:18) A { k }(cid:107) A { k } 1 (cid:107) 
 + h k (cid:19) (cid:107) A { k } A { k } 1 (cid:107)(cid:107) A { k } 1 (cid:107)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)-] + O (cid:0) (cid:107) θ (cid:107) 1 ξ r 1 n − δ (cid:1) [-for (cid:107) h k (cid:107) 
-](12) =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) r (cid:88) k
= r/θ k (cid:18)
A {[-r }(cid:107) A { r } 1 (cid:107) 
 + h k (cid:19) (cid:107) A { k } A {-] k }(cid:107)
[Equation]
[Equation]
where (10) and (14) are due to triangle inequalities, (11) and (15) are due to results in (Massoulié, 2014), and (13) is due to Lemma 6.
Therefore, we have (cid:107) Γ { r } θ A { r
[Equation]
(cid:107) 2 (cid:1) .
Similarly, we can prove that (cid:107)
[Equation]
Furthermore, we can show that Γ { r } θ satisﬁes the weak Ramanujan property (the proof is similar to (Massoulié, 2014) and is deferred to Section A.4).
Lemma 7.
For any ﬁxed (cid:15) > 0 , Γ { r } θ satisﬁes the following weak Ramanujan property w.h.p.
, sup
[Equation]
By Lemma 7, the leading two eigenvectors of Γ { r } θ will be asymptotically in the span of A { r } 1 and
A { r } ν according to the variational deﬁnition of eigenvalues.
By our previous analysis, this means that the top eigenvalue of Γ { r } θ will be Θ ( (cid:107) θ (cid:107) 1 ξ r 1 ) with eigenvector asymptotically aligned with
A { r } 1 .
Since by (Massoulié, 2014, Lemma 4.4), A { r } 1 and A { r } ν are asymptotically orthogonal, the second eigenvalue of Γ { r } θ will be Θ ( (cid:107) θ (cid:107)
ξ r 2 ) with eigenvector asymptotically aligned with
A { r } ν .
Since the perturbation term (cid:107) ∆ { r } θ (cid:107) 2 = o (cid:16) λ 2 ( Γ { r } θ ) (cid:17) by (8), using Weyl’s inequality (Weyl, 1912), we can conclude that the leading eigenvalue λ 1 (cid:16) A ( r ) θ (cid:17) =
Θ ( (cid:107) θ (cid:107) 1 ξ r 1 ) , the second eigenvalue λ 2 (cid:16) A
( r ) θ (cid:17) =
Θ ( (cid:107) θ (cid:107) 1 ξ r 2 ) , and the rest of the eigenvalues are bounded by (cid:107) θ (cid:107) 1 n (cid:15)
ξ r/ 2 1 O (log( n )) .
Moreover, since (cid:107) ∆ { r } θ (cid:107) 2 = o (cid:16) max (cid:110) λ 1 ( Γ { r } θ ) − λ 2 ( Γ { r } θ ) , λ 2 ( Γ { r } θ ) − λ 3 ( Γ { r } θ ) (cid:111)(cid:17) , by the
Davis-Kahan Theorem (Davis & Kahan, 1970), the leading two eigenvectors of A ( r ) θ are asymptotically aligned with A { r } 1 and A { r } ν , which is shown to be enough for the rounding procedure of (Massoulié, 2014) to achieve weak recovery.
A. P ROOF OF L EMMA
Denote (cid:101)
[Equation] [Equation] + νν [Equation]
) − I (cid:3) + a inter 2 n ( 11 (cid:62) − νν (cid:62) )
as the expected adjacency matrix conditioned on community labels ν .
Let P ij denote the set of all self-avoiding paths i k 0 :
= { i 0 , . .
, i k } from i to j , such that no nodes appear twice in the path, and deﬁne Q { m } ij : = (cid:80) [Equation] P
[Equation]
(cid:81) mt
{+=
+}
( A − (cid:101) A )
i t − 1 i t .
Let T mij be the concatenation of self-avoiding paths
i k − mand
i kk − m +1 , and let R mij denote
T m ij \ P ij .
Deﬁne W { k,m } ij
[Equation]
[Figure]
Then, by (Massoulié, 2014, Theorem 2.2), we can have
[Equation]
W
[Equation]
Hence, Γ { r } θ = r (cid:88) k
= r/θ k A { k } = r (cid:88) k
= r/θ k Q { k } + r (cid:88) k = r/θ k k (cid:88) m = (cid:16) Q { k − m } (cid:101) AA { m − 1 } (cid:17) − r (cid:88)
[Equation]
=
[Equation]
[Equation]
[Equation]
= r (cid:88) k
= r/θ k Q { k } + r (cid:88) m =   r (cid:88) k
=max( r/ 2 ,m )
θ k Q { k − m }
[Equation]
In particular, for any u that is orthogonal to A { m } 1 and A { m } ν with norm-1 (i.e., a feasible vector of the variational deﬁnition in Lemma 7), we have:
[Equation]
 
Since by (Massoulié, 2014), the terms ρ (cid:0) Q { k } (cid:1) and ρ (cid:0) W { k,m } (cid:1) are bounded by n (cid:15)
ξ r/ 2 1 , (cid:13)(cid:13)(cid:13) (cid:101) AA { m − 1 } x (cid:13)(cid:13)(cid:13) 2 is bounded by O (log( n ) + (cid:113) log( n ) ξ m − 1) , with some elementary calculations, we have that (cid:13)(cid:13)(cid:13) Γ { r } θ u (cid:13)(cid:13)(cid:13) 2 is bounded by (cid:107) θ (cid:107) 1 n (cid:15)
[Table]
(log( n ))
[Table]
[Table]
M ORE EXPERIMENTAL DETAILS AND RESULTS
Experimental setup.
We followed the setup of (Yang et al., 2016; Kipf & Welling, 2017) for citation networks Citeseer, Cora and Pubmed.
The statistics of datasets is in Table A.1.
Degree distribution in the original graph and the powered graphs for Citeseer and Cora is shown in Figure A.1.
We used the same dataset splits as in (Yang et al., 2016; Kipf & Welling, 2017) withlabels per class for training, 500 labeled samples for hyper-parameter setting, and 1 , 000 samples for testing.
To facilitate comparison, we use the same hyper-parameters as Vanilla GCN (Kipf &
Welling, 2017), including number of layers ( 2 ), number of hidden units ( 16 ), dropout rates ( 0 . 5 ), weight decay ( 5 × 10 − 4 ), and weight initialization following (Glorot & Bengio, 2010) implemented in Tensorﬂow.
The training process is terminated if validation accuracy does not improve forconsecutive steps.
We note that the citation datasets are extremely sensitive to initializations; as such, we report the test accuracy for the top 50 runs out of 100 random initializations sorted by the validation accuracy.
We used Adam (Kingma & Ba, 2014) with a learning rate of 0.01 except for θ (VPN), which was chosen to be 1 × 10 − 5 to stabilize training.
For VPN, we consider two-step training process, i.e. ,
we ﬁrst employ graph sparsiﬁcation on adjacency matrix
¯ A [ k ] based on X for the ﬁrst-step training, then process second graph sparsiﬁcation on adjacency matrix
¯ A [ k ] with respect to the feature embedding from hidden layer and fetch them back into model for second-step training with warm restart (continue training with the parameters from ﬁrst-step).
We set α k to be 0.5 for the corresponding power order and 0 other wise in r-GCN.
To enhance robustness against evasion attack, we adaptively chose the threshold of graph sparsiﬁcation. For each node, we consider all its neighbors within r hops, where r is the order of the powered graph. We order these neighbors based on their Euclidean distance to the current node in the feature space.
We then set the threshold for this node such that only the ﬁrst s ∗ d neighbors with the smallest distance are selected, where s is the sparsiﬁcation rate, and d is the degree of the node in the original graph.
If this node is a high-degree node (determined by if the degree is higher than the average degree by more than 2 standard deviations), we set s = 1 ; otherwise, we set s to be a small number slightly larger than 1. Nevertheless, we do not observe that the robustness performance is sensitive to this parameter. In addition, one can deﬁne their own distance function that better measures proximity between nodes, such as correlation distance or cosine distance. As a side remark, one can easily obtain A ( r ) with 1 [( I + A ) r > 0] , and recursively calculate A [ k ] .
The proposed methods have similar computational complexity as vanilla GCN as a result of sparsiﬁcation. [Figure] i [Figure]
v a l
[Figure] S p ec t r a l c o e ffi c i e n t [Figure]
Revisiting SBM.
To obtain networks with different SNRs (0.58, 0.68, 0.79, 0.85, 0.91), we set the intra-degree average a intra = 2 . 1 and inter-degree average a inter to be 0.4, 0.3, 0.2, 0.15, 0.
respectively.
We generate 10 networks for each SNR.
One of the networks with SNR of 0.91 is analyzed in Figure A.3 for different convolution operators.
As predicted by Theorem 3, the proposed operators have nontrivial spectral gap.
This is not enjoyed by other operators, so it is a quite unique property.
Furthermore, our learned operator w/ or w/o normalization obtained 81% accuracy based [Table] [Figure] [Table]
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
Spectral gap Spectral gap
Leadingeigenvalues
Leadingeigenvalues s p ec t r u m
S p e c t r a l c l u s t e r i n g on the second eigenvector, whereas adjacency matrix catches “high-degree nodes” and normalized (or powered) Laplacians catch “tails”, with spectral clustering accuracy of only 51%.
Visualization of network embeddings.
We used t-SNE (Maaten & Hinton, 2008) to visualize the network embeddings learned by r-GCN (Figure A.4) and VPN (Figure A.5) for Citeseer, Cora and
Pubmed.
As can be seen, with very limited labels, the proposed methods can learn an effective embedding of the nodes.
Evasion experiment.
Five representative global adversarial attack methods are considered as comparisons:
• DICE (Zügner & Günnemann, 2019) (remove internally, insert externally): it randomly disconnects nodes from the same class and connects nodes from different classes.
• A abr and A DW 3 (Bojchevski & Günnemann, 2019): both attack methods are matrix perturbation theory based.
A DW 3 is the closed-form attack for edge deletion and A abr is the add-by-remove strategy based method for edge insertion.
• Meta-Train and Meta-Self (Zügner & Günnemann, 2019): both attack approaches are meta learning based.
The meta-gradient approach with self-training is denoted as Meta-Self and the variant without self-training is denoted as Meta-Train.
Note that both attack methods result in OOM (out of memory) on Pubmed, we choose to report the performance against them only on Citeseer and Cora.
We provide the post-evasion accuracy in Table A.2 and Table A.3 and robustness enhancement in
Figure A.6, Figure A.7 and Figure A.8 for Citeseer, Cora, and Pubmed.
Results are obtained with 20 random initializations on a given attacked network.
Computational efﬁciency.
The adjacency matrix could be much denser after being powered several times, which poses a general challenge for all high-order matrix-based approaches.
To alleviate this issue, we employ a simple sparsiﬁcation strategy as described in the main paper.
To demonstrate the computational efﬁciency of the proposed method, we introduce a running time comparison experiment on the real world social network (Social circles: Facebook)
(Leskovec & Mcauley, 2012).
This small dataset consists of “circles” (or “friends lists”) from Facebook and becomes very dense from ﬁrst order ( 4 . 71% ) to second order ( 92 .
12% ), thus is suitable to be used for computational efﬁciency evaluation.
The results are shown in Table A.4.
We can ﬁnd that the running efﬁciency of the proposed method is compatible with baselines and the density does not affect the efﬁciency signiﬁcantly when the dataset is not too large.
Moreover, in this paper, our primary goal is to improve the robustness.
We
[Table]
[Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Table]
[Table] [Figure] [Figure] [Figure]