L OW - RANK PROJECTIONS OF GCN S L APLACIAN
A BSTRACT
In this work, we study the behavior of standard models for community detection under spectral manipulations.
Through various ablation experiments, we evaluate the impact of bandpass ﬁltering on the performance of a GCN: we empirically show that most of the necessary and used information for nodes classiﬁcation is contained in the low-frequency domain, and thus contrary to images, high frequencies are less crucial to community detection.
In particular, it is sometimes possible to obtain accuracies at a state-of-the-art level with simple classiﬁers that rely only on a few low frequencies.
I NTRODUCTION
Graph Convolutional Networks (GCNs) are the state of the art in community detection (Kipf &
Welling, 2016).
They correspond to Graph Neural Networks (GNNs) that propagate graph features through a cascade of linear operator and non-linearities, while exploiting the graph structure through a linear smoothing operator.
However, the principles that allow GCNs to obtain good performances remain unclear.
Our work actually suggests that, in the setting of community detection, graph Laplacian high frequencies have actually a minor impact on the classiﬁcation performances of a standard
GCN, as opposed to standard Convolutional Neural Networks for vision.
Spectral clustering is a rather different point of view from deep supervised GCNs which studies node labeling in unsupervised contexts: it generally relies on generative models based on the graph spectrum.
Our paper proposes to establish a clear link between these approaches: we show that the informative graph features are located in a low-frequency band of the graph Laplacian and can be efﬁciently used in a deep supervised classiﬁer.
This paper shows that experiments on standard community detection datasets like Cora, Citeseer,
Pubmed can be conducted using only few frequencies of their respective graph spectrum without observing any signiﬁcant performances drop.
Other contributions are as follows: (a) First we show that most of the necessary information exploited by a GCN for a community detection task can actually be isolated in the very ﬁrst eigenvectors of a Laplacian.
(b) We observe that a simple
Multi-Layer Perceptron (MLP) method fed with handcrafted features accounting for low-frequency eigenvalues allows to successfully deal with transdusctive datasets like Cora, Citeseer or Pubmed.
R ELATED W ORK
GCNs and Spectral GCNs Introduced in Kipf & Welling (2016), GCNs allow to deal with large graph structure in semi-supervised classiﬁcation contexts.
This type of model works at the node level, meaning that it uses locally the adjacency matrix.
This approach has inspired a wide range of models, such as linear GCN (Wu et al., 2019), Graph Attention Networks (Veliˇckovi´c et al., 2017),
GraphSAGE (Hamilton et al., 2017), etc.
In general, this line of work does not consider directly the graph Laplacian.
Another line of work corresponds to spectral methods, taking inspiration from
Graph Signal Processing (GSP), a popular ﬁeld whose objective is to manipulate signals spectrum whose topology is given by a graph.
They employ ﬁlters which are designed from the spectrum of a graph Laplacian (Bruna et al., 2013; Oyallon, 2020; 2017).
In general, those works make use of polynomials in the Laplacian (Defferrard et al., 2016) .
All those references share the idea to manipulate bandpass ﬁlters that discriminate the different ranges of frequencies.
Over-smoothing in GCNs In the context of GCN, Li et al.
(2018) is one of the ﬁrst papers to notice that cascading low-pass ﬁlters can lead to a substantial information loss.
The result of our work indicates that the important spectral components for detecting communities are already in the low-frequency domain and that this is not due to an architecture bias.
Zhao & Akoglu (2019); Yang et al.
(2020) propose to introduce regularizations which address the loss of information issues.
Cai & Wang (2020); Oono & Suzuki (2019) study the spectrum of a graph Laplacian under various transforms, yet they consider the spectrum globally and in asymptotic settings, with a deep cascade of layers.
Huang et al.
Rong et al.
(2019) introduce data augmentations, whose aim is to alleviate over-smoothing in deep networks: we study GCNs without this ad-hoc procedure.
Spectral clustering and low rank approximation As the literature about spectral clustering is large, we mainly focus on the subset that connects directly with GCN.
Mehta et al.
(2019) proposes to learn an unsupervised auto-encoder in the framework of a Stochastic Block Model.
Oono & Suzuki (2019) introduces the Erd ¨ os – Renyi model in the GCN analysis, but only in an asymptotic setting.
Loukas & Vandergheynst (2018) studies the graph topology preservation under the coarsening of the graph, which could be a potential direction for future works. Liao et al.
(2019) uses the Lanczos algorithm to construct low rank approximations of the graph Laplacian, which facilitates efﬁcient computation of matrix powers.
Node embedding A MLP approach can be understood as an embedding at the node level.
For instance, Aubry et al.
(2011) applies a spectral embedding combined with a diffusion process for shape analysis, which allows point-wise comparisons.
We also mention Deutsch & Soatto (2020) that uses a node embedding, based on the spectrum of a quite modiﬁed graph Laplacian, obtained from on a measure of node centrality.
GCN stability Stability of GCNs has been theoretically studied in Gama et al.
(2019), which shows that deﬁning a generic notion of deformations is difﬁcult.
Surprisingly, it was noted in Oyallon (2020) that stability is not a key component to good performances.
The stability of GCN has also been investigated in Verma & Zhang (2019) but only considers neural networks with a single layer and relies on the whole spectrum of the learned layer. Keriven et al. (2020) considers the stability of GCNs, and relies on an implicit Euclidean structure: it is unclear if this holds in our settings.
Sun et al.
(2020) is one of the ﬁrst works to study adversarial examples linked to the node connectivity and introduces a loss to reduce their effects.
Zhu et al.
(2019) also addresses the stability issues by embedding GCNs in a continuous representation.
None of these work directly related a trained GCN to spectral perturbations.
F RAMEWORK 3. M ETHOD
We ﬁrst describe our baseline model.
Our initial graph data are a graph with N nodes, its adjacency matrix A with diagonal degree matrix D , and node features X .
We consider ﬁrst-order GCNs as introduced in (Kipf & Welling, 2016), which correspond to GNNs that propagate node feature input H [Equation] 0) (cid:44) X through a cascade of layers, via the iteration:
[Equation]
(1) where ˜ A = 12 ( I + D − 1 / 2 AD − 1 / 2 ) , σ a point-wise non-linearity and W ( l ) a parametrized afﬁne operator.
Note that if ˜ A = I N , then Eq.
1 is simply an MLP.
The 12 factor is a normalization factor to obtain (cid:107) ˜
A (cid:107) = 1 .
In the semi-supervised setting, a ﬁnal layer f
( X, ˜ A )
(cid:44) H ( L ) is fed to a supervised loss (cid:96) (here a softmax) and { W (0) , ..., W L − 1 } are trained in an end-to-end manner to adjust the label of each node.
We note that for undirected graph, ˜ A is a positive deﬁnite matrix with positive weights, which can be understood as an averaging operator (Li et al., 2018).
We are interested in analyzing the properties of spectral approximations of ˜ A .
We consider the decreasing set of eigenvalues Λ = { λ k } k ≥ 0 of ˜ A , such that λ k ≥ λ k +1 , and we denote by u k the k -th eigenvector corresponding to λ k ∈ Λ .
We remind that Λ ⊂ [0 , 1] and that the eigenspace corresponding to λ = 1 can be interpreted as the kernel of the symmetric normalized graph Laplacian.
We then write ˜
A [ k 1 ,k 2 ] (cid:44) (cid:80) k 1 ≤ k ≤ k 2 λ k u k u Tk , such that ˜
A
=
˜
A [0 ,N ] .
We are interested in studying the degradation accuracy if we replace ˜ A with ˜
A
[0 ,k ] or ˜ A [ k,N ]
for some 0 < k < N .
E XPERIMENTS
Matching the previous work practice, we focus on the three classical benchmark dataset for community detection: Cora, Citeseer and Pubmed (Sen et al., 2008).
The task consists in classifying the research topic of papers in three citation datasets.
Those tasks are transductive, meaning all node features are accessible during training.
We use the same training/validation/test split as Huang et al.
(2018a), Chen et al.
(2018), and Rong et al.
(2019) on all datasets in our experiments, which corresponds to a supervised learning scenario.
The statistics of each dataset are listed in the supplemental materials, as well as the training details.
4. L OW RANK APPROXIMATION
[Figure]
GCN ablation Here, we consider the two projections ˜
A [0 ,k ] and ˜ A
[ N − k,N ] , where k is adjusted to retain only a portion of the spectrum.
Those projections can be interpreted respectively as highpass and low-pass ﬁlters.
As detailed in Appendix C, those projections will allow to study which frequency band is important for the community detection task.
Fig.
1 and Fig.
2 report the respective performance when considering the models f
( X, ˜
A [ N − k,N ] ) and f ( X, ˜
A [0 ,k ] ) for some k .
On Fig.
1, we observe that retaining only very few frequencies (less than 10%) does not degrade much the accuracy of the original network.
This does not contradict the observation of Li et al.
(2018) which studies empirically the over-smoothing phenomenon, as our ﬁnding indicates that a GCN uses mainly the low frequency domain.
For Pubmed, using almost all the high frequencies is required to recover the original accuracy of our model.
Interestingly, deeper GCNs seem to beneﬁt from the high frequency ablation, yet their accuracy remains below their shallow counter-part and they are still difﬁcult to train.
This instability to spectral perturbations is further studied in Appendix D. Fig.
2 indicates that the major information for supervised community detection is contained in the low frequencies: dropping the latter leads to substantial accuracy drop, even for a shallow GCN.
[Figure]
[Figure]
MLP augmentation We further study the importance of low frequencies via a spare ablation experiment based on a MLP.
We augment each graph features X through the concatenation X k =
[ X, u T 1 , ..., u Tk ] of the ﬁrst k eigenvectors.
Fig.
3 reports the accuracy of our MLP trained on X k as a function of k .
We see that using a fraction kN ≤ 20% of the eigenvectors allows to recover the performance of a GCN.
More surprisingly, we note that as k increases, the accuracy drops, which indicates that high frequencies behave like a residual noise that is overﬁtted.
This experiment emphasizes that a low rank approximation of the Laplacian can be beneﬁcial to a MLP classiﬁer.
[Table]
Boosting MLP performances We perform a hyper-parameter grid search on each dataset to investigate MLP strengths further.
We summarize our ﬁndings in Tab. 1.
In particular, one uses a fraction 6%, 15.0% and 0.7% of the spectrum respectively on Cora, Citeseer and Pubmed in order to obtain our best performances.
More details on the methodology are provided in the supplementary material.
This simple model is often competitive with concurrent works, and in particular with vanilla GCNs.
Note also that our method does not incorporate any data augmentation procedure.
We conclude that GCNs do not compute more complex invariants than a MLP fed with low frequencies, in the context of community detection.
C ONCLUSION
We have studied the classiﬁcation performance of a GCN under low-pass and high-pass ﬁltering, in the context of community detection.
Our ﬁnding is that GCNs rely signiﬁcantly on the low frequencies, and can even beneﬁt from high-frequency ablations.
Then, we are able to design MLPs that rely simply on a few eigenvectors of the graph Laplacian and can be competitive with GCNs.
We observed that the high-frequency does not bring a signiﬁcant amount of information, and can even be interpreted as a residual noise in this particular setting.
Our methodology can also help to identify if the accuracy improvement of a new architecture is due to a better processing of high frequencies. A CKNOWLEDGEMENTS
EO was granted access to the HPC resources of IDRIS under the allocation 2020-[AD011011216R1] made by GENCI and this work was partially supported by ANR-19-CHIA ”SCAI”.
Experiments presented in this paper were partially carried out using the Grid’5000 testbed, supported by a scientiﬁc interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations.
NG is recipient of a PhD funding from AMX program, Ecole polytechnique.
D ATASET
STATISTICS
[Table]
A.
S PECTRA OF C ORA , C ITESEER , P UBMED
[Figure]
Observe that the decay of the spectrum is fast.
B T RAINING D ETAILS
We choose σ to be the ReLU non-linearity (Krizhevsky et al., 2012).
Unless speciﬁed otherwise, the weights of our models (either GCN or MLP) are optimized via Adam, with an initial learning rate 0.01 and weight decay of 0.001, during 800 epochs.
We use by default a dropout of 0.5.
Our model consists in GCN layers with 2 hidden layers of size 128.
When using grid search, we cross validate our hyper-parameters on a validation set, using early stopping with a patience of 400.
The grid search is detailed in the following table.
Unless speciﬁed otherwise, each plot is obtained by an average over 3 different seeds.
B. H YPER - PARAMETER D ESCRIPTION C U NDERSTANDING LOW RANK APPROXIMATIONS FOR GCN S
We now justify our approach under the standard setting of the Stochastic Block Model (Abbe, 2017), and we will simply remind the reader of several elementary results.
This model corresponds to a generative model that describes the interaction between r communities { C 1 , ..., C r } .
Assuming that two nodes i, j belong to the communities
C r
i , C r j , an edge is sampled with a probability p r i ,r j (Rohe et al., 2011).
For the sake of simplicity, let us assume that r = 2 , that the probability of an edge between two nodes i, j is p if those nodes belong to the same community and q < p otherwise, and that both communities correspond to | C | nodes.
In this case, the unnormalized
[Table]
expected adjacency matrix is given by:   p · · · p q · · · q ... p · · · p q · · · q q · · · q p · · · p ... q · · · q p · · · p   , (2) where we grouped in matrix block the nodes from the same community.
We note that the two dominant eigenvectors are given by:
u 1 = [ | C | times (cid:122) (cid:125)(cid:124) (cid:123) 1 , ..., 1 , 1 , ...,(cid:124) (cid:123)(cid:122) (cid:125) | C | times ] and u 2 = [ | C | times (cid:122) (cid:125)(cid:124) (cid:123) 1 , ..., 1 , − 1 , ..., −(cid:124) (cid:123)(cid:122) (cid:125) | C | times ] .
Observe that the second eigenvector u 2 captures all the information about the two communities, through the sign of its coefﬁcients.
Here, the spectral gap (the ratio between the two dominant eigenvalues) is given by 0 ≤ p − q p + q < 1 and ideally this spectral gap should be as large as possible for identifying the two communities.
If the number of nodes is large, concentration results (Wainwright, 2019) imply that the empirical adjacency matrix concentrates around its expectation, and that under this assumption, a low-rank approximation ˜
A [0 , 2] captures most of the available information about the two communities.
We illustrate this idea on Fig.
While these assumptions might not hold in practice, it justiﬁes why low-rank approximations of a Laplacian are relevant in the setting of community detection and it explains why high frequencies might not be as important as low frequencies for supervised community detection task.
The next section validates empirically this approach in the context of GCNs and simpler architectures.
D S TABILITY TO H IGH F REQUENCIES
We now study the stability of a GCN w.r.t.
spectrum perturbations.
In the case of image processing, it is standard that low frequencies almost do not affect the classiﬁcation performances and that perturbations of high frequencies lead to instabilities.
We would like to validate that this principle does not hold here, and to do so, we consider ∇ Λ (cid:96) , which is the gradient w.r.t.
every singular value λ k .
Small amplitudes of | ∂(cid:96) ∂λ k | indicate more stable coefﬁcients.
Fig.
6 plots the amplitude of the gradient w.r.t.
λ k at the initialization of a GCN and after training.
First, we note that a
GCN is more sensitive to spectral perturbations after training, which is logical because the GCN adapts its weights to the speciﬁc structure of a given graph.
After training, we remark that the highfrequency perturbations have a small impact compared to the low-frequency perturbations on the three datasets, except for Pubmed which has dominant low frequencies.
This is consistent with our previous ﬁndings. [Figure] uuuuuuuu [Figure]
[Figure]
E N OTE ON THE COMPUTATIONAL OVERHEAD
The MLPs introduced above are of interest if the corresponding graph topology is ﬁxed, with a large graph [Figure] and high connectivity.
Indeed, using a MLP allows to easily employ mini-batch strategies and the training data can be reduced according to the fraction of low frequencies being kept: an exact k -truncated SVD has a complexity about O ( kN 2 ) .
We note that fast k -truncated (cid:15) -approximate
SVD algrotithms for sparse matrix exist (Allen-Zhu & Li, 2016): if ρ is the number of non-zero coefﬁcients of ˜ A , the complexity can be about O ( kρ (cid:15) + k 2 N (cid:15) ) .