S UPERVISED C OMMUNITY D ETECTION WITH L INE G RAPH N EURAL N ETWORKS
A BSTRACT
Community detection in graphs can be solved via spectral methods or posterior inference under certain probabilistic graphical models.
Focusing on random graph families such as the stochastic block model, recent research has uniﬁed both approaches and identiﬁed both statistical and computational detection thresholds in terms of the signal-to-noise ratio.
By recasting community detection as a node-wise classiﬁcation problem on graphs, we can also study it from a learning perspective.
We present a novel family of Graph Neural Networks (GNNs) for solving community detection problems in a supervised learning setting.
We show that, in a data-driven manner and without access to the underlying generative models they can match or even surpass the performance of the belief propagation algorithm on binary and multiclass stochastic block models which is believed to reach the computational threshold in these cases.
In particular, we propose to augment GNNs with the non-backtracking operator deﬁned on the line graph of edge adjacencies.
The GNNs are achieved good performance on real world datasets.
In addition we perform the ﬁrst analysis of the optimization landscape of using (linear) GNNs to solve community detection problems demonstrating that under certain simpliﬁcations and assumptions, the loss value at any local minimum is close to the loss value at the global minimum/minima.
I NTRODUCTION
Graph inference problems encompass a large class of tasks and domains, from posterior inference in probabilistic graphical models to community detection and ranking in generic networks, image segmentation or compressed sensing on non-Euclidean domains.
They are motivated both by practical applications, such as in the case of PageRank, and also by fundamental questions on the algorithmic hardness of solving such tasks.
From a data-driven perspective, these problems can be formulated in unsupervised, semi-supervised or supervised learning settings.
In the supervised case, one assumes a dataset of graphs with labels on its nodes and/or edges, and attempts to perform node/edge classiﬁcation by optimizing a loss over a certain parametric class, e.g. neural networks.
Graph Neural Networks ((Gori et al., 2005), (Bronstein et al., 2017) and references therein) are natural extensions of Convolutional Neural
Networks to graph-structured data, and have emerged as a powerful class of algorithms to perform complex graph inference leveraging labeled data.
In essence, these neural networks learn cascaded linear combinations of intrinsic graph operators interleaved with node-wise (or edge-wise) activation functions.
Since they learn from intrinsic graph operators, they can be applied to varying input graphs, and they offer the same parameter sharing advantages as their CNN counterparts.
In this work, we focus on community detection problems, a wide class of node classiﬁcation tasks that attempt to discover a clustered, segmented structure within a graph.
The algorithmic approaches to this problem include a rich class of spectral methods, which take advantage of the spectrum of certain operators deﬁned on the graph, as well as approximate message-passing methods such as belief propagation (BP), which performs approximate posterior inference under predeﬁned graphical models (Decelle et al., 2011).
Focusing on the supervised setting, we study the ability of GNNs to approximate, generalize or even improve upon these class of algorithms.
Our motivation is two-fold.
On the one hand, this problem exhibits algorithmic hardness on some settings, opening up the possibility to discover more efﬁcient algorithms than the current ones.
On the other hand, many practical scenarios fall beyond pre-speciﬁed probabilistic models, requiring data-driven solutions.
We propose modiﬁcations to the GNN architecture, which allows it to exploit edge adjacency information, by incorporating the non-backtracking operator of the graph.
This operator was introduced to community detection problems by (Krzakala et al., 2013), which proposed a spectral method based on the non-backtracking operator.
This operator is deﬁned over the edges of the graph and allows a directed ﬂow of information even when the original graph is undirected.
We refer to the resulting model as a Line Graph Neural Network (LGNN) .
Focusing on important random graph families exhibiting community structure, such as the stochastic block model (SBM) and the geometric block model (GBM), we demonstrate improvements in the performance by LGNN, even in regimes within the so-called computational-to-statistical gap.
A perhaps surprising aspect is that these gains can be obtained even with linear GNNs, which become parametric versions of power iteration algorithms.
We want to mention that besides community detection tasks, GNN and LGNN can be applied to other node-wise classiﬁcation problems too.
The reason we are focusing on community detection problems is that this is a relatively well-studied setup, for which different algorithms have been proposed and where computational and statistical thresholds have been studied in several scenarios.
Moreover, synthetic datasets can be easily generated for community detection tasks.
Therefore, we think it is a nice setup for comparing different algorithms, besides its practical values.
The good performances of GNN and LGNN motivate our second main contribution: the analysis of the optimization landscape of simpliﬁed and linear GNN models when trained with planted solutions of a given graph distribution.
Under reparametrization, we provide an upper bound on the energy gap controlling the energy difference between local and global minima (or minimum).
With some assumptions on the spectral concentration of certain random matrices, this energy gap will shrink as the size of the input graphs increases, which would mean that the optimization landscape is benign on large enough graphs.
Summary of Main Contributions:
• We propose an extension of GNNs that operate on the line graph using the non-backtracking operator, which yields improvements on hard community detection regimes.
• We show that on the stochastic block model we reach detection thresholds in a purely data-driven fashion, in the sense that our results improve upon belief propagation in hard SBM detection regimes, as well as in the geometric block model.
• We perform the ﬁrst analysis of the learning landscape of GNN models, showing that under certain simpliﬁcations and assumptions, they exhibit a form of “energy gap”, where local mimima are conﬁned in low-energy conﬁgurations.
• We show how our model can be applied to real-world datasets, leading to improved community detection results.
P ROBLEM S ETUP
We are interested in a speciﬁc class of node-classiﬁcation tasks in which given an input graph
G =
( V, E ) , a labeling y :
V → { 1 , . . , C } that encodes a partition of V into C communities is to be predicted at each node.
We assume that a training set { ( G t , y t ) } t ≤ T is given, with which we train a model that predicts ˆ y =
Φ( G, θ ) by minimizing
[Equation]
Since y encodes a partition of C groups, the speciﬁc label of each node is only important up to a global permutation of { 1 , . . , C } .
Section 4.3 describes how to construct loss functions (cid:96) with such a property.
A permutation of the observed nodes translates into the same permutation applied to the labels, which justiﬁes models Φ that are equivariant to permutations.
Also, we are interested in inferring properties of community detection algorithms that do not depend on the speciﬁc size of the graphs 1 .
We therefore require that the model Φ accepts graphs of variable size for the same set of parameters, similar to sequential RNN or spatial CNN models.
In our study of random graph models, we construct a training set of planted solutions.
Labels y
i are generated by sampling a balanced partition uniformly at random, and then we produce the input graphs G i by sampling G i | y
i according to each random graph model.
R ELATED W ORK
GNN was ﬁrst proposed in Gori et al. ; Scarselli et al.
Bruna et al. (2013) generalized convolutional neural networks on general undirected graphs by using the graph Laplacian’s eigenbasis.
This was the ﬁrst time the Laplacian operator was used in a neural network architecture to perform classiﬁcation on graph inputs.
Defferrard et al. (2016) considers a symmetric Laplacian generator to deﬁne a multiscale GNN architecture, demonstrated on classiﬁcation tasks.
Similarly, Kipf & Welling (2016) uses a similar generator as effective embedding mechanisms for graph signals and applies it to semi-supervised tasks.
This is the closest application of GNNs to our current contribution.
However, we highlight that semi-supervised learning requires bootstrapping the estimation with a subset of labeled nodes, and is mainly interested in generalization within a single, ﬁxed graph.
In comparison, our setup considers community detection across a distribution of input graphs and assumes no initial labeling on a given test-set input graph.
There have been several extensions of GNNs (Li et al., 2015; Sukhbaatar et al., 2016; Duvenaud et al., 2015; Niepert et al., 2016) by modifying their non-linear activation functions, their parameter sharing strategies, and their choice of graph operators.
In particular, Gilmer et al.
(2017) interpret the GNN architecture as learning an approximate message-passing algorithm, which extends the learning of hidden representations to graph edges in addition to graph nodes.
Recently, Velickovic et al. (2017) relate adjacency learning with attention mechanisms, and Vaswani et al. (2017) propose a similar architecture in the context of machine translation.
Another recent and related piece of work is Kondor et al. (2018), which propose a generalization of GNN that captures high-order node interactions through covariant tensor algebra.
Our approach to extend the expressive power of GNN using the line graph may be seen as an alternative to capture such high-order interactions.
Our energy landscape analysis is related to the recent paper Shamir (2018), which establishes an energy bound on the local minima arising in the optimization of ResNets.
In our case, we exploit the properties of the community detection problem to produce an energy bound that depends on the concentration of certain random matrices, which one may hope for as the size of the input graphs increases.
Finally, Zhang (2016)’s work on data regularization for clustering and rank estimation is also motivated by the success of using Bethe-Hessian-like perturbations to improve spectral methods on sparse networks.
It ﬁnds good perturbations via matrix perturbations and also has successes on the stochastic block model.
Yang & Leskovec (2012a) curate benchmark datasets for community detection and quantify the quality of these datasets, while Yang & Leskovec (2012b) develop new algorithms for community detection by ﬁtting to the networks the Afﬂiation Graph Model (AGM), a generative model for graphs with overlapping communities.
L INE G RAPH N EURAL N ETWORKS
This section introduces our GNN architectures based on the power graph adjacency (Section 4.1) and its extension to line graphs using the non-backtracking operator (Section 4.2), as well as the design of losses invariant to global label permutations (Section 4.3).
4. G RAPH N EURAL N ETWORKS USING A FAMILY OF GRAPH OPERATORS
The Graph Neural Network (GNN), introduced in Scarselli et al. (2009) and later simpliﬁed in Li et al. ; Duvenaud et al. ; Sukhbaatar et al. (2016), is a ﬂexible neural network architecture based on local operators on a graph G =
( V, E ) .
Given a state vector x ∈ R | V |× b on the vertices of G , we consider intrinsic linear operators of the graph that act locally on x , which can be represented as | V | -by- | V | matrices.
For example, the adjacency matrix A is deﬁned entry-wise by A
i 1 i 2 = 1 if ( i 1 , i 2 ) ∈ E and A
{+i 1 i 2 = 
 if ( i 1 , i 2 ) / ∈ E , for every pair ( i 1 , i 2 ) ∈ V × V . The degree matrix D is then deﬁned as diag (+} A 1 ) , i.e., D is a diagonal matrix with D ii being the number of edges that the i th node has.
We can also deﬁne power graph adjacency matrices as A ( j ) = min(1 , A 2 j ) , which encodes 2 j -hop neighborhoods into a binary graph. Finally, there is also the identity matrix , I . Given such a family of operators for each graph, F JA =
{ I, D, A, A (2) , ..., A ( J ) } , we deﬁne a GNN layer that maps x ( k ) ∈ R | V |× b k to x ( k +1) ∈ R | V |× b k +1 as follows. First, we compute
[Equation]
where θ j ∈ R b k × bk + are trainable parameters and ρ ( · ) is a point-wise nonlinear activation function, chosen in this work to be the ReLU function, i.e. ρ ( z ) = max(0 , z ) for z ∈ R .
Then we deﬁne x ( k +1) =
[ z ( k +1) , z ( k +1) ]
∈ R | V |× b k +1 as the concatenation of z ( k +1) and z ( k +1) .
The layer thus includes linear “residual connections" (He et al., 2016) via z ( k ) , both to ease with the optimization when using large number of layers and to increase the expressivity of the model by enabling it to perform power iterations.
Since the spectral radius of the learned linear operators in (1) can grow as the optimization progresses, the cascade of GNN layers can become unstable to training.
In order to mitigate this effect, we consider spatial batch normalization (Ioffe & Szegedy, 2015) at each layer.
In our experiments, the initial states are set to be the degrees of the nodes, i.e., x (0) = deg( A ) :
= A 1 .
In addition, the model Φ( G, x (0) ) =
x ( K ) satisﬁes the permutation equivariance property required for community detection: Given a permutation π among the nodes in the graph, Φ( G π , Π x (0) )
=
ΠΦ( G, x (0) ) , where Π is the permutation matrix associated with π .
Analogy between GNN and power iterations In our setup, spatial batch normalization not only prevents gradient blowup, but also performs the orthogonalisation relative to the constant vector, which reinforces the analogy with the spectral methods for community detection, some background of which is described in B.1.
In essence, in certain regimes, the eigenvector of A corresponding to its second largest eigenvalue and the eigenvector of the Laplacian matrix , L = D − A , corresponding to its second smallest eigenvalue (i.e. the Fiedler vector), are both correlated with the community structure of the graph. Thus, spectral methods for community detection performs power iterations on these matrices to obtain the eigenvectors of interest and predicts the community structure based on them.
For example, to extract the Fiedler vector of a matrix M , whose eigenvector corresponding to the smallest eigenvalue is known to be v , one performing power iterations on ˜ M =
(cid:107) M (cid:107)
I − M by y ( n +1) =
˜
Mx ( n ) , x ( n +1) =
y ( n +1) − v T vy ( n +1) (cid:107) y
( n +1)
− v T vy ( n +1) (cid:107) .
If v is a constant vector, which is the case for L , then the normalization above is precisely performed within the spatial batch normalization step.
By incorporating a family of operators into the neural network framework, the GNN can not only approximate but also go beyond power iterations.
As explained in Section B.1, the Krylov subspace generated by the graph Laplacian (Defferrard et al., 2016) is not sufﬁcient to operate well in the sparse regime, as opposed to the space generated by { I, D, A } .
The expressive power of each layer is further increased by adding multiscale versions of A , although this beneﬁt comes at the cost of computational efﬁciency, especially in the sparse regime.
The network depth is chosen to be of the order of the graph diameter, so that all nodes obtain information from the entire graph.
In sparse graphs with small diameter, this architecture offers excellent scalability and computational complexity. Indeed, in many social networks diameters are constant (due to hubs), or log( | V | ) , as in the stochastic block model in the constant average degree regime (Riordan & Wormald, 2010). This results in a model with computational complexity on the order of | V | log( | V | ) , making it amenable to large-scale graphs. 4. LGNN: GNN ON LINE GRAPHS WITH THE NON - BACKTRACKING OPERATOR
For graphs with few cycles, posterior inference can be remarkably approximated by loopy belief propagation (Yedidia et al., 2003).
As described in Section B.2, the message-passing rules are deﬁned over the edge adjacency graph (see equation 57).
Although its second-order approximation around the critical point can be efﬁciently approximated with a power method over the original graph, a datadriven version of BP requires accounting for the non-backtracking structure of the message-passing.
In this section we describe an upgraded GNN model that exploits the non-backtracking structure.
The line graph L ( G ) =
( V L , E L ) is a graph representing the edge adjacency structure of G .
If G =
( V, E ) is an undirected graph, then the vertices V L of L ( G ) are the ordered edges in E , i.e., V L
= { ( i → j ) : ( i, j ) ∈ E } , and so | V L | = 2 | E | .
The non-backtracking operator on the line graph is represented by a matrix B ∈ R 2 | E |× 2 | E | deﬁned as
[Equation]
This operator enables the directed propagation of information through on the line graph and was ﬁrst proposed in the context of community detection on sparse graphs in Krzakala et al.
The message-passing rules of BP can be expressed as a diffusion in the line graph L ( G ) using this non-backtracking operator, with speciﬁc choices of activation function that turn product of beliefs into sums.
A natural extension of the GNN architecture presented in Section 4.1 is thus to consider a second
GNN deﬁned on L ( G ) , where B and D B = diag ( B 1 ) play the role of the adjacency and the degree matrices, respectively.
This effectively deﬁnes edge features that are updated according to the edge adjacency of G .
Edge and node features communicate at each layer using the edge indicator matrices
P m , P d ∈ { 0 , 1 } | V |× 2 | E | , deﬁned as P m i, ( i → j ) = 1 , P d j, ( i → j ) = 1 , P d i, ( i → j ) = 1 , P d j, ( i → j ) =
{+−and 
 otherwise.+}
With the skip linear connections deﬁned similarly, the resulting model becomes
[Equation]
where F A = { I, D, A, A (2) , . . .
, A ( J ) } , F B = { I B , D B , B, B (2) , . . .
, B ( J ) } , F AB = { P m , P d } , and the trainable parameters are θ i , θ (cid:48)
i , θ (cid:48)(cid:48) i ∈ R b
k {+× b k +
 and+} θ (cid:48)(cid:48)(cid:48) i ∈
R b k +1 ×
b k +1 .
We call such a model a Line Graph Neural Network (LGNN).
In our experiments, we set x (0) = deg( A ) and y (0) =
deg( B ) .
For graph families with constant average degree d (as | V | grows), the line graph has size 2 | E
| ∼ O ( d | V | ) , and is therefore feasible from the computational point of view.
Furthermore, the construction of line graphs can be iterated to generate L ( L ( G )) , L ( L ( L ( G ))) , etc. to yield a graph hierarchy , which could capture high-order interactions among nodes of G .
Such an hierarchical construction is related to other recent efforts to generalize GNNs (Kondor et al., 2018).
Relationship between LGNN and edge feature learning approaches Several authors have proposed combining node and edge feature learning (Gori et al., 2005; Gilmer et al., 2017; Velickovic et al., 2017), although we are not aware of works that considered the edge adjacency structure provided by the line graph and the non-backtracking operator.
The GNN on the line graph using the non-backtracking operator can be interpreted as learning directed edge features from an undirected graph.
Indeed, if each node i contains two distinct sets of features x
s ( i ) and x r ( i ) , the non-backtracking operator constructs edge features from node features while preserving orientation:
For an edge e =
( i, j ) , our model constructs oriented edge features f i → j = g ( x s ( i ) , x r ( j )) and f j → i = g ( x r ( i ) , x s ( j ))
(where g is trainable and not necessarily commutative on its arguments) that are subsequently propagated through the graph.
Constructing such local oriented structure is shown to signiﬁcantly improve performance in the next section.
Battaglia et al. (2016) introduced edge features over directed and typed graphs, but does not discuss the undirected case.
Kearnes et al. (2016); Gilmer et al. (2017) learn edge features on undirected graphs using f e = g ( x ( i ) , x ( j )) for an edge e
= ( i, j ) , where g is commutative on its arguments.
Finally, Velickovic et al. (2017) learns directed edge features on undirected graphs using stochastic matrices as adjacencies (which are either row or column-normalized).
For comparison, we also deﬁne a linear LGNN (LGNN-L) as the the LGNN that drops the nonlinear activation functions ρ in (2), and a symmetric LGNN (LGNN-S) as the LGNN whose line graph is deﬁned on the undirected edges of the original graph.
In LGNN-S, two edges of G are connected in the line graph if and only if they share one common node; also, F = { P } , with P ∈ R | V |×| E | deﬁned as P i, ( j → k )
= 1 if i = j or k and 0 otherwise.
4. A L OSS F UNCTION I NVARIANT U NER L ABEL PERMUTATION
Let C = { 1 , . .
, C } denote the set of all community labels, and consider ﬁrst the case where communities do not overlap.
By applying the softmax function at the end, we interpret the c th dimension of the output of the models at node i as the conditional probability that the node belongs to community c , o i,c
= p ( y i =
c | θ, G ) .
Let G =
( V, E ) be the input graph and let y i be the ground truth community label of node i .
Since the community structure is deﬁned up to global permutations of the labels, we can deﬁne a loss function with respect to a given graph instance as
[Equation]
where S C denotes the permutation group of C elements.
This is essentially taking the the cross entropy loss minimized over all possible permutations of the C labels.
In our experiments, we considered examples with small number of communities such as 2 and 5.
In general scenarios where C is much larger, the evaluation of the loss function (3) can be impractical due to the minimization over S C .
A possible solution is to randomly partition C labels into ˜ C groups, and then to marginalize the model outputs
o i,c , c ≤ C into ¯ o i, ¯ c
= (cid:80) c
∈ ¯
c o i,c , ¯ c ∈ ˜ C , and and ﬁnally use (cid:96) ( θ
[Equation] = inf [Equation]
˜ [Equation] i ∈ V log ¯ o i,π ( ¯ y i ) as an approximate loss value, which only involves a permutation group of size ( ˜ C !) .
Finally, if we are in a setup where nodes can belong to multiple communities, we can enlarge C to include subsets of communities as well, and deﬁne the permutation group accordingly.
For example,
If there are two overlapping communities, we let C
= { c 1 , c 2 , c { 1 , 2 } } , and only allow the permutation between c 1 and c 2 when computing the loss function.
E NERGY L ANDSCAPE OF L INEAR GNN OPTIMIZATION
As described in the numerical experiments, we found that the GNN models without nonlinear activations already provide substantial gains relative to baseline (non-trainable) algorithms.
This section studies the optimization landscape of linear GNNs.
Despite deﬁning a non-convex objective, we prove that the landscape is “benign” under certain further simpliﬁcations, in the sense that the local minima are conﬁned in sublevel sets of low energy.
For simplicity, we consider only the binary c = 2 case where we replace the node-wise binary cross-entropy loss by the squared cosine distance 3 , assume a single feature map ( b k = 1 for all k ), and focus on the GNN described in Section 4.1 (although our analysis carries equally to describe the line graph version; see remarks below).
We also make the simplifying assumption to replace the layer-wise spatial batch normalization by a simpler projection onto the unit (cid:96) 2 ball (thus we do not remove the mean).
Without loss of generality, assume that the input graph G has size n , and denote by F =
{ A 1 , . . .
, A Q } the family of graph operators appearing in (1).
Each layer thus applies an arbitrary polynomial (cid:80) Qq
=1 θ ( k ) q
A q to the incoming node feature vector x ( k ) .
Given an input node vector w ∈ R n , the network output can thus be written as
[Equation]
We highlight that this multilinear GNN setup is fundamentally different from the multilinear fullyconnected neural networks (that is, neural networks with linear activation function), whose landscape has been analyzed before (Kawaguchi, 2016).
First, the output is normalized in the sphere, which has important effects in the geometry.
Next, the network parametrization is intrinsic (the operators
O j depend on the input), which introduces ﬂuctuations in the landscape that we analyze.
In general, the operators in F are not commutative, but by considering the generalised Krylov subspace generated by powers of F , F K = { O 1 =
A K 1 , O
=
A 1 A K − 1, O
=
A 1 A 2 A K − 2, . . .
O Q K =
A KQ } , one can reparametrise (4) as e = (cid:80) Q K j =1 β j O j w with β ∈ R M , with M = Q K .
Given the target y ∈ R n , the loss incurred by each pair ( G, y ) becomes 1 − |(cid:104) e,y (cid:105)| 2 (cid:107) e (cid:107) 2 , and therefore the population loss, when expressed in terms of β , equals
[Equation]
n [Equation] z [Equation]
M [Equation] ( z n [Equation]
O [Equation] w, y (cid:105) and X n = U n U (cid:62) n ∈ R M × M , U n
[Equation]
The landscape is thus speciﬁed by a pair of random matrices Y n , X n ∈ R M × M .
The following theorem establishes that under appropriate assumptions, the concentration of certain random matrices around their mean controls the energy gaps between local and global maxima of L .
We deﬁne a “mean-ﬁeld“ loss function ˜
L n ( β ) = 1 − E X n ,Y n β T Y n β β T E X n β = 1 − β T E Y n β β T E X n β , and consider
L n as a perturbation of ˜ L n .
Assuming that E X n (cid:31) 0 , we write the Cholesky decomposition of E X n as E X n = R n R Tn , and deﬁne A n = R − 1 n Y n ( R − 1 n ) T , ¯ A n = E A n = R − 1 n E Y n ( R − 1 n ) T ,
B n = R − 1 n X n ( R − 1 n ) T , and ∆ B n = B n − I n .
Given a symmetric matrix K ∈ R M × M , we let λ 1 ( K ) , λ 2 ( K ) , ..., λ M ( K ) denote the eigenvalues of K in nondecreasing order.
Theorem 5.1.
For a given n , let η n = ( λ 1 ( ¯ A n ) − λ 2 ( ¯ A n )) − 1 , µ n = E
[ | λ 1 ( A n ) | 6 ] , ν n = E
[ | λ 1 ( B n ) | − 6 ] , δ n = E [ (cid:107) ∆ B n (cid:107) 6 ] , and assume that all four quantities are ﬁnite.
Then if β l ∈ S M − 1 is a local minimum of L n , and β g ∈ S M − 1 is a global minimum of L n , we have
L n ( β l ) ≤ (1 + (cid:15) η n ,µ n ,ν n ,δ n ) · L n ( β g ) , where (cid:15) η n ,µ n ,ν n ,δ n = O ( δ n ) for given η n , µ n , ν n as δ n → 0 and its formula is given in the appendix.
Corollary 5.2.
If ( η n ) n ∈ N ∗ , ( µ n ) n ∈ N ∗ , ( ν n ) n ∈ N ∗ are all bounded sequences, and lim n →∞ δ n = 0 , then ∀ (cid:15) > 0 , ∃ n (cid:15) such that ∀ n > n (cid:15) , | L n ( β l ) − L n ( β g ) | ≤ (cid:15) · L n ( β g ) .
The main strategy of the proof is to consider the actual loss function L n as a perturbation of ˜ L n , which has a landscape that is easier to analyze and does not have poor local minima, since it is equivalent to a quadratic form deﬁned over the sphere S M − 1 .
For a given graph inverse problem, applying this theorem thus requires estimating spectral ﬂuctuations of the pair X n , Y n , which in turn involve the spectrum of C ∗ algebras generated by the non-commutative family F .
For example, for stochastic block models, which we will discuss in the next section, it is an open problem how the bound behaves as a function of the parameters p and q .
That said, one should expect concentration to happen in general, since the dimension M is ﬁxed as n grows.
Another interesting question is to understand how the asymptotics of our landscape analysis relate to the hardness of estimation as a function of the signal-to-noise ratio.
Finally, another open question is to what extent our result could be extended to the non-linear residual GNN case, perhaps leveraging ideas from (Shamir, 2018).
E XPERIMENTS
We present experiments on community detection in synthetic datasets (Sections 6.1, 6.2 and Appendix
C.1) as well as real-world datasets (Section 6.3).
In the synthetic experiments, the performance is measured by the overlap between predicted ( ˆ y ) and true labels ( y ), which quantiﬁes how much better than random guessing a predicted labeling is, given by (cid:0) 1 n (cid:80) u δ y ( u ) , ˆ y ( u ) − 1 C (cid:1) / (1 − 1 C ) , where δ is the Kronecker delta function, and this quantity is maximized over global permutations of the labels for the graph.
We used Adamax (Kingma & Ba, 2014) with learning rate 0 . {+
+} across all experiments.
All the neural network models have 30 layers and 8 features in the middle layers (i.e., b k = 8 ) for experiments in Sections 6.1 and 6.2, and 20 layers and 6 features for Section 6.3.
GNNs and LGNNs have J = 2 across the experiments.
6. B INARY S TOCHASTIC B LOCK M ODEL
The stochastic block model is a random graph model with planted community structure.
In its simplest form, the graph consists of n = | V | nodes, which are partitioned into C communities, that is, each node is assigned a label y ∈ { 1 , ..., C } .
An edge connecting any two vertices u, v is drawn independently at random with probability p if y ( v ) =
y ( u ) , and with probability q otherwise.
In the binary case (i.e. C = 2 ), the sparse regime, where p, q (cid:39) 1 /n , is well understood and provides an initial platform to compare the GNN and LGNN with provably optimal recovery algorithms (Appendix B).
We consider two learning scenarios.
In the ﬁrst scenario, we choose different pairs of p and q , and train the models for each pair separately.
In particular, for each pair of ( p i , q i ) , we sample 6000 graphs under G ∼ SBM ( n = 1000 , p i , q i , C = 2) and then train the models for each i .
In the second scenario, reported in Appendix C.2, we train a single set of parameters θ from a set of 6000 graphs sampled from a mixture of SBM with different pairs of ( p i , q i ) , and average degree.
Importantly, his setup shows that our models are not simply approximating known algorithms such as
BP for particular SBM parameters, since the parameters vary in this dataset.
For the ﬁrst scenario, we chose ﬁve different pairs of ( p i , q i ) while ﬁxing p i
+ q
i , thereby corresponding to different signal-to-noise ratios (SNRs).
Figure 3 reports the performance of our models on the binary SBM model for the different SNRs, compared with baseline methods including BP, spectral methods using the normalized Laplacian and the Bethe Hessian as well as Graph Attention Networks (GAT) 5 from Velickovic et al.
We observe that both GNN and LGNN reach the performance of BP.
In addition, even the linear LGNN achieves a performance that is quite close to that of BP, in accordance to the spectral approximations of BP given by the Bethe Hessian (see supplementary), and signiﬁcantly outperforms performing 30 power iterations on the Bethe Hessian or the normalized Laplacian, as was done in the spectral methods.
We also notice that our models outperform GAT in this task.
We ran experiments in the dissociative case ( q > p ), as well as with C = 3 communities and obtained similar results, which are not reported here. [Table]
6. C OMPUTATIONAL - TO -S TATISTICAL T HRESHOLDS IN THE SBM
In SBM with fewer than 4 communities, it is known that BP provably reaches the information-theoretic threshold (Abbe, 2017; Massoulié, 2014; Coja-Oghlan et al., 2016).
The situation is different for k > 4 , where it is conjectured that a gap emerges between the theoretical performance of MLE estimators and the performance of any polynomial-time estimation procedure (Decelle et al., 2011).
In this context, one can use the GNN models to search the space of the generalizations of BP, and attempt to improve upon the detection performance of BP for scenarios where the SNR falls within the computational-to-statistical gap.
Table 1 presents results for the 5 -community dissociative SBM, with n = 400 , p = 0 and q = 18 /n .
The SNR in this setup is above the information-theoretic threshold but below the asymptotic threshold above which BP is able to detect (Decelle et al., 2011).
Note that since p = 0 , this also amounts to a graph coloring problem.
We see that the GNN and LGNN models outperform BP in this experiment, indeed opening up the possibility to reduce the computation-information gap.
That said, our model may taking advantage of ﬁnite-size effects, which will vanish as n → ∞ .
The asymptotic study of these gains is left for future work.
In terms of average test accuracy, LGNN has the best performance.
In particular, it outperforms the symmetric version of LGNN, emphasizing the importance of the non-backtracking matrix used in LGNN.
Further ablation studies are included in Appendix C.
6. R EAL D ATASETS FROM SNAP
We now compare the models on the SNAP datasets, whose domains range from social networks to hierarchical co-purchasing networks.
We obtain the training set as follows.
For each SNAP dataset, we start by focusing only on the 5000 top quality communities provided by the dataset.
We then identify edges ( i, j ) that cross at least two different communities.
For each of such edges, we consider pairs of communities C 1 , C 2 such that i / ∈ C 2 and j / ∈ C 1 , i ∈ C 1 , j ∈ C 2 , and extract the subset of nodes determined by C 1 ∪ C 2 together with the edges among them. The resulting graph is connected since each community is connected.
Finally, we divide the dataset into training and testing sets by enforcing that no community belongs to both the training and the testing set.
In our experiment, due to computational limitations, we restrict our attention to the three smallest datasets in the SNAP collection (Youtube, DBLP and Amazon), and we restrict the largest community size to 200 nodes, which is a conservative bound.
We compare the performance of GNN and LGNN models with GAT as well as the CommunityAfﬁliation Graph Model (AGM), which is a generative model proposed in Yang & Leskovec (2012b) that captures the overlapping structure of real-world networks.
Community detection can be achieved by ﬁtting AGM to a given network, which was shown to outperform some state-of-the-art algorithms.
Table 4 compares the performance, measured with a 3-class ( C = {{ 1 } , { 2 } , { 1 , 2 }} ) classiﬁcation accuracy up to global permutation 1 ↔ 2 .
GNN, LGNN, LGNN-S and GAT yield similar results and outperform AGMﬁt.
It further illustrates the beneﬁts of data-driven models that strike the right balance between expressivity and structural design
[Table]
C ONCLUSION
In this work, we have studied data-driven approaches to supervised community detection with graph neural networks.
Our models achieve comparable performance to BP in binary SBM for various SNRs, and outperform BP in the sparse regime of 5-class SBM that falls between the computationalto-statistical gap.
This is made possible by considering a family of graph operators including the power graph adjacency matrices, and importantly by introducing the line graph equipped with the non-backtracking matrix.
We also provided a theoretical analysis of the optimization landscapes of simpliﬁed linear GNN for community detection and showed the gap between the loss value at local and global minima are bounded by quantities related to the concentration of certain random matricies.
One word of caution is that our empirical results are inherently non-asymptotic.
Whereas models trained for given graph sizes can be used for inference on arbitrarily sized graphs (owing to the parameter sharing of GNNs), further work is needed in order to understand the generalization properties as | V | increases.
Nevertheless, we believe our work opens up interesting questions, namely better understanding how our results on the energy landscape depend upon speciﬁc signal-to-noise ratios, or whether the network parameters can be interpreted mathematically.
This could be useful in the study of computational-to-statistical gaps, where our model can be used to inquire about the form of computationally tractable approximations.
Another current limitation of our model is that it presumes a ﬁxed number of communities to be detected.
Other directions of future research include the extension to the case where the number of communities is unknown and varied, or even increasing with | V | , as well as applications to ranking and edge-cut problems.
A P ROOF OF T HEOREM 5.
For simplicity and with an abuse of notation, in the remaining part we redeﬁne L and ˜ L in the following way, to be the negative of their original deﬁnition in the main section: L n ( β ) =
E X n ,Y [Equation] ˜ L n ( β ) =
E X n ,Y n β T Y n β β T E X n β .
Thus, minimizing the loss function (5) is equivalent to maximizing the function L n ( β ) redeﬁned here.
We write the Cholesky decomposition of E X n as E X n = R n R Tn , and deﬁne A n = R − 1 n Y n ( R − 1 n ) T , ¯
A n = E A n = R − 1 n E Y n ( R − 1 n ) T , B n = R − 1 n X n ( R − 1 n ) T , and ∆ B n = B n − I n .
Given a symmetric matrix K ∈ R M × M , we let λ 1 ( K ) , λ 2 ( K ) , ..., λ M ( K ) denote the eigenvalues of K in nondecreasing order.
First, we have
[Equation]
Let us denote by ˜
β g a global minimum of the mean-ﬁeld loss ˜
L n .
Taking a step further, we can extend this bound to the following one (the difference is in the second term on the right hand side):
Lemma A.1.
[Equation]
Proof of Lemma A.1.
We consider two separate cases: The ﬁrst case is when ˜ L n ( β l ) ≥ ˜ L n ( β g ) .
Then ˜ L n ( β l ) − ˜ L n (˜ β g ) ≥ ˜ L n ( β l ) − ˜ L n ( β g ) ≥ 0 , and so | L n ( β l ) − L n ( β g ) | ≤ | L n ( β l ) − ˜ L n ( β l
[Equation]
| + | ˜ L n ( β l ) − ˜ L n (˜ β g ) | + | ˜
L n ( β g ) − L n ( β g ) | .
The other case is when ˜ L n ( β l ) <
˜ L n ( β g ) .
Note that L n ( β l ) ≥ L n ( β g ) .
Then | L n ( β l ) − L n ( β g ) | ≤ | L n ( β l ) − ˜ L n ( β l ) | + | ˜ L n ( β g ) − L n ( β g ) | ≤ | L n ( β l ) − ˜ L n ( β l ) | + | ˜
L n ( β l ) − ˜ L n (˜ [Equation] g ) | + | ˜
L n ( β g ) −
L n ( β g ) | .
Hence, to bound the "energy gap" | L n ( β l ) − L n ( β g ) | , if sufﬁces to bound the three terms on the right hand side of Lemma A.1 separately.
First, we consider the second term, | ˜
L n ( β l ) − ˜ L n (˜ β g ) | .
Let γ l = R T n β l , γ g = R T n β g and ˜ γ g = R T n ˜ β g .
Deﬁne S n ( γ )
=
L n ( R − T n γ ) and ˜ S n ( γ ) =
˜ L n ( R − Tn γ ) , for any γ ∈ R M .
Thus, we apply a change-of-variable and try to bound | ˜ S n ( γ l ) − ˜ S n (˜ γ g ) | .
Since β l is a local maximum of L n , λ 1 ( ∇ 2 L n ( β l )) ≤ 0 .
Since ∇ 2 S n ( γ l )
=
R − 1 n ∇ 2 L n ( β l ) R − Tn , where R n is invertible, we know that λ 1 ( ∇ 2 S n ( γ l )) ≤ 0 , thanks to the following lemma:
Lemma A.2.
If R, Q ∈ R M × M , R is invertible, Q is symmetric and λ q > 0 is an eigenvalue of Q , then λ 1 ( RQR T ) ≥ λ · λ M ( RR T )
Proof of Lemma A.2.
Say Qw = λw for some vector w ∈ R M .
Let v =
R − T w .
Then v T ( RQR T ) v = w T Qw = λ [Equation] w (cid:107) 2 .
Note that (cid:107) w (cid:107) 2 = v T RR T v ≥ (cid:107) v (cid:107) 2 λ M ( RR T ) .
Hence
[Equation]
Since ∇ 2 S n ( γ l ) = ∇ 2 ˜ S n ( γ l ) + ( ∇ 2 S n ( γ l ) − ∇ 2 ˜ S n ( γ l )) , there is 0 ≥ λ 1 ( ∇ 2 S n ( γ l )) ≥ λ 1 ( ∇ 2 ˜ S n ( γ l )) − (cid:107)∇ 2 S n ( γ l ) − ∇ 2 ˜ S n ( γ l ) (cid:107) .
Hence,
[Equation]
[Equation] Next, we relate the left hand side of the inequality above to cos( γ l , ˜ γ g ) , thereby obtaining an upper bound on [1 − cos 2 ( γ l , ˜ γ g )] , which will then be used to bound | ˜ S n ( γ l ) − ˜ S n (˜ γ g ) | .
Lemma A.3.
∀ γ ∈ R d ,
[Equation]
Proof of Lemma A.3.
[Equation]
Thus, if we deﬁne Q 1 = ( γ T γ )
[( γ T γ ) ¯ A n − ( γ T ¯ A n γ ) I ] , Q 2 = 4[( γ T γ )
¯ A n − ( γ T ¯ A n γ )
I ] γγ T , we have
[Equation]
To bound λ 1 ( ∇ 2 ˜ S n ( γ )) , we bound λ 1 ( Q 1 ) and (cid:107) Q 2 (cid:107) as follows:
Since ¯ A n is symmetric, let ˆ γ 1 , . . .
ˆ γ M be the orthonormal eigenvectors of ¯ A n corresponding to nonincreasing eigenvalues l 1 , . . .
l M .
Note that the global minimum satisﬁes ˜ γ g = ±
ˆ γ 1 .
Write γ = (cid:80) {+Mi =
+}
α i ˆ γ
i , and let ¯ α i
= α i √ (cid:80) Mi =
α 2 i .
Then | cos( γ, ˜ γ g ) | = | cos
[Equation]
=
[Equation]
Then,
[Equation]
To bound (cid:107) Q 2 (cid:107) :
[( γ T γ ) ¯ A n − ( γ T ¯ A n γ ) I ]
[Equation]
=
[Equation]
(cid:88)
[Equation]
(cid:34)
[Equation]
(cid:88)
[Equation]
=
[Equation]
(cid:88)
[Equation]
(cid:35)
[Equation]
Note that given vectors v, w ∈ R M , (cid:107)
[Equation]
[Equation]
|
[Equation]
w |
Therefore, (cid:107) Q
[Equation]
(cid:107) =4 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:32)
M (cid:88) k
=
[Equation]
Thus,
[Equation]
This yields the desired lemma.
Combining inequality 8 and Lemma A.3, we get
[Equation]
Thus, to bound the angle between γ l and ˜ γ g , we can aim to bound (cid:107)∇ ˜ S n ( γ l ) (cid:107) and (cid:107)∇ 2 S n ( γ l ) − ∇ 2 ˜ S n ( γ l ) (cid:107) as functions of the quantities µ n , ν n and δ n .
Lemma A.4.
[Equation]
Proof of Lemma A.4.
[Equation]
Combining equations 17 and 18, we get
[Equation]
Since ∇ S n ( γ l ) =
[Equation]
we have
[Equation]
Then, by the generalized Hölder’s inequality,
[Equation]
[Equation] Hence, written in terms of the quantities µ n , ν n and δ n , we have
[Equation]
Lemma A.5.
With δ n = ( E (cid:107) ∆ B n (cid:107) 6 ) 16 , E | λ 1 ( B n ) | 6 ≤ 64 + 63 δ 6 n
Proof of Lemma A.5.
[Equation]
[Equation]
Note that gma
[Equation]
[Equation]
and for k ∈ { 1 , 2 , 3 , 4 , 5 } , if X is a nonnegative random variable,
[Equation]
E X(25) Therefore, E | λ 1 ( B n ) | 6 ≤ 64 + 63 E (cid:107) ∆ B n (cid:107) 6 .
From now on, for simplicity, we introduce δ (cid:48) n = (64 + 63 δ 6 n ) 16 , as a function of δ n .
Lemma A.6.
∀ γ ∈ R M ,
[Equation]
Proof of Lemma A.6.
[Equation]
where
[Equation]
Thus, (cid:107)∇ 2 S n ( γ ) − ∇ 2 ˜ S n ( γ ) (cid:107) ≤ 2 E (cid:107)
[Equation]
E
[Equation]
H [Equation] , and we try to bound each term on the right hand side separately [Equation]
For the ﬁrst term
[Equation]
there is
[Equation]
[Equation]
Applying generalized Hölder’s inequality, we obtain
[Equation]
For the second term, there is
[Equation]
Hence,
[Equation]
Applying generalized Hölder’s inequality
[Equation]
we obtain
[Equation] [Equation]
For H 3 , note that
[Equation]
Hence,
[Equation]
Thus,
[Equation]
Applying generalized Hölder’s inequality
[Equation]
we obtain
[Equation]
For the last term
[Equation]
Thus
[Equation]
Applying generalized Hölder’s inequality
[Equation]
we obtain
[Equation]
Therefore
[Equation]
summing up the bounds above
[Equation]
we obtain
[Equation]
Hence
[Equation]
combining inequality
[Equation]
Lemma A.4 and Lemma A
[Equation] [Equation]
we get
[Equation]
[Equation]
For simplicity, we deﬁne C
[Equation]
n ,
[Equation]
Thus,
[Equation]
Following the notations in the proof of Lemma A.3, we write γ l = (cid:80) Mi
=
α i ˆ γ i .
Note that ˜ γ g = ± ˆ γ 1 , and | cos( γ, ˆ γ i ) | =
| ¯ α i | .
Thus,
[Equation]
Since Y n is positive semideﬁnite, E Y n is also positive semideﬁnite, and hence ¯ A n = R Tn E Y n ( R − 1 n ) T is positive semideﬁnite as well.
This means that l i ≥ 0 , ∀ i ∈ { 1 , ..., M } .
Since ˜ L n ( ˜ β g ) =
˜ S n (˜ γ g ) = ˜ S n (ˆ γ 1 ) =
l 1 , there is
[Equation]
Next, we bound the ﬁrst and the third term on the right hand side of the inequality in Lemma A.1.
Lemma A.7.
∀ β ,
[Equation]
Proof of Lemma A.7.
Let γ =
T Tn β .
[Equation] [Equation]
Thus, we get the desired lemma by the generalized Hölder’s inequality.
Combining inequality 46, inequality 48 and Lemma A.7, we get | L n
[Equation] [Equation]
Meanwhile,
[Equation]
≤
[Equation]
n
[Equation] [Equation]
Hence,
[Equation]
, or
[Equation]
Therefore,
[Equation]
Hence
[Equation]
we have proved the theorem
[Equation]
with (cid:15)
[Equation]
B B ACKGROUND B. G RAPH M IN -C UTS AND S PECTRAL C LUSTERING
We consider graphs G =
( V, E ) ,
modeling a system of N = | V | elements presumed to exhibit some form of community structure.
The adjacency matrix A associated with G is the N × N binary matrix such that A i,j = 1 whenever ( i, j ) ∈ E .
We assume for simplicity undirected graphs, yielding symmetric adjacency matrices.
The community structure is encoded in a discrete label vector s :
V → { 1 , K } that assigns a community label to each node, and the goal is to estimate s from observing the adjacency matrix.
In the setting of binary, associative communities, where s ( i ) = ± 1 , two nodes i, j with s ( i ) =
s ( j ) are more likely to be connected ( A i,j = 1 ) than two nodes i, j with s ( i ) (cid:54) =
s ( j ) .
Thus a quantity of the form
[Equation]
measures the cost associated with cutting the graph between communities encoded by s that we wish to minimize under appropriate constraints Newman (2006).
Note that (cid:80) i,j A
i,j = s T Ds , with
D = diag ( A 1 ) (called the degree matrix), so the cut cost can be expressed as a positive semideﬁnite quadratic form
[Equation]
that we wish to minimize.
This shows a fundamental connection between the community structure and the spectrum of certain linear operators of the graph, which provides a powerful and stable relaxation of the discrete combinatorial optimization problem of estimating the community labels for each node.
In the case of the graph Laplacian ∆ = D − A , its eigenvector associated with the smallest eigenvalue is trivial, but its Fiedler vector (the eigenvector associated with the second smallest eigenvalue) reveals important community information of the graph Newman (2006) under appropriate conditions, and is associated with the graph conductance Spielman (2015) under certain normalization schemes.
For a given linear operator L ( A ) extracted from the graph (that we assume symmetric), we are thus interested in extracting eigenvectors at the edge of its spectrum.
A particularly simple algorithmic framework is given by the power iteration method.
Indeed, the Fiedler vector of L ( A ) can be obtained by ﬁrst extracting the leading eigenvector v of ˜
A =
(cid:107)L ( A ) (cid:107)
I − L ( A ) , and then iterating
[Equation]
[Table]
Unrolling power iterations and recasting the resulting model as a trainable neural network is akin to the LISTA Gregor & LeCun (2010) sparse coding model, which unrolled iterative proximal splitting algorithms.
Despite the appeal of graph Laplacian spectral approaches, it is well known Krzakala et al.
that these methods fail in sparsely connected graphs.
Indeed, in such scenarios, the eigenvectors of graph Laplacians concentrate on nodes with dominant degree, losing their ability to correlate with community structure.
In order to overcome this important limitation, authors have resorted to ideas inspired from statistical physics, as explained next.
B. P ROBABILISTIC G RAPHICAL M ODELS AND B ELIEF -P ROPAGATION
Graphs with labels on nodes and edges can be cast as a graphical model where the aim of clustering is to optimize label agreement.
This can be seen as a posterior inference task.
If we simply assume the graphical model is a Markov Random Field (MRF) with trivial compatibility functions for cliques greater than 2, the probability of a label conﬁguration σ is given by
[Equation]
Generally, computing marginals of multivariate discrete distributions is exponentially hard.
For instance, in the case of P ( σ i ) we are summing over | X | n − 1 terms (where X is the state space of discrete variables).
But if the graph is a tree, we can factorize the MRF more efﬁciently to compute the marginals in linear time via a dynamic programming method called the sum-product algorithm, also known as belief propagation (BP).
An iteration of BP is given by
[Equation]
The beliefs ( b i → j ( σ i ) ) are interpreted as the marginal distributions of σ i .
Fixed points of BP can be used to recover marginals of the MRF above.
In the case of the tree, the correspondence is exact: P i ( σ i ) =
b i ( σ i ) .
Certain sparse graphs, like SBM with constant degree (Mossel et al., 2014), are locally similar to trees for such an approximation to be successful.
BP approximates the MLE solutions but convergence is not guaranteed in graphs that are not trees.
Furthermore, in order to apply BP, we need a generative model and the correct parameters of the model.
If unknown, the parameters can be derived using expectation maximization, further adding complexity and instability to the method since iterations may learn parameters for which BP does not converge.
B. N ON - BACKTRACKING OPERATOR AND B ETHE H ESSIAN
The BP equations have a trivial ﬁxed-point where every node takes equal probability in each group.
Linearizing the BP equation around this point is equivalent to spectral clustering using the nonbacktracking matrix (NB), a matrix deﬁned on the edges of the graph that indicates whether two edges are adjacent and do not coincide.
Spectral clustering using NB gives signiﬁcant improvements over spectral clustering with versions of the Laplacians (L) and the adjacency matrix (A); High degree ﬂuctuations drown out the signal of the informative eigenvalues in the case of A and L, whereas
NB’s eigenvalues are conﬁned to a disk in the complex plane, so its eigenvalues corresponding to community structure lay outside the disk and are easily distinguishable.
NB matrices are still not optimal in that they are matrices on the edge set, and are not symmetric (so cannot enjoy tools of numerical linear algebra for symmetric matrices).
Recently Saade et al.
(2014) showed that a spectral method can do as well as BP in this regime, using the Bethe Hessian operator given by BH ( r ) :
= ( r 2 − 1)
I − rA + D
(where r is a scalar value).
This is due to a one-to-one correspondence between the ﬁxed points of BP and the stationary points of the Bethe free energy (corresponding Gibbs energy of the Bethe approximation)
Saade et al.
The Bethe Hessian is a scaling of the Hessian of the Bethe free energy at an extrema corresponding to the trivial ﬁxed point of BP.
Negative eigenvalues of BH ( r ) correspond to phase transitions in the Ising model where new clusters become identiﬁable.
This all gives theoretical motivation for why [ I , D, A ] deﬁned in
Section 3 are a good family of generators to do spectral clustering on.
In the case of the SBM, they generate the Bethe Hessian which can achieve community detection down to the information theoretic threshold.
The GNN is capable of expressing spectral approximations of complicated functions of [ I , D, A ] , and performing nonlinear power method iterations in order to infer global structure (for instance community structure).
Furthermore, unlike belief propagation, the method does not require a generative model, oftentimes requires a lot statistical analysis to motivate and is exposed to model misspeciﬁcations when deployed on real data.
Instead, our framework ﬁnds structure in a data driven way, learning it from the available training data.
B. S TOCHASTIC B LOCK M ODEL
We brieﬂy review the main properties needed in our analysis, and refer the interested reader to Abbe (2017) for an excellent recent review.
The Stochastic Blockmodel (SBM) is a random graph model denoted by SBM ( n, p, q, C ) .
Implicitly there is an F : V → { 1 , C } associated with each SBM graph, which assigns community labels to each vertex.
One obtains a graph from this generative model by starting with n vertices and connecting any two vertices u, v independently at random with probability p if F ( v ) =
F ( u ) , and with probability q if F ( v ) (cid:54)
=
F ( u ) .
We say the SBM is balanced if the communities are the same size.
Let ¯ F n : V → { 1 , C } be our predicted community labels for
SBM ( n, p, q, C ) , F n ’s give exact recovery on a sequence { SBM ( n, p, q ) } n if P ( F n = ¯ F n ) → n 1 , and give detection ∃ (cid:15) > 0 : P ( | F n − ¯ F n | ≥ 1 /k + (cid:15) )
→ n 1 (i.e ¯ F n ’s do better than random guessing).
It is harder to tell communities apart if p is close to q (if p =
q
we just get an Erd˝os Renyi random graph, which has no communities).
In the two community case, It was shown that exact recovery is possible on SBM ( n, p = a log n n , q = b log n n ) if and only if a + b 2 ≥ 1 + √ ab Mossel et al.
Abbe et al.
For exact recovery to be possible, p, q must grow at least O (log n ) or else the sequence of graphs will not to be connected, and thus vertex labels will be underdetermined.
There is no information-computation gap in this regime, so there are polynomial time algorithms when recovery is possible Abbe (2017); Mossel et al.
In the much sparser regime of constant degree SBM ( n, p = an , q = bn ), detection is the best we hope for.
The constant degree regime is also of most interest to us for real world applications, as most large datasets have bounded degree and are extremely sparse.
It is also a very challenging regime; spectral approaches using the Laplacian in its various (un)normalized forms and the adjacency matrix, as well as SDP methods cannot detect communities in this regime Abbe (2017) due to large ﬂuctuations in the degree distribution that prevent eigenvectors form concentrating on the clusters.
Decelle et al.
ﬁrst proposed the BP algorithm on the SBM, which was proven to yield Bayesian optimal values in Coja-Oghlan et al.
In the constant degree regime with balanced k communities, the Kesten-Stigum threshold is given by
SNR :=
( a − b ) 2 / ( k ( a + ( k + 1) b ))
Abbe (2017).
It has been shown for k = 2 that SNR = 1 is both the information theoretic and efﬁcient computational threshold where belief propagation (BP) via a polynomial time algorithm.
For k > 4 a gap emerges between the information theoretic threshold and computational one.
It’s conjectured that no polynomial time algorithm exist for SNR < 1 , while a BP algorithm works for SNR > 1 Abbe (2017); Decelle et al.
The existence of the gap was shown by Abbe (2017) by proving a non-polynomial algorithm can do detection for some SNR < 1 .
C F URTHER EXPERIMENTS
[Equation]
[Table]
[Equation]
[Table] [Table]
The success of belief propagation on the SBM relies on its locally hyperbolic properties, which make it treelike with high probability.
This behavior is completely different if one considers random graphs with locally
Euclidean geometry.
The Geometric
Block Model (Sankararaman & Baccelli, 2018) is a random graph generated as follows.
We start by sampling n points x 1 , . . .
, x n i.i.d.
from a Gaussian mixture model given by means µ 1 , . . .
µ k ∈ R d at distances S apart and identity covariances.
The label of each sampled point corresponds to which Gaussian it belongs to.
We then draw an edge between two nodes i, j if (cid:107) x i − x j (cid:107) ≤ T/ √ n .
Due to the triangle inequality, the model contains a large number of short cycles, which affects the performance of loopy belief propagation.
This motivates other estimation algorithms based on motif-counting that require knowledge of the model likelihood function (Sankararaman & Baccelli, 2018).
Table 3 shows the performance of GNN and LGNN on the binary GBM model, obtained with d = 2 , n = 500 , T = 5 √ 2 and varying S , as well as the performances of two spectral methods, using respectively the normalized Laplacian and the Bethe Hessian, which approximates BP around its stationary solution.
We note that LGNN model, thanks to its added ﬂexibility and the multiscale nature of its generators, is able to signiﬁcantly outperform both spectral methods as well as the baseline GNN.
C. M IXTURE OF BINARY SBM
We report here our experiments on the SBM mixture, generated with
[Equation]
where the average degree ¯ d is either ﬁxed constant or also randomized with ¯ d ∼ Unif (1 , t ) .
Figureshows the overlap obtained by our model compared with several baselines.
Our GNN model is either competitive with BH or outperforms BH, which achieves the state of the art along with BP Saade et al.
(2014), despite not having any access to the underlying generative model (especially in cases where
GNN was trained on a mixture of SBM and thus must be able to generalize the r parameter in BH).
They all outperform by a wide margin spectral clustering methods using the symmetric Laplacian and power method applied to (cid:107) BH (cid:107) I − BH using the same number of layers as our model.
Thus
GNN’s ability to predict labels goes beyond approximating spectral decomposition via learning the optimal r for BH ( r ) .
The model architecture could allow it to learn a higher dimensional function of the optimal perturbation of the multiscale adjacency basis, as well as nonlinear power iterations, that amplify the informative signals in the spectrum.
C. A BLATION STUDIES OF GNN AND LGNN
Compared to f , each of h , i and k has one fewer operator in F , and j has two fewer.
We see that with the absence of A (2) , k has much worse performance than the other four, indicating the importance of the power graph adjacency matrices.
Interestingly, with the absence of I , i actually has better average accuracy than f .
One possibly explanation is that in SBM, each node has the same expected degree, and hence I may be not very far from D , which makes having both I and D in the family redundant to some extent.
Comparing GNN models a , b and c , we see it is not the case that having larger J will always lead to better performance.
Compared to f , GNN models c , d and e have similar numbers of parameters but all achieve worse average test accuracy, indicating that the line graph structure is essential for the good performance of LGNN in this experiment.
In addition, l also performs worse than f , indicating the signiﬁcance of the non-backtracking line graph as opposed to the symmetric line graph.
[Table]