S UPERVISED
C OMMUNITY
D ETECTION WITH L INE
G RAPH N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study data-driven methods for community detection on graphs, an inverse problem that is typically solved using the spectrum of certain operators or via posterior inference under certain probabilistic graphical models.
Focusing on random graph families such as the stochastic block model, recent research has uniﬁed both approaches and identiﬁed both statistical and computational detection thresholds in terms of the signal-to-noise ratio.
Community detection can be recast as a node-wise graph classiﬁcation problem and therefore studied from a learning perspective.
We present a novel family of Graph Neural Networks (GNNs) for solving community detection problems in a supervised learning setting.
We show that, in a data-driven manner and without access to the underlying generative models, they can match or even surpass the performance of the belief propagation algorithm on binary and multiclass stochastic block models, which is known to reach the computational threshold in these cases.
In particular, we propose to augment GNNs with the non-backtracking operator deﬁned on the line graph of edge adjacencies.
The GNNs are also tested on real datasets, yielding better performance than other published models.
In addition, we perform the ﬁrst analysis of the optimization landscape of using GNNs to solve community detection problems, demonstrating that under certain simpliﬁcations and assumptions, the loss value at any local minimum is close to the loss value at the global minimum/minima.
I NTRODUCTION
Graph inference problems encompass a large class of tasks and domains, from posterior inference in probabilistic graphical models to community detection and ranking in generic networks, image segmentation or compressed sensing on non-Euclidean domains.
They are motivated both by practical applications, such as in the case of PageRank, and also by fundamental questions on the algorithmic hardness of solving such tasks.
From a data-driven perspective, these problems can be formulated in unsupervised, semi-supervised or supervised learning settings.
In the supervised case, one assumes a dataset of graphs with labels on its nodes and/or edges, and attempts to perform node/edge classiﬁcation by optimizing a loss over a certain parametric class, e.g. neural networks.
Graph Neural Networks ((Gori et al., 2005), (Bronstein et al., 2017) and references therein) are natural extensions of Convolutional Neural Networks to graph-structured data, and have emerged as a powerful class of algorithms to perform complex graph inference leveraging labeled data.
In essence, these neural networks learn cascaded linear combinations of intrinsic graph operators interleaved with node-wise (or edge-wise) activation functions.
Since they learn from intrinsic graph operators, they can be applied to varying input graphs, and they offer the same parameter sharing advantages as their CNN counterparts.
In this work, we focus on community detection problems, a wide class of node classiﬁcation tasks that attempt to discover a clustered, segmented structure within a graph.
The algorithmic approaches to this problem include a rich class of spectral methods, which take advantage of the spectrum of certain operators deﬁned on the graph, as well as approximate message-passing methods such as belief propagation (BP), which performs approximate posterior inference under predeﬁned graphical models (Decelle et al., 2011).
Focusing on the supervised setting, we study the ability of GNNs to approximate, generalize or even improve upon these class of algorithms.
Our motivation is two-fold.
On the one hand, this problem exhibits algorithmic hardness on some settings, opening up the possibility to discover more efﬁcient algorithms than the current ones.
On the other hand, many practical scenarios fall beyond pre-speciﬁed probabilistic models, requiring data-driven solutions.
We propose modiﬁcations to the GNN architecture, which allows it to exploit edge adjacency information, by incorporating the non-backtracking operator of the graph.
This operator was introduced to community detection problems by (Krzakala et al., 2013), which proposed a spectral method based on the non-backtracking operator.
This operator is deﬁned over the edges of the graph and allows a directed ﬂow of information even when the original graph is undirected.
We refer to the resulting model as a Line Graph Neural Network (LGNN) .
Focusing on important random graph families exhibiting community structure, such as the stochastic block model (SBM) and the geometric block model (GBM), we demonstrate improvements in the performance by LGNN, even in regimes within the so-called computational-to-statistical gap.
A perhaps surprising aspect is that these gains can be obtained even with linear GNNs, which become parametric versions of power iteration algorithms.
We want to mention that besides community detection tasks, GNN and LGNN can be applied to other node-wise classiﬁcation problems too.
The reason we are focusing on community detection problems is that this is a relatively well-studied setup, for which different algorithms have been proposed and where computational and statistical thresholds have been studied in several scenarios.
Moreover, synthetic datasets can be easily generated for community detection tasks.
Therefore, we think it is a nice setup for comparing different algorithms, besides its practical values.
The good performances of GNN and LGNN motivate our second main contribution: the analysis of the optimization landscape of simpliﬁed and linear GNN models when trained with planted solutions of a given graph distribution.
Under reparametrization, we provide an upper bound on the energy gap controlling the energy difference between local and global minima (or minimum).
With some assumptions on the spectral concentration of certain random matrices, this energy gap will shrink as the size of the input graphs increases, which would mean that the optimization landscape is benign on large enough graphs.
Summary
of Main Contributions:
• We propose an extension of GNNs that operate on the line graph using the non-backtracking operator, which yields improvements on hard community detection regimes.
• We show that on the stochastic block model we reach detection thresholds in a purely data-driven fashion, in the sense that our results improve upon belief propagation in hard SBM detection regimes, as well as in the geometric block model.
• We perform the ﬁrst analysis of the learning landscape of GNN models, showing that under certain simpliﬁcations and assumptions, they exhibit a form of “energy gap”, where local mimima are conﬁned in low-energy conﬁgurations.
• We show how our model can be applied to real-world datasets, leading to improved community detection results.
P ROBLEM S ETUP
We are interested in a speciﬁc class of node-classiﬁcation tasks in which given an input graph
G =
( V, E ) , a signal y :
V → { 1 , C } encoding a partition of V into C groups is to be predicted at each node.
We assume that a training set { ( G t , y t ) } t ≤ T is given, which we use to learn a model ˆ y =
Φ( G, θ ) trained by minimising
[Equation]
Since y encodes a partition of C groups, the speciﬁc label of each node is only important up to a global permutation of { 1 , C } .
Section 4.3 describes how to construct losses (cid:96) ( a, b ) with such a property.
A permutation of the observed nodes translates into the same permutation applied to the labels, which justiﬁes models Φ that are equivariant to permutations.
Also, we are interested in inferring properties of community detection algorithms that do not depend on the speciﬁc size of the graphs 1 .
We therefore require that the model Φ accepts graphs of variable size for the same set of C is parameters, similarly as in sequential RNN or spatial CNN models.
In our study of random graph models (SBM and GBM), we construct a training set of planted solutions.
Labels y
i are generated by sampling a balanced partition uniformly at random, and then we produce the input graphs G i by sampling G i | y
i according to each random graph model.
R ELATED W ORK
GNN was ﬁrst proposed in (Gori et al., 2005; Scarselli et al., 2009).
(Bruna et al., 2013) generalized convolutional neural networks on general undirected graphs by using the graph Laplacian’s eigenbasis.
This was the ﬁrst time the Laplacian operator was used in a neural network architecture to perform classiﬁcation on graph inputs.
(Defferrard et al., 2016) considers a symmetric Laplacian generator to deﬁne a multiscale GNN architecture, demonstrated on classiﬁcation tasks.
Similarly, (Kipf &
Welling, 2016) uses a similar generator as effective embedding mechanisms for graph signals and applies it to semi-supervised tasks.
This is the closest application of GNNs to our current contribution.
However, we highlight that semi-supervised learning requires bootstrapping the estimation with a subset of labeled nodes, and is mainly interested in generalization within a single, ﬁxed graph.
In comparison, our setup considers community detection across a distribution of input graphs and assumes no initial labeling on a given test-set input graph.
There have been several extensions of GNNs (Li et al., 2015; Sukhbaatar et al., 2016; Duvenaud et al., 2015; Niepert et al., 2016) by modifying their non-linear activation functions, their parameter sharing strategies, and their choice of graph operators.
In particular, (Gilmer et al., 2017) interpreted the
GNN architecture as learning an approximate message-passing algorithm, which extends the learning of hidden representations to graph edges in addition to graph nodes.
Recently, (Velickovic et al., 2017) relates adjacency learning with attention mechanisms, and (Vaswani et al., 2017) proposes a similar architecture in the context of machine translation.
Another recent and related piece of work is (Kondor et al., 2018), which proposes a generalization of GNN that captures high-order node interactions through covariant tensor algebra.
Our approach to extend the expressive power of GNN using the line graph may be seen as an alternative to capture such high-order interactions.
Our energy landscape analysis is related to the recent paper (Shamir, 2018), which establishes an energy bound on the local minima arising in the optimization of ResNets.
In our case, we exploit the properties of the community detection problem to produce an energy bound that depends on the concentration certain random matrices, which one may hope for as the size of the input graphs increases.
Finally, (Zhang, 2016)’s work on data regularization for clustering and rank estimation is also motivated by the success of using Bethe-Hessian-like perturbations to improve spectral methods on sparse networks.
It ﬁnds good perturbations via matrix perturbations, and also has success on the stochastic block model.
(Yang & Leskovec, 2012a) curates benchmark datasets for community detection and quantiﬁes the quality of these datasets, while (Yang & Leskovec, 2012b) develops new algorithms for community detection by ﬁtting data to newly designed generative models, which exhibit similar statistical structure learned from their analysis of the aforementioned datasets.
L INE G RAPH N EURAL N ETWORKS
This section introduces our GNN architectures based on the power graph adjacency (Section 4.1) and its extension to line graphs using the non-backtracking operator (Section 4.2), as well as the design of losses invariant to global label permutations (Section 4.3).
4. P OWER G RAPH N EURAL N ETWORKS
The Graph Neural Network (GNN), introduced in (Scarselli et al., 2009) and later simpliﬁed in (Li et al., 2015; Duvenaud et al., 2015; Sukhbaatar et al., 2016) is a ﬂexible neural network architecture that is based on local operators on a graph G =
( V, E ) .
We start by brieﬂy reviewing the generic
GNN architecture, and next describe our modiﬁcations to make it suitable to our interests.
Given some input signal x ∈ R | V |× b on the vertices of G , we consider graph intrinsic linear operators that act locally on this signal: The degree operator is the linear map D : F (cid:55)→ DF where ( Dx ) i :=
deg ( i ) · x i , D ( x ) = diag ( A 1 ) x .
The adjacency operator A is the linear map given by the adjacency matrix
A i,j = 1 iff ( i, j ) ∈ E .
In this way, J -th powers of A encode J
-hop neighborhoods of each node, and allow us to combine and aggregate local information at different scales.
We consider in this work the power graph adjacency A j = min(1 , A 2 j ) , which encodes 2 j -hop neighborhoods into a binary graph.
We consider a multiscale GNN layer that receives as input a signal x ( k ) ∈ R | V |× b k and produces x ( k +1) ∈ R | V |× b k +1 as
[Equation]
where Θ = { θ ( k ) 1 , . .
, θ ( k ) J +3 } , θ ( k ) s ∈ R b
k × b k +1 are trainable parameters and ρ ( · ) is a point-wise nonlinearity, chosen in this work to be ρ ( z )
= max(0 , z ) .
We thus consider a layer with linear “residual connections" (He et al., 2016), both to ease with the optimization when using large number of layers and to give the model the ability to perform power iterations.
Since the spectral radius of the learned linear operators in (1) can grow as the optimization progresses, the cascade of GNN layers can become unstable to training.
In order to mitigate this effect, we consider spatial batch normalization (Ioffe & Szegedy, 2015) at each layer.
As explained in Section B.1, the Krylov subspace generated by the graph Laplacian (Defferrard et al., 2016) is not sufﬁcient in this case to operate well in the sparse regime, as opposed to the generators { I, D, A } .
The expressive power of each layer is increased by adding multiscale versions of A , although this beneﬁt comes at the cost of computational efﬁciency, especially in the sparse regime.
The network depth is chosen to be of the order of the graph diameter, so that all nodes obtain information from the entire graph.
In sparse graphs with small diameter, this architecture offers excellent scalability and computational complexity.
Indeed, in many social networks diameters are constant (due to hubs), or ∼ log( | V | ) , as in the stochastic block model in the constant average degree regime (Riordan & Wormald, 2010).
This results in a model with computational complexity of the order of ∼ | V | log( | V ) , making it amenable to large-scale graphs.
In our setup, batch normalization not only prevents gradient blowup, but also performs the orthogonalisation relative to the constant vector, which is associated with the smallest eigenvector of the graph operator whose spectrum contains community information.
This reinforces the analogy between cascading layers of (1) and the power iterations to obtain the Fiedler vector of such operator.
Indeed, if one wants to extract the Fiedler vector of a matrix M , whose smallest eigenvector is known to be v , one can do so by performing power iterations on ˜ M = (cid:107) M (cid:107) I − M as y ( n +1) =
˜
Mx ( n ) , x ( n +1) =
y ( n +1) − v T vy ( n +1) (cid:107) y
( n +1)
− v T vy ( n +1) (cid:107) .
If v is a constant vector, then the normalization above is precisely performed within the batch normalization step.
4. LGNN: P OWER GNN ON L INE G RAPHS WITH N ON - BACKTRACKING O PERATOR
For graphs that have few cycles, posterior inference can be remarkably approximated by loopy belief propagation (Yedidia et al., 2003).
As described in Section B.2, the message-passing rules are deﬁned over the edge adjacency graph; see equation 57.
Although its second-order approximation around the critical point can be efﬁciently approximated with a power method over the original graph, a data-driven version of BP requires accounting for the non-backtracking structure of the message-passing.
In this section we describe how to upgrade the GNN model so that it can exploit non-backtracking operators.
The line graph L ( G ) =
( V L , E L ) is the graph representing the edge adjacency structure of G .
If G =
( V, E ) is an undirected graph, then the vertices V L of L ( G ) are the ordered edges in E , that is V L
= { ( i → j ); ( i, j ) ∈
E } ∪ { ( j → i ); ( i, j ) ∈ E } , so | V L | = 2 | E | .
The nonbacktracking operator B ∈ R 2 | E |× 2 | E | encodes the edge adjacency structure as follows.
Two nodes in L ( G ) are connected if
[Equation]
) = (cid:26)
if j = i (cid:48) and j (cid:48) (cid:54)
=
i , otherwise.
This operator thus enables the propagation of directed information through the graph, and was ﬁrst proposed in the context of community detection on sparse graphs in (Krzakala et al., 2013).
The message-passing rules of BP can be expressed as a diffusion in the line graph
L ( G ) using this non-backtracking operator, with speciﬁc choices of activation function that turn product of beliefs into sums.
A natural extension of the GNN architecture presented in
Section 4.1 is thus to consider a second GNN deﬁned on
L ( G ) , generated by the corresponding non-backtracking operator B and degree D B = diag( B 1 ) operators.
This effectively deﬁnes edge features that are diffused and updated according to the edge adjacency of G .
Edge and node features are combined at each layer using the edge indicator matrices
Pm , Pd ∈ { 0 , 1 } | V |
[Equation]
2 | E | , deﬁned as Pm i, ( i → j ) = 1 , Pm j, ( i → j ) = 1 , Pd i, ( i → j ) = 1 ,
Pd j, ( i → j ) = − 1 and 0 otherwise.
Dropping the skip linear connections for ease of exposition, the resulting model becomes
[Equation]
(cid:48)
+
J − 1 (cid:88)
j
=
( A 2 j L ( G ) y ( k ) )
i (cid:48) γ ( k ) 3+ j,l (cid:48)
+ [ { Pm , Pd } (cid:62) x ( k +1) ]
i (cid:48) γ ( k )
+ J,l (cid:48) (cid:35) , i (cid:48) ∈ V L .
with additional parameters { γ ( k ) 1 , . . .
, γ ( k ) J +3 } , γ ( k ) s ∈ R b
k × b k +1 .
The resulting architecture is named as a Line Graph Neural Network (LGNN) .
It can be veriﬁed that the resulting model Φ( G ) :
= x ( K ) satisﬁes the permutation equivariance property required for the task: Φ( G π ) =
ΠΦ( G ) , where Π is the permutation matrix associated with π .
Several authors have proposed combining node and edge feature learning (Gori et al., 2005;
Gilmer et al., 2017; Velickovic et al., 2017), although we are not aware of works that considered the edge adjacency structure provided by the line graph and the non-backtracking operator.
For graph families with constant average degree d , the line graph has size 2 | E | =
d | V | of the same order, making this model feasible from the computational point of view.
The line graph construction can be iterated with L ( L ( G )) , L ( . . .
L ( L ( G )) . . . )
to yield a graph hierarchy , which would capture high-order interactions between the elements of G .
Such hierarchical construction relates to other recent efforts to generalize GNNs (Kondor et al., 2018).
In our experiments, we use the input signals x (0) =
deg ( G ) and y (0) =
deg ( L ( G )) in the line graph version.
Relationship between LGNN and edge feature learning approaches: The GNN on the line graph using the non-backtracking operator can be interpreted as learning directed edge features from an undirected graph.
Indeed, if each node i contains two distinct sets of features x
s ( i ) and x r ( i ) , the non-backtracking operator constructs edge features from node features while preserving orientation:
For an edge e =
( i, j ) , our model constructs oriented edge features f i → j = g ( x s ( i ) , x r ( j )) and f j → i = g ( x r ( i ) , x s ( j ))
(where g is trainable and not necessarily commutative on its arguments) that are subsequently propagated through the graph.
Constructing such local oriented structure is shown to signiﬁcantly improve performance in the next section.
(Battaglia et al., 2016) introduced edge features over directed and typed graphs, but does not discuss the undirected case.
(Kearnes et al., 2016; Gilmer et al., 2017) learn edge features on undirected graphs using f e = g ( x ( i ) , x ( j )) for an edge e
= ( i, j ) , where g is now commutative on its arguments.
Finally, (Velickovic et al., 2017) learns directed edge features on undirected graphs using stochastic matrices as adjacencies (which are either row or column-normalized).
4. A L OSS F UNCTION I NVARIANT U NER L ABEL PERMUTATION
Let C
= { c 1 , . .
c C } denote the possible community labelings that each node can take.
Consider ﬁrst the case where communities do not overlap: C equals the number existing communities.
We deﬁne the network output at each node using standard softmax, computing the conditional probability that node i belongs to community c : o i,c = p ( y i =
c | θ, G ) .
Let y ∈ C V be the ground truth community structure.
Since community belonging is deﬁned up to global label changes in communities, we deﬁne the loss associated with a given graph instance as
[Equation]
where S C denotes the permutation group of C elements.
In our experiments we considered examples with small number of communities C ∈ { 2 , 5 } , but general scenarios, where C is suspected to be much larger, might make the evaluation of (3) over the permutation group of C elements impractical.
A possible solution is to randomly partition for each sample
C/ ˜ C labels into ˜ C groups, then marginalize the model outputs
o i,c , c ≤ C into ¯ o i, ¯ c
= (cid:80) c
∈ ¯
c o i,c and use (cid:96) ( θ ) =
inf π ∈ S ˜ C − (cid:80)
i ∈ V log ¯ o i,π (
[Equation]
) , which only involves a permutation group of size ˜ C ! .
Finally, if we are in a setup where nodes can belong to multiple communities, we simply redeﬁne C to include subsets of communities instead of just singletons, and modify the permutation group S C accordingly.
E NERGY L ANDSCAPE OF L INEAR GNN OPTIMIZATION
As described in the numerical experiments, we found that the GNN models without non-linear activations already provide substantial gains relative to baseline (non-trained) algorithms, by ﬁnding suitable generalizations of power iterations.
This section studies the optimization landscape resulting from this linear assumption.
Despite deﬁning a non-convex objective, we prove that the landscape is ‘benign’ under certain further simpliﬁcations, in the sense that the local minima are conﬁned on sublevel sets of low energy.
For simplicity, we consider only the binary c = 2 case where we replace the node-wise binary cross-entropy by the squared cosine distance 2 , and we assume a single feature map ( d k = 1 for all k ), and focus on the power GNN described in Section 4.1 (although our analysis carries equally to describe the line graph version; see remarks below).
We also make the simplifying assumption to replace the layer-wise spatial batch normalization by a simpler projection onto the unit (cid:96) 2 ball (thus we do not remove the mean).
Without loss of generality, assume that the input graph G has size n , and denote by F =
{ A 1 , . . .
, A Q } the family of graph operators appearing in (1).
Each layer thus applies an arbitrary polynomial (cid:80) Qq
=1 θ ( k ) q
A q to the incoming node feature vector x ( k ) .
Given an input node vector w ∈ R n , the network output can thus be written as
[Equation]
We highlight that this multilinear GNN setup is fundamentally different from the multilinear fullyconnected neural networks (that is, neural networks with linear activation function), whose landscape has been analyzed before (Kawaguchi, 2016).
First, the output is normalized in the sphere, which has important effects in the geometry.
Next, the network parametrization is intrinsic (the operators
O j depend on the input), which introduces ﬂuctuations in the landscape that we analyze.
In general, the operators in F are not commutative, but by considering the generalised Krylov subspace generated by powers of F , F K = { O 1 =
A K 1 , O
=
A 1 A K − 1, O
=
A 1 A 2 A K − 2, . . .
O Q K =
A KQ } , one can reparametrise (4) as e = (cid:80) Q K j =1 β j O j w with β ∈ R M , with M = Q K .
Given the target y ∈ R n , the loss incurred by each pair ( G, y ) becomes 1 − |(cid:104) e,y
[Equation]
| 2 (cid:107) e (cid:107) 2 , and therefore the population loss, when expressed in terms of β , equals
[Equation]
, with (5)
[Equation]
X n
[Equation]
n
[Equation]
The landscape is thus speciﬁed by a pair of random matrices Y n , X n ∈ R M × M .
The following theorem establishes that under appropriate assumptions, the concentration of certain random matrices around their mean controls the energy gaps between local and global maxima of L .
We deﬁne a “mean-ﬁeld“ loss function ˜
L n ( β ) = 1 − E X n ,Y n β T Y n β β T E X n β = 1 − β T E Y n β β T E X n β , and consider
L n as a perturbation of ˜ L n .
Assuming that E X n (cid:31) 0 , we write the Cholesky decomposition of E X n as E X n = R n R Tn , and deﬁne A n = R − 1 n Y n ( R − 1 n ) T , ¯ A n = E A n = R − 1 n E Y n ( R − 1 n ) T ,
B n = R − 1 n X n ( R − 1 n ) T , and ∆ B n = B n − I n .
Given a symmetric matrix K ∈ R M × M , we let λ 1 ( K ) , λ 2 ( K ) , ..., λ M ( K ) denote the eigenvalues of K in nondecreasing order.
Theorem 5.1.
For a given n , let η n = ( λ 1 ( ¯ A n ) − λ 2 ( ¯ A n )) − 1 , µ n = E
[ | λ 1 ( A n ) | 6 ] , ν n = E
[ | λ 1 ( B n ) | − 6 ] , δ n = E [ (cid:107) ∆ B n (cid:107) 6 ] , and assume that all four quantities are ﬁnite.
Then if β l ∈ S M − 1 is a local minimum of L n , and β g ∈ S M − 1 is a global minimum of L n , we have
L n ( β l ) ≤ (1 + (cid:15) η n ,µ n ,ν n ,δ n ) · L n ( β g ) , where (cid:15) η n ,µ n ,ν n ,δ n = O ( δ n ) for given η n , µ n , ν n as δ n → 0 and its formula is given in the appendix.
Corollary 5.2.
If ( η n ) n ∈ N ∗ , ( µ n ) n ∈ N ∗ , ( ν n ) n ∈ N ∗ are all bounded sequences, and lim n →∞ δ n = 0 , then ∀ (cid:15) > 0 , ∃ n (cid:15) such that ∀ n > n (cid:15) , | L n ( β l ) − L n ( β g ) | ≤ (cid:15) · L n ( β g ) .
The main strategy of the proof is to consider the actual loss function L n as a perturbation of ˜ L n , which has a landscape that is easier to analyze and does not have poor local minima, since it is equivalent to a quadratic form deﬁned over the sphere S M − 1 .
For a given graph inverse problem, applying this theorem thus requires estimating spectral ﬂuctuations of the pair X n , Y n , which in turn involve the spectrum of C ∗ algebras generated by the non-commutative family F .
For example, for stochastic block models, which we will discuss in the next section, it is an open problem how the bound behaves as a function of the parameters p and q .
That said, one should expect concentration to happen in general, since the dimension M is ﬁxed as n grows.
Another interesting question is to understand how the asymptotics of our landscape analysis relate to the hardness of estimation as a function of the signal-to-noise ratio.
Finally, another open question is to what extent our result could be extended to the non-linear residual GNN case, perhaps leveraging ideas from (Shamir, 2018).
E XPERIMENTS
We present experiments on community detection in synthetic datasets (Sections 6.1, 6.2 and Appendix
C) as well as real-world datasets (Section 6.3).
In the synthetic experiments, our performance measure is the overlap between predicted ( ˆ y ) and true labels ( y ), which quantiﬁes how much better than random guessing a predicted labelling is.
The overlap is given by (cid:0) 1 n (cid:80)
u δ y ( u ) , ˆ y ( u ) − 1 C (cid:1) / (1 − 1 C ) , where δ is the Kronecker delta function, and the labels are deﬁned up to global permutation.
We used Adamax (Kingma & Ba, 2014) with learning rate 0 .
across all experiments.
6. B INARY S TOCHASTIC B LOCK M ODEL
The stochastic block model is a random graph model with planted community structure.
In its simplest form, the graph consists of n = | V | nodes, which are partitioned into C communities, that is, each node is assigned a label y ∈ { 1 , ..., C } .
An edge connecting any two vertices u, v is drawn independently at random with probability p if y ( v ) =
y ( u ) , and with probability q otherwise.
In the binary case (i.e. C = 2 ), the sparse regime, where p, q (cid:39) 1 /n , is well understood and provides an initial platform to compare the GNN and LGNN with provably optimal recovery algorithms (Appendix B).
We consider two learning scenarios.
In the ﬁrst scenario, we train parameters θ for different values of p and q .
We sample 6000 graphs under G ∼ SBM ( n = 1000 , p i , p i , C = 2) for different pairs of ( p i , q i ) and then train θ ( p i , q i ) for each i .
In the second scenario, reported in Appendix D, we train a single set of parameters θ from a sample of 6000 samples containing a mixture of SBM with different parameters p, q and average degree.
This setup is important as it shows our GNN is not just approximating known algorithms such as BP for particular SBM parameters, since the parameters are not constant in this dataset.
For the ﬁrst scenario, different pairs of ( p i , q i ) correspond to different signal-to-noise ratios (SNRs).
We chose ﬁve such pairs by ﬁxing p + q and varying SNR.
Figure 3 reports the performance of our models on the binary SBM model for the different SNRs, compared with baseline methods including BP, spectral methods using the normalized Laplacian and the Bethe Hessian as well as
Graph Attention Networks (GAT) 4 from Velickovic et al.
The GNN and LGNN had 30 layers and 8 features (i.e. b k ’s) in the middle layers.
We observe that both GNN and LGNN reach the performance of BP.
In addition, even the linear LGNN achieves a performance that is quite close to that of BP, in accordance to the spectral approximations of BP given by the Bethe Hessian (see supplementary), and signiﬁcantly outperforms performing 30 power iterations on the Bethe Hessian or the normalized Laplacian, as was done in the spectral methods.
We also notice that our models outperform GAT in this task.
We ran experiments in the dissociative case ( q > p ), as well as with C = 3 communities and obtained similar results, which are not reported here.
A possible reason that our models outperform GAT is that our graph operators include the degree matrix and powers of graph adjacency, which are important in sparse graphs to prevent hub nodes from dominating the diffusion.
We thus performed an ablation study by not including the degree matrix in GNN and LGNN.
The results together with that of GNN and LGNN including the degree matrix are presented in Figure 3, right panel.
6. C OMPUTATIONAL - TO -S TATISTICAL T HRESHOLDS IN THE SBM
In SBM with fewer than 4 communities, it is known that BP provably reaches the information-theoretic threshold (Abbe, 2017; Massoulié, 2014; Coja-Oghlan et al., 2016).
The situation is different for k > 4 , where it is conjectured that a gap emerges between the theoretical performance of MLE estimators and the performance of any polynomial-time estimation procedure (Decelle et al., 2011).
In this context, one can use the GNN models to search the space of the generalizations of BP, and attempt to improve upon the detection performance of BP for scenarios where the SNR falls within the computational-to-statistical gap.
Table 1 presents results for the 5 -community dissociative SBM, with n = 400 , p = 0 and q = 18 /n .
The SNR in this setup is above the information-theoretic threshold but below the asymptotic threshold above which BP is able to detect (Decelle et al., 2011).
Note that since p = 0 , this also amounts to a graph coloring problem.
We see that the GNN models outperform BP in this experiment, and that the LGNN version yields the best average overlap on test data, opening up the possibility to reduce the computation-information gap.
That said, our model 3 Code will be released at ???.???.???
We based our implementation from https://github.com/Diego999/pyGAT .
We modiﬁed the code so that the number of layers in the network is ﬂexible, and also added spatial batch normalization at the end of each layer.
Our experiments showed that including spatial batch normalization improves the performance.
The results in sections 6.1 and 6.2 are from a trained GAT with 30 layers and 8 features.
[Figure]
[Table]
may be picking ﬁnite-size effects, which will vanish as n → ∞ .
The asymptotic study of these gains is left for future work.
6. R EAL D ATASETS FROM SNAP
We now compare the models on the SNAP datasets, whose domains range from social networks to hierarchical co-purchasing networks.
We obtain the training set as follows.
For each SNAP dataset, we start by focusing only on the 5000 top quality communities provided by the dataset.
We then identify edges ( i, j ) that cross at least two different communities.
For each of such edges, we consider the two largest communities C 1 , C 2 such that i / ∈ C 2 and j / ∈ C 1 , i ∈ C 1 , j ∈ C 2 , and extract the subgraph determined by C 1 ∪ C 2 , which is connected since all the communities are connected.
Finally, we divide the dataset into training and testing sets by enforcing the testing examples to contain disjoint communities from those in the training set.
In this experiment, due to computational limitations, we restrict our attention to the three smallest graphs in the SNAP collection (Youtube,
DBLP and Amazon), and we restrict the largest community size to 800 nodes, which is a conservative bound, since the average community size on these graphs is below 30 .
We compare the performance of our GNN and LGNN with the Community-Afﬁliation Graph Model (AGM) and another variant of GNN that uses symmetric edge features instead of the non-backtracking operator, which ﬁts into the framework of Message Passing Neural Networks (MPNNs)
(Gilmer et al., 2017) using ReLU activations.
AGM is a generative model proposed in (Yang & Leskovec, 2012b) that captures the overlapping structure of real-world networks, and community detection can be achieved by ﬁtting
AGM to a given network, which was shown to outperform some state-of-the-art algorithms.
Tablecompares the performance, measured with a 3-class { 1 , 2 , 1&2 } classiﬁcation accuracy up to global permutation 1 ↔ 2 .
It illustrates the beneﬁts of data-driven models that strike the right balance between expressive power to adapt to model misspeciﬁcations and structural assumptions of the task at hand.
[Table]
C ONCLUSION
In this work we have studied data-driven approaches to community detection with graph neural networks.
Our results conﬁrm that, even when the signal-to-noise ratio is at the lowest detectable regime, it is possible to backpropagate detection errors through a graph neural network that can ‘learn’ to extract the spectrum of appropriate operators.
This is made possible by considering a family of graph operators that work effectively in sparsely connected graphs, in particular by considering a hierarchical extension that uses the non-backtracking operator in the line graph.
We also provide a theoretical analysis of the optimization landscapes in the linearized regime, which shows an interesting transition from rugged to simple as the size of the graphs increase under appropriate concentration conditions.
One word of caution is that our empirical results are inherently non-asymptotic.
Whereas models trained for given graph sizes can be used for inference on arbitrarily sized graphs (owing to the parameter sharing of GNNs), further work is needed in order to understand the generalisation properties as | V | increases.
Nevertheless, we believe our work opens up interesting questions, namely better understanding how our results on the energy landscape depend upon speciﬁc signal-to-noise ratios, or whether the network parameters can be interpreted mathematically.
This could be useful in the study of computational-to-statistical gaps, where our model can be used to inquire about the form of computationally tractable approximations.
Another current limitation of our model is that it presumes a ﬁxed number of communities to be detected.
Other directions of future research include the extension to the case where the number of communities is unknown and variable, and potentially increasing with | V | , as well as applications to ranking and edge-cut problems.
A P ROOF OF T HEOREM 5.
For simplicity and with an abuse of notation in the remaining part we redeﬁne L and ˜ L in the following way, to be the negative of their original deﬁnition in the main section: L n ( β ) =
E X n ,Y n β (cid:62) Y n β β (cid:62) X n β , ˜ L n ( β ) =
E X n ,Y n β T Y n β β T E X n β .
Thus, minimizing the loss function (5) is equivalent to maximizing the function L n ( β ) redeﬁned here.
We write the Cholesky decomposition of E X n as E X n = R n R Tn , and deﬁne A n = R − 1 n Y n ( R − 1 n ) T , ¯
A n = E A n = R − 1 n E Y n ( R − 1 n ) T , B n = R − 1 n X n ( R − 1 n ) T , and ∆ B n = B n − I n .
Given a symmetric matrix K ∈ R M × M , we let λ 1 ( K ) , λ 2 ( K ) , ..., λ M ( K ) denote the eigenvalues of K in nondecreasing order.
First, we have
[Equation]
Let us denote by ˜
β g a global minimum of the mean-ﬁeld loss ˜
L n .
Taking a step further, we can extend this bound to the following one (the difference is in the second term on the right hand side):
Lemma A.1.
[Equation]
Proof of Lemma A.1.
We consider two separate cases: The ﬁrst case is when ˜ L n ( β l ) ≥ ˜ L n ( β g ) .
Then ˜ L n ( β l ) − ˜ L n ( ˜ β g ) ≥ ˜ L n ( β l ) − ˜ L n ( β g ) ≥ 0 , and so | L n ( β l ) − L n ( β g ) | ≤ | L n ( β l ) − ˜ L n ( β l ) | + | ˜
L n ( β l ) − ˜ L n (˜ β g ) | + | ˜
L n ( β g ) − L n ( β g ) | .
The other case is when ˜ L n ( β l ) <
˜ L n ( β g ) .
Note that L n ( β l ) ≥ L n ( β g ) .
Then | L n ( β l ) − L n ( β g ) | ≤ | L n ( β l ) − ˜ L n ( β l ) | + | ˜ L n ( β g ) − L n ( β g ) | ≤ | L n ( β l ) − ˜ L n ( β l ) | + | ˜
L n ( β l ) − ˜ L n (˜ β g ) | + | ˜
L n ( β g ) −
L n ( β g ) | .
Hence, to bound the "energy gap" | L n ( β l ) − L n ( β g ) | , if sufﬁces to bound the three terms on the right hand side of Lemma A.1 separately.
First, we consider the second term, | ˜
L n ( β l ) − ˜ L n (˜ β g ) | .
Let γ l = R Tn β l , γ g = R Tn β g and ˜ γ g = R Tn ˜ β g .
Deﬁne S n ( γ )
=
L n ( R − Tn γ ) and ˜ S n ( γ ) =
˜ L n ( R − Tn γ ) , for any γ ∈ R M .
Thus, we apply a change-of-variable and try to bound | ˜ S n ( γ l ) − ˜ S n (˜ γ g ) | .
Since β l is a local maximum of L n , λ 1 ( ∇ 2 L n ( β l )) ≤ 0 .
Since ∇ 2 S n ( γ l )
=
R − 1 n ∇ 2 L n ( β l ) R − Tn , where R n is invertible, we know that λ 1 ( ∇ 2 S n ( γ l )) ≤ 0 , thanks to the following lemma:
Lemma A.2.
If R, Q ∈ R M × M , R is invertible, Q is symmetric and λ q > 0 is an eigenvalue of Q , then λ 1 ( RQR T ) ≥ λ · λ M ( RR T )
Proof of Lemma A.2.
Say Qw = λw for some vector w ∈ R M .
Let v =
R − T w .
Then v T ( RQR T ) v = w T Qw = λ (cid:107) w (cid:107) 2 .
Note that (cid:107) w (cid:107) 2 = v T RR T v ≥ (cid:107) v (cid:107) 2 λ M ( RR T ) .
Hence
[Equation]
Since ∇ 2 S n ( γ l ) = ∇ 2 ˜ S n ( γ l ) + ( ∇ 2 S n ( γ l ) − ∇ 2 ˜ S n ( γ l )) , there is 0 ≥ λ 1 ( ∇ 2 S n ( γ l )) ≥
[Equation]
n ( γ l ) (cid:107) .
Hence,
[Equation]
Next, we relate the left hand side of the inequality above to cos( γ l , ˜ γ g ) , thereby obtaining an upper bound on [1 − cos 2 ( γ l , ˜ γ g )] , which will then be used to bound | ˜ S n ( γ l ) − ˜ S n (˜ γ g ) | .
Lemma A.3.
∀ γ ∈ R d ,
[Equation]
Proof of Lemma A.3.
[Equation]
Thus, if we deﬁne Q 1 = ( γ T γ )
[( γ T γ ) ¯ A n − ( γ T ¯ A n γ ) I ] , Q 2 =
[Equation] [Equation]
γ T γ
[Equation]
¯
[Equation]
n −
[Equation]
T ¯
[Equation]
n
[Equation]
I ] γγ T , we have
[Equation]
To bound λ 1 ( ∇ 2 ˜ S n ( γ )) , we bound λ 1 ( Q 1 ) and (cid:107) Q 2 (cid:107) as follows:
Since ¯ A n is symmetric, let ˆ γ 1 , . . .
ˆ γ M be the orthonormal eigenvectors of ¯ A n corresponding to nonincreasing eigenvalues l 1 , . . .
l M .
Note that the global minimum satisﬁes ˜ γ g = ±
ˆ γ 1 .
Write γ = (cid:80) Mi
=
α i ˆ γ
i , and let ¯ α i
= α i √ (cid:80) Mi
=
α 2 i .
Then | cos( γ, ˜ γ g ) | = | cos( γ, ˆ γ 1 ) | =
| ¯ α 1 | .
Then,
[Equation]
To bound (cid:107) Q 2 (cid:107) :
[Equation]
[Equation]
Note that given vectors v, w ∈ R M ,
[Equation]
Therefore,
[Equation]
Thus
[Equation]
This yields the desired lemma
[Equation]
Combining inequality 8 and Lemma A.3, we get
[Equation]
Thus, to bound the angle between γ l and ˜ γ g , we can aim to bound (cid:107)∇ ˜ S n ( γ l ) (cid:107) and (cid:107)∇ 2 S n ( γ l ) − ∇ 2 ˜ S n ( γ l ) (cid:107) as functions of the quantities µ n , ν n and δ n .
Lemma A.4.
[Equation]
Proof of Lemma A.4.
[Equation]
Combining equations 17 and 18, we get
[Equation]
Since ∇ S n ( γ l ) = 0 , we have
[Equation]
Then, by the generalized Hölder’s inequality,
[Equation]
Hence, written in terms of the quantities µ n , ν n and δ n , we have
[Equation]
Lemma A.5.
With δ n = ( E (cid:107) ∆ B n
[Equation] [Equation]
16 , E |
[Equation]
B
[Equation] [Equation]
[Equation] [Equation]
Proof of Lemma A.5.
[Equation]
Note that gma
[Equation]
and for k ∈ { 1 , 2 , 3 , 4 , 5 } , if X is a nonnegative random variable,
[Equation]
Therefore, E | λ 1 ( B n ) | 6 ≤ 64 + 63 E (cid:107) ∆ B n (cid:107) 6 .
From now on, for simplicity, we introduce δ (cid:48) n = (64 + 63 δ 6 n ) 16 , as a function of δ n .
Lemma A.6.
∀ γ ∈ R M ,
[Equation]
Proof of Lemma A.6.
[Equation]
27) where
[Equation] [Equation]
Thus, (cid:107)∇ 2 S n
[Equation]
γ )
[Equation]
∇
[Equation]
˜ S n
[Equation]
γ )
[Equation]
H
[Equation]
+
[Equation]
E
[Equation]
H
[Equation]
+ 8 E
[Equation]
H
[Equation]
+ 8 E (cid:107) H 4 (cid:107) , and we try to bound each term on the right hand side separately.
For the ﬁrst term, there is
[Equation]
Applying generalized Hölder’s inequality, we obtain
[Equation]
For the second term, there is
[Equation]
Hence,
[Equation]
Applying generalized Hölder’s inequality
[Equation]
we obtain
[Equation]
For H 3 , note that
[Equation]
Hence,
[Equation]
Thus
[Equation]
Applying generalized Hölder’s inequality
[Equation]
we obtain
[Equation]
6 ) 16 ( E | λ 1 ( B n ) | 6 )+(cid:18) E | λ M ( B n ) |(cid:19)
( E | λ 1 ( A n ) | 3 ) 13 ( E (cid:107) ∆ B n (cid:107) 6 )+ (cid:18) E | λ M ( B n ) |(cid:19)
( E | λ 1 ( A n ) | 3 ) 13 ( E (cid:107) ∆ B n (cid:107) 3 )+ (cid:18) E | λ M ( B n ) |(cid:19)( E | λ 1 ( A n ) | 3 ) 13 ( E (cid:107) ∆ B n (cid:107) 3 )≤ µ n ν n δ n ( δ (cid:48) n ν 2 n + 2 ν 2 n + ν n + 1) (40)
[Equation]
Thus,
[Equation]
Applying generalized Hölder’s inequality
[Equation]
we obtain
[Equation]
Therefore, summing up the bounds above, we obtain
[Equation]
Hence, combining inequality 15, Lemma A.4 and Lemma A.6, we get
[Equation]
[Equation]
For simplicity, we deﬁne C ( δ n , ν n ) =
9+19 ν n +5 δ n ν n +8 ν 2 n
[Equation]
8 δ (cid:48) n ν n +4 δ (cid:48) n ν n 2 +4 δ (cid:48) n ν n +4 δ n δ (cid:48) n ν n .
Thus,
[Equation]
Following the notations in the proof of Lemma A.3, we write γ l = (cid:80) Mi
=
α i ˆ γ i .
Note that ˜ γ g = ± ˆ γ 1 , and | cos( γ, ˆ γ i ) | =
| ¯ α i | .
Thus,
[Equation]
Since Y n is positive semideﬁnite, E Y n is also positive semideﬁnite, and hence ¯ A n = R T n E Y n ( R − 1 n ) T is positive semideﬁnite as well.
This means that l i ≥ 0 , ∀ i ∈ { 1 , ..., M } .
Since ˜ L n ( ˜ β g ) =
˜ S n (˜ γ g ) = ˜ S n (ˆ γ 1 ) =
l 1 , there is
[Equation]
Next, we bound the ﬁrst and the third term on the right hand side of the inequality in Lemma A.1.
Lemma A.7.
∀ β ,
[Equation]
Proof of Lemma A.7.
Let γ =
T Tn β .
[Equation]
Thus
[Equation]
we get the desired lemma by the generalized Hölder’s inequality.
Combining inequality 46, inequality 48 and Lemma A.7, we get
[Equation]
Meanwhile,
[Equation]
) − ˜ L n (˜ β g ) |} ≤ ( E (cid:107) ∆ B n (cid:107) 3 ) 13 · ( E | λ 1 ( A n ) | 3 ) 13 · ( E | λ M ( B n ) | 3 )≤ µ n ν n δ n (52)
Hence,
L n ( β g ) ≥ ˜ L n (˜ β g ) − µ n ν n δ n ≥ λ 1 ( ¯ A n ) − µ n ν n δ n ≥ η − 1 n − µ n ν n δ n (53) , or λ 1 ( ¯ A n ) ≤ L n ( β g ) + µ n ν n δ n (54)
Therefore, | L n ( β l ) − L n ( β g ) | ≤ 2 µ n ν n δ n + (1 − cos 2 ( γ l , ˜ γ g ))
[ L n ( β g ) + µ n ν n δ n ] ≤ µ n ν n δ n [2 + η n µ n ν n δ n C ( δ n , ν n )] + η n µ n ν n δ n C ( δ n , ν n ) L n ( β g ) ≤ L n ( β g ) (cid:26) µ n ν n δ n [2 + η n µ n ν n δ n C ( δ n , ν n )]
η − 1 n − µ n ν n δ n + η n µ n ν n δ n C ( δ n , ν n ) (cid:27) =
2 η n µ n ν n δ n [2 + C ( δ n , ν n )] 1 − η n µ n ν n δ n
L n ( β g ) (55)
Hence, we have proved the theorem, with (cid:15) η n ,µ n ,ν n ,δ n = 2 η n µ n ν n δ n [2+ C ( δ n ,ν n )] 1 − η n µ n ν n δ n .
(cid:3)
B B ACKGROUND B. G RAPH M IN -C UTS AND S PECTRAL C LUSTERING
We consider graphs G =
( V, E ) ,
modeling a system of N = | V | elements presumed to exhibit some form of community structure.
The adjacency matrix A associated with G is the N × N binary matrix such that A i,j = 1 whenever ( i, j ) ∈ E .
We assume for simplicity undirected graphs, yielding symmetric adjacency matrices.
The community structure is encoded in a discrete label vector s :
V → { 1 , K } that assigns a community label to each node, and the goal is to estimate s from observing the adjacency matrix.
In the setting of binary, associative communities, where s ( i ) = ± 1 , two nodes i, j with s ( i ) =
s ( j ) are more likely to be connected ( A i,j = 1 ) than two nodes i, j with s ( i ) (cid:54) =
s ( j ) .
Thus a quantity of the form (cid:88)
[Equation]
measures the cost associated with cutting the graph between communities encoded by s that we wish to minimize under appropriate constraints Newman (2006).
Note that (cid:80) i,j A
i,j = s T Ds , with
D = diag ( A 1 ) (called the degree matrix), so the cut cost can be expressed as a positive semideﬁnite quadratic form
[Equation]
that we wish to minimize.
This shows a fundamental connection between the community structure and the spectrum of certain linear operators of the graph, which provides a powerful and stable relaxation of the discrete combinatorial optimization problem of estimating the community labels for each node.
In the case of the graph Laplacian ∆ = D − A , its eigenvector associated with the smallest eigenvalue is trivial, but its Fiedler vector (the eigenvector associated with the second smallest eigenvalue) reveals important community information of the graph Newman (2006) under appropriate conditions, and is associated with the graph conductance Spielman (2015) under certain normalization schemes.
For a given linear operator L ( A ) extracted from the graph (that we assume symmetric), we are thus interested in extracting eigenvectors at the edge of its spectrum.
A particularly simple algorithmic framework is given by the power iteration method.
Indeed, the Fiedler vector of L ( A ) can be obtained by ﬁrst extracting the leading eigenvector v of ˜
A =
(cid:107)L
[Equation]
A
[Equation]
− L ( A ) , and then iterating
[Equation]
Unrolling power iterations and recasting the resulting model as a trainable neural network is akin to the LISTA Gregor & LeCun (2010) sparse coding model, which unrolled iterative proximal splitting algorithms.
Despite the appeal of graph Laplacian spectral approaches, it is well known Krzakala et al.
that these methods fail in sparsely connected graphs.
Indeed, in such scenarios, the eigenvectors of graph Laplacians concentrate on nodes with dominant degree, losing their ability to correlate with community structure.
In order to overcome this important limitation, authors have resorted to ideas inspired from statistical physics, as explained next.
B. P ROBABILISTIC G RAPHICAL M ODELS AND B ELIEF -P ROPAGATION
Graphs with labels on nodes and edges can be cast as a graphical model where the aim of clustering is to optimize label agreement.
This can be seen as a posterior inference task.
If we simply assume the graphical model is a Markov Random Field (MRF) with trivial compatibility functions for cliques greater than 2, the probability of a label conﬁguration σ is given by
[Equation]
Generally, computing marginals of multivariate discrete distributions is exponentially hard.
For instance, in the case of P ( σ i ) we are summing over | X | n − 1 terms (where X is the state space of discrete variables).
But if the graph is a tree, we can factorize the MRF more efﬁciently to compute the marginals in linear time via a dynamic programming method called the sum-product algorithm, also known as belief propagation (BP).
An iteration of BP is given by
[Equation]
The beliefs ( b i → j ( σ i ) ) are interpreted as the marginal distributions of σ i .
Fixed points of BP can be used to recover marginals of the MRF above.
In the case of the tree, the correspondence is exact: P i ( σ i ) =
b i ( σ i ) .
Some sparse graphs, like the Stochastic Blockmodel with constant degree
Mossel et al.
(2014) are locally similar to trees for such an approximation to be successful.
BP approximates the MLE solutions but convergence is not guaranteed in graphs that are not trees.
Furthermore, in order to apply BP, we need a generative model and the correct parameters of the model.
If unknown, the parameters can be derived using expectation maximization, further adding complexity and instability to the method since iterations may learn parameters for which BP does not converge.
B. N ON - BACKTRACKING OPERATOR AND B ETHE H ESSIAN
The BP equations have a trivial ﬁxed-point where every node takes equal probability in each group.
Linearizing the BP equation around this point is equivalent to spectral clustering using the nonbacktracking matrix (NB), a matrix deﬁned on the edges of the graph that indicates whether two edges are adjacent and do not coincide.
Spectral clustering using NB gives signiﬁcant improvements over spectral clustering with versions of the Laplacians (L) and the adjacency matrix (A); High degree ﬂuctuations drown out the signal of the informative eigenvalues in the case of A and L, whereas
NB’s eigenvalues are conﬁned to a disk in the complex plane, so its eigenvalues corresponding to community structure lay outside the disk and are easily distinguishable.
NB matrices are still not optimal in that they are matrices on the edge set, and are not symmetric (so cannot enjoy tools of numerical linear algebra for symmetric matrices).
Recently Saade et al.
(2014) showed that a spectral method can do as well as BP in this regime, using the Bethe Hessian operator given by BH ( r ) :
= ( r 2 − 1)
I − rA + D
(where r is a scalar value).
This is due to a one-to-one correspondence between the ﬁxed points of BP and the stationary points of the Bethe free energy (corresponding Gibbs energy of the Bethe approximation)
Saade et al.
The Bethe Hessian is a scaling of the Hessian of the Bethe free energy at an extrema corresponding to the trivial ﬁxed point of BP.
Negative eigenvalues of BH ( r ) correspond to phase transitions in the Ising model where new clusters become identiﬁable.
This all gives theoretical motivation for why [ I , D, A ] deﬁned in
Section 3 are a good family of generators to do spectral clustering on.
In the case of the SBM, they generate the Bethe Hessian which can achieve community detection down to the information theoretic threshold.
The GNN is capable of expressing spectral approximations of complicated functions of [ I , D, A ] , and performing nonlinear power method iterations in order to infer global structure (for instance community structure).
Furthermore, unlike belief propagation, the method does not require a generative model, oftentimes requires a lot statistical analysis to motivate and is exposed to model misspeciﬁcations when deployed on real data.
Instead, our framework ﬁnds structure in a data driven way, learning it from the available training data.
B. S TOCHASTIC B LOCK M ODEL
We brieﬂy review the main properties needed in our analysis, and refer the interested reader to Abbe (2017) for an excellent recent review.
The Stochastic Blockmodel (SBM) is a random graph model denoted by SBM ( n, p, q, C ) .
Implicitly there is an F : V → { 1 , C } associated with each SBM graph, which assigns community labels to each vertex.
One obtains a graph from this generative model by starting with n vertices and connecting any two vertices u, v independently at random with probability p if F ( v ) =
F ( u ) , and with probability q if F ( v ) (cid:54)
=
F ( u ) .
We say the SBM is balanced if the communities are the same size.
Let ¯ F n : V → { 1 , C } be our predicted community labels for
SBM ( n, p, q, C ) , F n ’s give exact recovery on a sequence { SBM ( n, p, q ) } n if P ( F n = ¯ F n ) → n 1 , and give detection ∃ (cid:15) > 0 : P ( | F n − ¯ F n | ≥ 1 /k + (cid:15) )
→ n 1 (i.e ¯ F n ’s do better than random guessing).
It is harder to tell communities apart if p is close to q (if p =
q
we just get an Erd˝os Renyi random graph, which has no communities).
In the two community case, It was shown that exact recovery is possible on SBM ( n, p = a log n n , q = b log n n ) if and only if a + b 2 ≥ 1 + √ ab Mossel et al.
Abbe et al.
For exact recovery to be possible, p, q must grow at least O (log n ) or else the sequence of graphs will not to be connected, and thus vertex labels will be underdetermined.
There is no information-computation gap in this regime, so there are polynomial time algorithms when recovery is possible Abbe (2017); Mossel et al.
In the much sparser regime of constant degree SBM ( n, p = a n , q = b n ), detection is the best we hope for.
The constant degree regime is also of most interest to us for real world applications, as most large datasets have bounded degree and are extremely sparse.
It is also a very challenging regime; spectral approaches using the Laplacian in its various (un)normalized forms and the adjacency matrix, as well as SDP methods cannot detect communities in this regime Abbe (2017) due to large ﬂuctuations in the degree distribution that prevent eigenvectors form concentrating on the clusters.
Decelle et al.
ﬁrst proposed the BP algorithm on the SBM, which was proven to yield Bayesian optimal values in Coja-Oghlan et al.
In the constant degree regime with balanced k communities, the Kesten-Stigum threshold is given by
SNR :=
( a − b ) 2 / ( k ( a + ( k + 1) b ))
Abbe (2017).
It has been shown for k = 2 that SNR = 1 is both the information theoretic and efﬁcient computational threshold where belief propagation (BP) via a polynomial time algorithm.
For k > 4 a gap emerges between the information theoretic threshold and computational one.
It’s conjectured that no polynomial time algorithm exist for SNR < 1 , while a BP algorithm works for SNR > 1 Abbe (2017); Decelle et al.
The existence of the gap was shown by Abbe (2017) by proving a non-polynomial algorithm can do detection for some SNR < 1 .
C G EOMETRIC B LOCK M ODEL
[Table]
The success of Belief-Propagation on the SBM relies on its locally hyperbolic properties, which make it treelike with high probability.
This behavior is completely different if one considers random graphs with locally
Euclidean geometry.
The Geometric
Block Model Sankararaman & Baccelli (2018) is a random graph generated as follows.
One starts by sampling n iid points x 1 , . . .
, x n from a
Gaussian mixture model given by means µ 1 , . . .
µ k ∈ R d at distance S apart and identity covariances.
The labels correspond to which Gaussian each sample belongs to.
We draw an edge between two nodes i, j if (cid:107) x i − x j (cid:107) ≤ T/ √ n .
Due to the triangle inequality, the model contains a large number of short cycles, impacting the performance of loopy belief propagation.
This motivates other estimation algorithms based on motif-counting Sankararaman & Baccelli (2018) that require knowledge of the model likelihood function.
Table 3 shows the performance of the GNN models on the binary GBM model, obtained with d = 2 , n = 500 , T = 5 √ 2 and varying S , and its comparison with several spectral methods, including the Bethe Hessian, which approximates BP around its stationary solution.
We verify that the line GNN model, thanks to its added ﬂexibility and the multiscale nature of its generators, is able to signiﬁcantly outperform both spectral methods as well as the baseline GNN.
D F URTHER E XPERIMENTS ON THE SBM
We report here our experiments on the SBM mixture, generated with
[Equation]
where the average degree ¯ d is either ﬁxed constant or also randomized with ¯ d ∼ Unif (1 , t ) .
Figure 4 shows the overlap obtained by our model compared with several baselines.
Our GNN model is either competitive with BH or beats BH, which achieves the state of the art along with BP Saade et al.
(2014), despite not having any access to the underlying generative model (especially in cases where
GNN was trained on a mixture of SBM and thus must be able to generalize the r parameter in BH).
They all beat by a wide margin spectral clustering methods using the symmetric Laplacian (Ls) and power method (pm) applied to (cid:107) BH (cid:107) I − BH using the same number of layers as our model.
Thus
GNN’s ability to predict labels goes beyond approximating spectral decomposition via learning the optimal r for BH ( r ) .
The model architecture allows it to learn a higher dimensional function of the optimal perturbation of the multiscale adjacency basis, as well as nonlinear power iterations, that amplify the informative signals in the spectrum; In a data driven way it can generalize the problem without needing to study a generative model.