G RAPHON BASED C LUSTERING AND T ESTING OF N ETWORKS : A LGORITHMS AND T HEORY
A BSTRACT
Network-valued data are encountered in a wide range of applications, and pose challenges in learning due to their complex structure and absence of vertex correspondence.
Typical examples of such problems include classiﬁcation or grouping of protein structures and social networks.
Various methods, ranging from graph kernels to graph neural networks, have been proposed that achieve some success in graph classiﬁcation problems.
However, most methods have limited theoretical justiﬁcation and their applicability beyond classiﬁcation remains unexplored
In this work, we propose methods for clustering multiple graphs without vertex correspondence that are inspired by the recent literature on estimating graphons— symmetric functions corresponding to inﬁnite vertex limit of graphs
We propose a novel graph distance based on sorting-and-smoothing graphon estimators
Using the proposed graph distance
, we present two clustering algorithms and show that they achieve state -of- the art results.
We prove the statistical consistency of both algorithms under
Lipschitz assumptions on the graph degrees.
We further study the applicability of the proposed distance for graph two-sample testing problems
I NTRODUCTION
Machine learning on graphs has evolved considerably over the past two decades.
The traditional view towards network analysis is limited to modelling interactions among entities of interest, for instance social networks or world wide web, and learning algorithms based on graph theory have been commonly used to solve these problems (Von Luxburg, 2007; Yan et al., 2006).
However, recent applications in bioinformatics and other disciplines require a different perspective, where the networks are the quantities of interest.
For instance, it is of practical interest to classify protein structures as enzyme or non-enzyme (Dobson & Doig, 2003) or detect topological changes in brain networks caused by Alzheimer’s disease (Stam et al., 2007).
In this paper, learning from network-valued data refers to clustering where each network is treated as an entity, as opposed to the traditional network analysis problems that involve a single network of interactions (Newman, 2003).
Machine learning on network-valued data has been an active area of research in recent years, although most works focus on the network classiﬁcation problem.
The generic approach is to convert the network-valued data into a standard representation.
Graph neural networks are commonly used for network embedding, that is, ﬁnding Euclidean representations of each network that can be further used in standard machine learning models (Narayanan et al., 2017; Xu et al., 2019).
In contrast, graph kernels capture similarities between pairs of networks that can be used in kernel based learning algorithms (Shervashidze et al., 2011; Kondor & Pan, 2016; Togninalli et al., 2019).
In particular, the graph neural tangent kernel deﬁnes a graph kernel that corresponds to inﬁnitely wide graph neural networks, and typically outperforms neural networks in classiﬁcation tasks (Du et al., 2019).
A more classical equivalent for graph kernels is to deﬁne metrics that characterise the distances be- tween pairs of graphs (Bunke & Shearer, 1998), but there has been limited research on designing efﬁcient graph distances and developing algorithms for clustering network-valued data.
The motivation for this paper stems from two shortcomings in the literature on network-valued data analysis: ﬁrst, the efﬁcacy of existing kernels or embeddings have not been studied beyond network classiﬁcation, and second is the lack of theoretical analysis of these methods, particularly in the small sample setting.
Generalisation error bounds for graph kernel based learning exist (Du et al., 2019), but these bounds, based on learning theory, are meaningful only when many networks are available.
However, in many applications, one needs to learn from a small population of large networks and, in such cases, an informative statistical analysis should consider the small sample, large graph regime.
To address this issue, we take inspiration from the recent statistics literature on graph two-sample testing—given two (populations of) large graphs, the goal is to decide if they are from same statistical model or not.
Although most theoretical studies in graph two-sample testing focus on graph with vertex correspondence (Tang et al., 2017a; Ghoshdastidar & von Luxburg, 2018), some works address the problem of testing graphs on different vertex sets either by deﬁning distances between graphs (Tang et al., 2017b; Agterberg et al., 2020) or by representing networks in terms of pre-speciﬁed network statistics (Ghoshdastidar et al., 2017).
The use of network statistics for clustering network-valued data is studied in Mukherjee et al.
Another fundamental approach for dealing with graphs of different sizes is graph matching, where the objective is to determine the vertex correspondence.
Graph matching is often solved by formulating it as an optimization problem (Zaslavskiy et al., 2008; Guo et al., 2019) or deﬁning graph edit distance between the graphs (Riesen & Bunke, 2009; Gao et al., 2010).
Although there is extensive research on graph matching, the efﬁcacy of these methods in clustering network-valued data remains unexplored.
Contribution and organisation.
In this work, we follow the approach of deﬁning meaningful graph distances based on statistical models, and use the proposed graph distance in the context of learning from networks without vertex correspondence.
In particular, we propose graph distances based on graphons.
Graphons are symmetric bivariate functions that represent the limiting structure for a sequence of graphs with increasing number of nodes (Lov ´ asz & Szegedy, 2006), but can be also viewed as a nonparametric statistical model for exchangeable random graphs (Diaconis & Janson, 2007; Bickel & Chen, 2009).
The latter perspective is useful for the purpose of machine learning since it allows us to view the multiple graphs as random samples drawn from one or more graphon models.
This perspective forms the basis of our contributions, which are listed below:
In Section 2, we propose a distance between two networks, that do not have vertex correspondence and could have different number of vertices.
We view the networks as random samples from (unknown) graphons, and propose a graph distance that estimates the L 2 -distance between the graphons.
The distance is inspired by the sorting-and-smoothing graphon estimator (Chan & Airoldi, 2014).
In Section 3, we present two algorithms for clustering network-valued data based on the proposed graph distance: a distance-based spectral clustering algorithm, and a similarity based semi-deﬁnite programming (SDP) approach.
We derive performance guarantees for both algorithms under the assumption that the networks are sampled from graphons satisfying certain smoothness conditions.
We empirically compare the performance of our algorithms with other clustering strategies based on graph kernels, graph matching, network statistics etc. and show that, on both simulated and real data, our graph distance-based spectral clustering algorithm outperforms others while the SDP approach also shows reasonable performance, and they also scale to large networks (Section 3.3).
Inspired by the success of the proposed graph distance in clustering, we use the distance for graph two-sample testing.
In Section 4, we show that the proposed two-sample test is statistically consistent for large graphs, and also demonstrate the efﬁcacy of the test through numerical simulation.
We provide further discussion in Section 5 and present the proofs of theoretical results in Appendix.
G RAPH D ISTANCE BASED ON G RAPHONS
Clustering or testing of multiple networks requires a notion of distance between the networks.
In this section, we present a transformation that converts graphs of different sizes into a ﬁxed size representation, and subsequently, propose a graph distance inspired by the theory of graphons.
We ﬁrst provide some background on graphons and graphon estimation.
Graphon has been studied in the literature from two perspectives: as limiting structure for inﬁnite sequence of growing graphs (Lov´asz & Szegedy, 2006), or as exchangeable random graph model.
In this paper, we follow the latter perspective.
A random graph is said to be exchangeable if its distribution is invariant under permutation of nodes.
Diaconis & Janson (2007) showed that any statistical model that generates exchangeable random graphs can be characterised by graphons, as introduced by Lov´asz & Szegedy (2006).
Formally, a graphon is a symmetric measurable continuous function w : [0 , 1] 2 → [0 , 1] where w ( x, y ) can be interpreted as the link probability between two nodes of the graph that are assigned values x and y , respectively.
This interpretation propounds the following two stage sampling procedure for graphons.
To sample a random graph G with n nodes from a graphon w , in the ﬁrst stage, one samples n variables U 1 , . .
, U n uniformly from [0 , 1] and constructs a latent mapping between the sampled points and the node labels.
In the second stage, edges between any two nodes i, j are randomly added based on the link probability w ( U i , U j ) .
Mathematically, if we abuse notation to denote the adjacency matrix by G ∈ { 0 , 1 } n × n , we have
U 1 , . . .
, U n iid ∼ Uniform [0 , 1] and
G ij | U i , U j ∼ Bernoulli ( w ( U i , U j )) for all i <
j.
We consider problems involving multiple networks sampled independently from the same (or different) graphons.
We make the following smoothness assumptions on the graphons.
Assumption 1 (Lipschitz continuous)
A graphon w is Lipschitz continuous with constant L if | w ( u,
[Equation]
) −
[Equation]
( u
[Equation] [Equation]
) | ≤ L
[Equation]
( u − u (cid:48) )
2 + ( v − v (cid:48) )
for every u, v, u (cid:48) , v (cid:48) ∈ [0 , 1] .
Assumption 2 (Two-sided Lipschitz degree)
A graphon w has two-sided Lipschitz degree with constants λ 1 , λ 2 > 0 if its expected degree function g , deﬁned by g ( u ) =
(cid:82) 1 0 w ( u, v )d v , satisﬁes
[Equation] for every u, u (cid:48) ∈ [0 , 1] .
One of the challenges in graphon estimation is due to the issue of non-identiﬁability, that is, different graphon functions w can generate the same random graph model.
In particular, two graphons w and w (cid:48) generate the same random graph model if they are weakly isomorphic—there exist two measure preserving transformations φ, φ (cid:48) :
[0 , 1] → [0 , 1] such that w ( φ ( u ) , φ ( v ))
=
w (cid:48) ( φ (cid:48) ( u ) , φ (cid:48) ( v )) .
Moreover, the converse also holds meaning that such transformations are known to be the only source of non-identiﬁability (Diaconis & Janson, 2007).
This weak isomorphism induces equivalence classes on the space of graphons.
Since our goal is only to cluster graphs belonging to random graph models, we simply make the following assumption on our graphons.
Assumption 3 (Equivalence classes)
Any reference to K graphons, w 1 , . . .
, w K , assumes that, for every i, j , either w i = w j or w i and w j belong to different equivalence classes.
Furthermore, without loss of generality, we assume that every graphon w i is represented such that the corresponding degree function g i is non-decreasing.
Remark on the necessity of Assumptions 1–3.
Assumption 1 is standard in graphon estimation literature (Klopp et al., 2017) since it avoids graphons corresponding to inhomogeneous random graph models.
It is known that two graphs from widely separated inhomogeneous models (in L 2distance) are statistically indistinguishable (Ghoshdastidar et al., 2020), and hence, it is essential to ignore such models to derive meaningful guarantees.
Assumption 2 ensures that, under a measurepreserving transformation, the graphon has strictly increasing degree function, which is a canonical representation of an equivalence class of graphons (Bickel & Chen, 2009).
Assumption 3 is needed since graphons can only be estimated up to measure-preserving transformation.
As noted above, it is inconsequential for all practical purposes but simpliﬁes the theoretical exposition.
Graph transformation.
In order to deal with multiple graphs and measure distances among pairs of graphs, we require a transformation that maps all graphs into a common metric space—the space of all n 0 × n 0 symmetric matrices for some integer n 0 .
While the graphon estimation literature provides several consistent estimators (Klopp et al., 2017; Zhang et al., 2017), only the histogram based sorting-and-smoothing graphon estimator of Chan & Airoldi (2014) can be adapted to meet the above requirement.
We use the following graph transformation, inspired by Chan & Airoldi (2014).
The adjacency matrix G of size n × n is ﬁrst reordered based on a non-unique permutation σ , such that the empirical degree based on this permutation is monotonically increasing.
The degree sorted adjacency matrix is denoted by G σ .
It is then transformed to a ‘histogram’ A ∈ R n 0 × n 0 as
[Equation]
1 , where h = (cid:22) n n(cid:23) and (cid:98)·(cid:99) is the ﬂoor function.
[Equation]
[Equation]
Proposed graph distance.
Given two directed or undirected graphs G 1 and G 2 with n 1 and nnodes, respectively, we apply the transformation (1) to both the graphs with n 0 ≤ min { n 1 , n 2 } .
We propose to use the graph distance
[Equation]
where A 1 and A 2 denote the transformed matrices and (cid:107) · (cid:107) F denotes the matrix Frobenius norm.
Proposition 1 shows that, if G 1 and G 2 are sampled from two graphons, then the graph distance (2) consistently estimates the L 2 -distance between the two graphons, which is deﬁned as
[Equation]
Proposition 1 (Graph distance is consistent)
Let w 1 and w 2 satisfy Assumptions 1–3.
Let G 1 and
G 2 be random graphs with at least n nodes sampled from the graphons w 1 and w 2 , respectively.
If n → ∞ and n 0 is chosen such that n 20 log n n → 0 , then with high probability (w.h.p.), [Equation] (cid:107) w
[Equation]
− w
[Equation]
(cid:107)
[Equation]
( G 1 , G 2 ) (cid:12)(cid:12) =
O (cid:16)
[Equation]
Proof sketch.
We deﬁne a novel technique for approximating the graphon.
The proof in Appendix A.1 ﬁrst establishes that the approximation error is bounded using Assumption 1.
Consequently, a relation between approximated graphons and transformed graphs is derived using lemmas from Chan & Airoldi (2014).
Proposition 1 is subsequently proved using the above two results.
(cid:3)
Remark on Proposition 1 for sparse graphs.
The deﬁned sampling procedure for graphon generates dense graphs, deviating from the real world sparse networks.
To adapt it to sparse graphs, one may modify the sampling procedure to G ij | U i , U j ∼ Bernoulli ( ρ w ( U i , U j )) where ρ depends only on n (Olhede & Wolfe, 2014).
Under this process, the consistency result in Proposition 1 remains unchanged for ρ = Ω( (cid:112) log n/n ) (proof in Appendix A.2).
This bound cannot be improved to the expected real world measure where ρ = Ω(log n/n ) , because of the degree sorting step in (1).
Nevertheless, our analysis allows for relatively sparser graphs with strong consistency result.
Notation.
For ease of exposition, Proposition 1 as well as main results are stated asymptotically using the standard O ( · ) and Ω( · ) notations, which subsume absolute and Lipschitz constants.
We use “with high probability” (w.h.p.) to state that the probability of an event converges to 1 as n → ∞ .
G RAPH C LUSTERING
We now present the ﬁrst application of the proposed graph distance (2) in the context of clustering network-valued data.
We are particularly interested in the setting where one needs to cluster a small population of large graphs, that is, minimum graph size n grows faster than the sample size m .
This scenario is relevant in practice as bioinformatics or neuroscience application often deals with very few graphs (see real datasets in Section 3.3).
Theoretically, this perspective complements guarantees for (graph) kernels that are applicable only in supervised setting and large sample regime, m → ∞ .
In contrast, our guarantees are more conclusive for bounded m and large graph size, n → ∞ .
3. D ISTANCE B ASED S PECTRAL C LUSTERING (DSC)
Given m graphs with adjacency matrices G 1 , ..., G m , we propose a distance based clustering algorithm where we apply spectral clustering to an estimated distance matrix.
The distance matrix (cid:98) D ∈ R m × m is computed on all pairs of graphs using the deﬁned estimator function (2), that is (cid:98) D ij =
d ( G i , G j ) .
Unlike the standard Laplacian based spectral clustering, which is applicable for adjacency or similarity matrices, we use the method suggested by Mukherjee et al.
(2017) that computes the K leading eigenvectors of (cid:98) D (corresponding to the K smallest eigenvalues in magnitude) and applies k-means clustering to the rows of the eigenvector matrix resulting in K number of clusters.
We refer to this distance based clustering algorithm as DSC, described in Algorithmof Appendix.
To derive the statistical consistency of DSC, we consider the problem of clustering m random graphs of potentially different sizes, each sampled from one of K graphons.
We establish the consistency in Theorem 1 by proving that the number of misclustered graphs → 0 asymptotically.
Theorem 1 (Consistency of DSC) Consider K graphons satisfying Assumptions 1–3, and m random graphs G 1 , . .
, G m , each sampled from one of the K graphons (assume there is at least one graph from each graphon).
Deﬁne the distance matrix D ∈ R m × m such that D ij = (cid:107) w i − w j (cid:107) Lwhere w i and w j are the graphons from which G i and G j are generated.
Let n be the size of the smallest graph, and γ be the K -th smallest eigenvalue value of D in magnitude.
As n → ∞ , if n 0 is chosen such that m 2 n 20 log n n → 0 , then DSC misclusters at most O (cid:16) m 3 γ 2 n 20 (cid:17) graphs w.h.p.
Proof sketch.
The proof, given in Appendix A.3, uses the Davis-Kahan spectral perturbation theorem to bound the error in terms of (cid:107) (cid:98) D − D (cid:107) F , which is further bounded using Proposition 1.
(cid:3)
While the number of misclustered graphs seem to depend on m 3 , we note that there is an inverse dependence on γ 2 which has dependence on m (see Corollary 1 that illustrates it for a speciﬁc case).
Moreover, our focus is on the setting where m = O (1) and n, n 0 → ∞ , in which case, the error asymptotically vanishes.
It is natural to wonder whether the dependence on m and n 0 is tight in the above bounds.
Currently, we do not know the optimal rates, but deriving this would be difﬁcult due to the strong dependency of entries in (cid:98) D and slow rate of convergence of the graph distance in Proposition 1.
The presence of γ in the above clustering error bound makes Theorem 1 less interpretable.
Hence, we also consider the speciﬁc case of K = 2 (two graphons) in the following result, along with the assumption that equal number of graphs are generated from both graphons.
Corollary 1 Let w (cid:54) =
w (cid:48) be two graphons satisfying Assumptions 1–3, and m is a bounded even number.
Assume that equal number of graphs are generated from w and w (cid:48) .
For any n 0 and large enough constant C such that (cid:107) w − w (cid:48) (cid:107) L 2 ≥ C mn 0 and m 2 n 20 log n n → 0 as n → ∞ , the number of graphs misclustered by Algorithm 1 goes to zero w.h.p.
The corollary implies that given the observed graphs are large enough, and if the choice of nis relatively small, n 0 (cid:28) (cid:112) n/ log n , and the graphons are Ω( 1 n 0 ) apart in L 2 -distance, then the clustering is consistent.
Intuitively, it can be understood that if we condense large graphs to a small representation (small n 0 ), then the clusters can be identiﬁed only if the models are quite dissimilar.
3. S IMILARITY B ASED S EMI -D EFINITE P ROGRAMMING (SSDP)
We propose another algorithm for clustering m graphs based on similarity between pairs of graphs.
The pairwise similarity matrix (cid:98) S ∈ R m × m is computed by applying Gaussian kernel on the distance between the graphs, that is (cid:98) S ij = exp (cid:16) − d
( G i ,G j ) σ i σ j (cid:17) , where σ 1 , . . .
, σ n are parameters.
For theoretical analysis, we assume σ 1 = . . .
= σ n is ﬁxed, but in experiments, the parameters are chosen adaptively.
We use the following semi-deﬁnite program (SDP)
(Yan et al., 2018; Perrot et al., 2020) to ﬁnd membership of the observed graphs.
Let X ∈ R m × m be the normalised clustering matrix, that is X ij = 1 / |C| if i and j belong to cluster C , and 0 otherwise.
Then, SDP for estimating X is:
[Equation]
where X ≥ 0 , X (cid:23) 0 ensure that X is a non-negative, positive semi-deﬁnite matrix, and 1 denotes the vector of all ones.
We denote the optimal X from the SDP as (cid:98) X .
Once we have (cid:98) X , we apply standard spectral clustering on (cid:98) X to obtain a clustering of the graphs.
We refer to this algorithm as
SSDP, described in Algorithm 2 of Appendix.
We present strong consistency result for SSDP below.
Theorem 2 (Consistency of SSDP) Consider K graphons, w 1 , . .
. , w K , satisfying Assumptions 1– 3, and m random graphs, each sampled from one of the K graphons.
Let n be the size of the smallest graph.
As n → ∞ , if n 0 is chosen such that m 2 n 20 log n n → 0 and min
[Equation]
(cid:17) , then the number of graphs misclustered by SSDP is zero w.h.p.
Proof sketch.
The proof in Appendix A.4 adapts Perrot et al.
(2020, Proposition 1) to the present setting and combines it with Proposition 1 to derive the stated condition for zero error.
(cid:3)
Theorem 2 is slightly stronger than Theorem 1, or Corollary 1, since SSDP achieve a zero clustering error for large enough graphs.
This theoretical merit of SDP over spectral clustering is known in the statistics literature.
Similar to Corollary 1, the choice of n 0 is important such that it does not violate the minimum L 2 -distance condition in the theorem to ensure consistency.
Remark on the knowledge of K .
Above discussions assume that the number of clusters K is known, which is not necessarily the case in practice.
To tackle this issue, one can estimate K using
Elbow method (Thorndike, 1953) or approach from Perrot et al.
(2020),and then use it as input in our algorithms, DSC and SSDP.
One can modify the SDP (4) and Theorem 2 to the case where K is adaptively estimated.
However, we found the corresponding algorithm, adapted from Perrot et al.
(2020), to be empirically unstable in the present context.
Hence, the knowledge of K is assumed in the following experiments, which also allows the efﬁcacy of the proposed algorithms and graph distance to be evaluated without the error induced by incorrect estimation of K .
3. E XPERIMENTAL A NALYSIS
In this section, we evaluate the performance of our algorithms DSC and SSDP, both in terms of accuracy and computational efﬁcacy.
We measure the performance of the algorithms in terms of error rate, that is, the fraction of misclustered graphs by using the source of the graphs as labels.
Since clustering provides labels up to permutation, we use the Hungarian method (Kuhn, 1955) to match the labels.
The performance can also be measured in terms of Adjusted Rand Index (results in Appendix C.3).
We use simulated and real datasets for evaluation and obtain all our experimental results using Tesla K80 GPU instance with 12GB memory from Google. Code available in Github.
Simulated data.
We generate graphs of varied sizes from four graphons, W 1 ( u, v )
= uv ,
W 2 ( u, v ) = exp (cid:8) − max( u, v ) 0 . 75 (cid:9) , W 3 ( u, v ) = exp (cid:8) − 0 .
5 ∗ (min( u, v ) + u 0 . 5 + v 0 .
5 ) (cid:9) and
W 4 ( u, v ) =
| u − v | .
The simulated graphs are dense and the graph sizes are controlled to study how algorithms scale.
Their corresponding L 2 distances between pairs of graphons is shown later in Figure 3 and the heatmap of the graphons are visualised in Figure 4 in Appendix.
Real data.
We use datasets from two contrasting domains: small molecule datasets from Bioinformatics and large network datasets from Social Networks.
We use Proteins (Borgwardt et al., 2005), KKI (Pan et al., 2016), OHSU
(Pan et al., 2016) and Peking 1 (Pan et al., 2016) datasets from Bioinformatics, and Facebook Ct1 (Oettershagen et al., 2020), Github Stargazers (Rozemberczki et al., 2020), Deezer Ego Nets (Rozemberczki et al., 2020) and Reddit Binary (Yanardag & Vishwanathan, 2015) datasets from Social Networks.
We sub-sample a few graphs from each dataset by setting a minimum number of nodes to validate clustering small number of large graphs (small m , large n ).
The number (#graphs) and size (#nodes) of the graphs are listed in Figure 1 tables.
We consider all combinations of the datasets for three and four clusters in both the domains separately.
Choice of n 0 and σ i .
As noted in our algorithms DSC and SSDP, n 0 is an input parameter.
Theorems 1 and 2 show the choice of n
= O (cid:0)(cid:112) n/ log n (cid:1) .
Hence, we set n 0 = (cid:112) n/ log n where n is the minimum number of nodes.
In Appendix C.2, we use simulated data to show that the above choice of n 0 is reasonable (if not the best) for both DSC and SSDP.
Furthermore, the similarity matrix (cid:98) S in SSDP is computed using parameters σ i and we set σ i =
d ( G i , G 5 nn ) where G 5 nn is the ﬁfth nearest neighbour of G i .
Hence, apart from knowledge of K , our algorithms are parameter-free.
Performance comparison with existing methods.
We compare our algorithms with a range of approaches for measuring similarity or distance among multiple networks.
Most methods discussed below provide a kernel or distance matrix to which we apply spectral clustering to obtain the clusters:
1) Network Clustering based on Log-Moments (NCLM) is an embedding based clustering strategy for different sized graphs (Mukherjee et al., 2017) that is based on network statistics called log moments.
Log moments for a graph with adjacency matrix A and number of nodes n is obtained by (log ( m 1 ( A )) , log ( m 2 ( A )) , . .
, log ( m J ( A )))
where m i ( A )
= trace( A/n ) i and J is a parameter.
2) Wasserstein Weisfeiler-Lehman Graph Kernels (WWLGK) is a recent graph kernel that is based on the Wasserstein distance between the node feature vector distributions of two graphs proposed by Togninalli et al. W 1 , W 2 , W 3 W 2 , W 3 , W 4 W 1 , W 3 , W 4 W 1 , W 2 , W 4 W 1 , W 2 , W [Figure] W
[Figure]
[Figure] [Figure] E rr o r r a t e
DSC
[Figure]
WWLGK GNTK
[Figure]
B 1 , B 2 , B
[Figure]
B 1 ,
[Figure]
B 1 , B
[Figure]
[Figure]
B 2 , B 3 , B 4 B 1 , B 2 ,
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] 3) Graph Neural Tangent Kernel (GNTK) is another graph kernel that describes inﬁnitely wide graph neural networks derived by Du et al. Both WWLGK and GNTK provide state-of-the-art performance in graph classiﬁcation with GNTK outperforming most graph neural networks.
4) Network Clustering algorithm based on Maximum Mean Discrepancy (NCMMD) considers a graph metric (MMD) to cluster the graphs.
MMD distance between random graphs is proposed as an efﬁcient test statistic for random dot product graphs (Agterberg et al., 2020).
We compute
MMD between the graphs that are represented by latent ﬁnite dimensional embedding called spectral adjacency embedding with the dimension r as a parameter.
In Network Clustering algorithm based on Graph Matching Metric (NCGMM), we match two graphs of different sizes by appending null nodes to the small graph (Guo et al., 2019) and compute
Frobenius norm between the matched graphs as their distance.
Although the graph metrics (MMD and graph matching) are for different purposes, we evaluate their efﬁcacy in the context of clustering.
The parameters in the algorithms are n 0 and σ i in DSC and SSDP, J in NCLM, number of iterations ( # itr ) to perform in WWLGK, number of layers ( # layer ) in graph neural networks for GNTK, r in NCMMD and none in NCGMM.
We ﬁx n 0 in our algorithms using the theoretical bound and σ i is set adaptively as discussed, whereas use grid search to tune the parameters for other algorithms.
Evaluation on simulated data.
We sample 10 graphs of varied sizes between 50 and 100 nodes from each of the four graphons in Figure 4, and perform the experiments by considering all combinations of three and four clusters of the graphons.
Based on the theoretical bound, n 0 is ﬁxed to 5 as n = 50 .
We report the performance for J = 8 , r = 3 , # itr = 1 and # layer = 2 as these produce the best results.
The ﬁrst row of Figure 1 shows the average performance of the algorithms computed over 5 independent runs.
We observe that our algorithm DSC outperforms all the other algorithms, achieving nearly zero error in all cases, and SSDP also performs competitively by standing second or third best.
The graph kernels, WWLGK and GNTK, and the graph metric based method NCGMM typically do not perform well.
NCMMD either performs very well or quite poorly.
We sample small graphs since otherwise GNTK cannot run due to memory requirement for dense large graphs and NCGMM has high computation time.
Appendix C.4 includes evaluation of the algorithms except GNTK and NCGMM on larger graphs, where we observe similar behaviour.
Evaluation on real data.
We consider all combinations of three and four clusters of both Bioinformatics and Social Networks separately.
The second and third rows of Figure 1 show the performance with n 0 = 30 , J = 8 , r = 3 , # itr = 1 and # layer = 2 , and the upper limit of 7200 seconds (hours) as running time of algorithms.
We observe DSC outperforms other algorithms by a large margin in majority of the combinations, while in the other combinations like { Proteins,KKI,Peking 1 } ,
DSC performs well with a very small margin to the best performing one.
Although NCLM and
GNTK compare favorably in Social Networks datasets, they typically have high error rate in Bioinformatics or simulated datasets, suggesting that they could be well suited for large networks, whereas
DSC is more versatile and suitable for all networks.
SSDP performs moderately on real data, but it achieves the smallest error in some cases, implying that it is suited for certain types of networks.
Computation time comparison.
Figure 2 shows the time (measured in seconds) taken by each algorithm for four clusters case, plotted in log scale.
Appendix C.5 illustrates similar behavior in three clusters case as well.
Our algorithms, DSC and SSDP, perform competitively with respect to time as well.
In addition, it scales effectively for large graphs unlike other algorithms.
[-W 1 , W 2 , W 3 , W 4 B 1 , B 2 , B 3 , B 4 S 1 , S 2 , S 3 , S 
 L o g T i m e ( s ) DSC (ours) SSDP (ours) NCLM-]It is worth noting that
[Figure]
although NCLM takes lesser time than DSC and SSDP for small graphs, it takes longer for large social networks datasets, thus favoring our methods in terms of both accuracy and scalability.
Graph matching based algorithm, NCGMM, has severe scalability issue showing the inapplicability of such methods to learning problems.
We also evaluate the scalability of all algorithms by measuring the time for clustering different sets of varied sized graphs from graphons W 1 , W 2 , W 3 and W 4 .
Detailed discussion on high scalability of DSC and SSDP is given in Appendix C.6.
G RAPH T WO -S AMPLE T ESTING
Inspired by the remarkable performance of our graph distance (2) in clustering, we analyse its applicability in graph two-sample testing.
Two-sample testing is usually studied in the large sample case m → ∞ , and several nonparametric tests are known that could also be applied to graphs.
However, it is also relevant to study the small sample setting in graphs, particularly m = 2 , that is, whether two large graphs are statistically identical or not (Ghoshdastidar et al., 2020; Agterberg et al., 2020).
We consider the following formulation of the graph two-sample problem, stated under the assumption that the graphs are sampled from graphons.
Given two random graphs, G 1 sampled from some model (here, graphon w 1 ), and G 2 from another model w 2 , the goal is to determine which of the following hypothesis is true: H 0 : { w 1 = w 2 } or H a : (cid:8) w 1 (cid:54) =
w 2 : (cid:107) w 1 − w 2 (cid:107) L 2 ≥ φ (cid:9) for some φ > 0 .
Existing works consider alternative random graph model, such as inhomogeneous Erd ˝ osR´enyi models or random dot product graph models, which are more restrictive.
The condition φ >is necessary if one only has access to ﬁnitely many independent samples (Ghoshdastidar et al., 2020).
A two-sample test T is a binary function of the given samples such that T = 1 denotes that the test rejects the null hypothesis H 0 and T = 0 implies that the test rejects the alternate hypothesis H a .
The goodness of a two-sample test is measured in terms of the Type-I and Type-II errors, which denote the probabilities of incorrectly rejecting the null and alternate hypotheses, respectively.
The goal of this section is to show that one can construct a test T that has arbitrarily small Type-I and Type-II errors.
For this purpose, we consider the test
[Equation]
for some ξ > 0 , where I {·} is the indicator function and d ( G 1 , G 2 ) is the proposed graph distance for some choice of integer n 0 .
We state the following theoretical guarantee for the test T .
Theorem 3 Assume that the graphons w 1 , w 2 satisfy Assumptions 1–3, and let the graphs G 1 ∼ wand G 2 ∼ w 2 have at least n nodes.
As n → ∞ , there is a choice of ξ such that the Type-I and
Type-II errors of the test T in (5) go to 0 if n 20 log n n → 0 and φ ≥ Cn 0 , where the constant C depends only on the Lipschitz constants.
Theorem 3 shows that the test T in (5) can distinguish between any pair of graphons that have separation (cid:107) w 1 − w 2 (cid:107) L 2 = Ω(1 /n 0 ) with arbitrarily small error, if the graphs are large enough.
Empirical analysis.
We validate the consistency result in Theorem 3 by computing power of the proposed test T , which measures the probability of rejecting the null hypothesis H 0 .
Intuitively, power of the test for graphs sampled from same graphons should be small (close to a pre-speciﬁed signiﬁcance level) since H 0 must not be rejected, whereas, it should be close to 1 for graphs sampled from different graphons.
As known in the testing literature, theoretical threshold, ξ in (5), is typically conservative in practice and the rejection/acceptance is decided based on p -values, computed using bootstrap samples.
To this end, we follow the bootstrapping strategy in Ghoshdastidar & von Luxburg (2018, Boot-ASE algorithm).
We also compare the test T by replacing d ( G 1 , G 2 ) in wwwwwwww0.00 0.25 0.22 0. wwwwwwww0.06 1.00 1.00 1. wwwwwwww0.04 0.99 0.90 0.
wwwwwwww0.57 1.00 1.00 1.
L
[Figure]
distance Proposed Distance Log Moments MMD with two other statistics, log moments (Mukherjee et al., 2017) and MMD, an efﬁcient test statistics for random dot product graphs (Agterberg et al., 2020).
We perform the experiment by sampling graphs G 1 ∼ w 1 and G 2 ∼ w 2 of size n and 2 n , respectively, where w 1 and w 2 are chosen from graphons W 1 , W 2 , W 3 , W 4 .
The power of T is computed for n = 100 (thus n 0 = 10 from the theoretical bound) and the signiﬁcance level 0 . 05 , averaged over 500 trials of bootstrapping 100 samples generated from all pairs of graphons.
The plots in Figure 3 show the average power of T with our proposed distance, log moments and MMD as d ( G 1 , G 2 ) , respectively.
From the results, it is clear that T using our distance can distinguish between graphons that are quite close too (smallest Ldistance for W 2 and W 3 ), whereas, other test statistics are weak as log moments statistic accepts [-H 
 even-]Heven when it is wrong (see W 1 and W 4 ) and MMD based test rejects it strongly almost always (see diagonal).
Appendix C shows similar result for small and large n , and evaluation on real datasets.
C ONCLUSION
There has been signiﬁcant progress in learning on complex data, including network-valued data.
However, much of the theoretical and algorithmic development have been in large sample problems, where one has access to m → ∞ independent samples.
Practical applications of network-valued data analysis often leads to small sample, large graph problems—a setting where the machine learning literature is quite limited.
Inspired by graph limits and high-dimensional statistics, this paper proposes a simple graph distance (2) based on non-parametric graph models (graphons).
Sections 3–4 demonstrate that the proposed graph distance leads to provable and practically effective algorithms for clustering (DSC and SSDP) as well as two-sample testing (5).
Extensive empirical studies on simulated and real data show that the clustering based on the graph distance (2) outperforms methods based on more complex graph similarities or metrics, both in terms of accuracy and scalability.
Figures 1–2 show that DSC achieves best performance for both small dense graphs (simulated graphons) as well as large sparse graphs (social networks).
On the other hand, popular machine learning approaches—graph kernels or graph matching—can be computationally expensive in large graphs and their performance may not improve as n → ∞ , see WWLGK in Figure 7.
Statistical approaches, such as the proposed clustering algorithms and two-sample test, show better performance on large graphs (Figures 1, 3 and Appendix C.4).
Theorems 1–3 theoretically support this observation by showing consistency of the clustering and testing methods in the limit of n → ∞ .
The theoretical results, however, hinge on Assumptions 1–3.
We remark that such smoothness and equivalence assumptions could be necessary for meaningful non-parametric approaches, which is also supported by the graph testing and graphon estimation literature.
Further insights about the necessity of smoothness assumptions would aid in theoretical and algorithmic development.
The poor performance of graph kernels and graph matching in clustering and small sample problems calls for further studies on these methods, which have shown success in network classiﬁcation.
Fundamental research, combining graphon based approaches and kernels, could lead to improved techniques.
Instead of sorting-and-smoothing graphon estimator, other histogram based methods that reduce graphons to a ﬁxed size by identifying block structures and averaging (Olhede & Wolfe, 2014) can be explored.
However, the distance deﬁned on such reduced graphons require alignment of the blocks between the graphons.
While such methods do not require Assumption 2 and allow sparse graphs, the computationally inefﬁcient alignment step pose a practical challenge.
Algorithmic modiﬁcations, such as estimation of K , would be also useful in practice.  A CKNOWLEDGEMENTS
This work has been supported by the German Research Foundation (Research Training Group GRK 2428) and the Baden-W¨urttemberg Stiftung (Eliteprogram for Postdocs project “Clustering large evolving networks”).
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Leena Chennuru Vankadara.
E THICS S TATEMENT
The proposed clustering algorithms for network-valued data can naturally be evaluated on realworld data, but no social interpretation about the results will be drawn.
Moreover, development of fair clustering algorithms is not the focus of this project.
R EPRODUCIBILITY S TATEMENT
The assumptions for the theory are stated clearly in Assumptions 1–3 and all the theoretical results, Proposition 1, Theorems 1-3, Corollary 1, are proved in detail in Appendix A.
The implementation of the considered algorithms with steps to reproduce the results is available in Githubhttps://tinyurl.com/3ycmmj4u.
Datasets used in the experiments are public and the links are provided in download datasets function of utils.py .
The experimental results can be reproduced by following graph clustering.ipynb .
A P ROOFS OF THEORETICAL RESULTS
We discuss the proofs of Proposition 1 and Theorems 1–3 with supporting lemmas in this section.
A. P ROPOSITION
The distance function deﬁned in (2) estimates the L 2 -distance between graphons that are continuous.
To prove this, we introduce a method to discretize the continuous graphons in the following so that it is comparable with the transformed graphs described in the graph distance estimator (2).
Graphon discretization.
We discretize the graphon by applying piece-wise constant function approximation that is inspired from Chan & Airoldi (2014), similar to the graph transformation.
More precisely, any continuous graphon w is discretized to a matrix W of size n 0 × n 0 with
[Equation]
We make Assumptions 1–3 to derive the concentration bound stated in Proposition 1.
The proof structure is as follows:
We bound the point-wise deviation of graphon with its discretized correspondence using Lipschitzness assumption (Lemma 1).
We derive the error bound between the L 2 -distance and Frobenius norm of the discretized graphons using Lemma 1 (Lemma 2).
We establish a relation between Frobenius norm of the histograms of graphons and graphs (Lemma 3).
Finally, we prove Proposition 1 by combining Lemmas 2 and 3.
Lemma 1 (Lipschitz condition)
For any graphon w and corresponding discretization W , deﬁne a piecewise constant function w ( x, y ) =
W ij where x ∈ [ in 0 , in 0 + 1 n 0 ] and y ∈ [ jn 0 , jn 0 + 1 n 0 ] .
Using the Lipschitz continuous assumption, we have
[Equation]
where L is the Lipschitz constant in Assumption
[Equation]
Proof of Lemma 1 (Lipschitz condition).
We use Assumption 1 on Lipschitzness to prove this lemma.
The following holds for a graphon w with Lipschitz constant L ,
[Equation]
We prove Lemma 1 using (8) and the deﬁnition of w ( x, y )
=
W ij where x ∈ [ in 0 , in 0 + 1 n 0 ] and y ∈ [ jn 0 , jn 0 + 1 n 0 ] ,
[Equation]
Lemma 2 (Error bound of discretization) For two graphons w 1 and w 2 , the error bound between the L
[Equation]
Proof of Lemma 2 (Error bound of the approximation).
Lemma 1 is used to prove this lemma.
Let L 1 and L 2 be the Lipschitz constants of w 1 and w 2 .
[Equation]
± w 1 ( x, y ) and ± w 2 ( x, y ) within the square, expand and apply Lipschitzness condition from Lemma
[Equation]
Similarly, by ± w 1 ( x, y ) and ± w 2 ( x, y ) to 1 n 20 (cid:107) W 1 − W 2 (cid:107) 2 F and applying Lipschitzness condition from Lemma 1 we get,
[Equation]
Combining (10) and (11), we prove
[Equation]
We derive the following relation between histogram of graphs and graphons by adapting lemmas from Chan & Airoldi (2014) for our problem and the error bound of discretization (9).
Lemma 3 Let G 1 ∼ w 1 and G 2 ∼ w 2 have respective graph transformations A 1 and A 2 .
Let Wand W 2 be the corresponding discretized graphons of w 1 and w 2 , respectively.
As n → ∞ and n 20 log n n → 0 , then for any (cid:15) > 0 ,
[Equation]
with probability converging to 1 .
Proof of Lemma 3.
The proof of this lemma is inspired from Chan & Airoldi (2014).
For i = { 1 , 2 } , let matrix W i of size n 0 × n 0 be the discretized graphon w i and let matrix (cid:98)
A i of size n 0 × n 0 be another transformation of graph G i based on the true permutation (cid:98) σ i , that is, (cid:98) σ i denotes the ordering of the graph G i based on the corresponding graphon w i .
In other words, the discretized graphon W i is the expectation of (cid:98) A i .
The reordered graph is denoted by G (cid:98) σi and
then (cid:98) A i is obtained by, ( (cid:98) A i ) kl
= 1 hh (cid:88) k 1 = h (cid:88)
l 1 =
( G (cid:98) σ i ) kh
+ k 1 ,lh + lwhere h = (cid:98) n n 0 (cid:99) and (cid:98)·(cid:99) is the ﬂoor function.
We bound (cid:107) A 1 − A 2 (cid:107) F using (cid:98)
A 1 , (cid:98)
A 2 , W 1 and W 2 as,
[Equation]
We have the following for all i using Assumption 2 on Lipschitzness of the expected degree function g ( u ) i of graphon w i and Lemma 3 of Chan & Airoldi (2014),
[Equation]
depends on Lipschitz constants of g
[Equation]
[Equation]
Applying Markov’s inequality to bound the probability P (cid:18)
[Equation]
using (14),
[Equation]
Thus asymptotically, as n → ∞ and n 20 log n n → 0 , for all i and any (cid:15) 1 > 0 ,
[Equation]
A
[Equation]
1 with probability converging to 1 .
From Lemma 4 of Chan & Airoldi (2014), we have the following for all i ,
[Equation]
Applying Markov’s inequality to bound the probability P (cid:18)
[Equation]
using (16),
[Equation]
Union bound
[Equation]
Again asymptotically as n → ∞ , for all i and any (cid:15) 2 > 0 , (cid:107) (cid:98) A
i − W
i (cid:107) F < (cid:15) 2 with probability converging to 1 .
Lets assume (cid:15) 2 = (cid:15) 1 = (cid:15) .
Substituting (15) and (17) in (13),
[Equation]
[Equation]
with probability converging to 1 as n → ∞ and n 20 log n n → 0 .
The lower bound can similarly be obtained,
[Equation]
A i − [Equation] i
[Equation]
i
[Equation]
i
[Equation]
F ≥ (cid:107) W
[Equation]
W 2 (cid:107)
[Equation]
[Equation]
(19) with probability converging to 1 as n → ∞ and n 20 log n n → 0 .
Thus, |(cid:107)
A 1 − A 2 (cid:107) F − (cid:107) W 1 − W 2 (cid:107) F | ≤ 4 (cid:15) satisfy for any (cid:15) > 0 with probability converging toas n → ∞ and n 20 log n n → 0 from equations (18) and (19).
(cid:3)
Proof of Proposition 1 (Graph distance is consistent).
Proposition 1 immediately follows from
Lemmas 2 and 3 after a simple decomposition step as shown below.
[Equation]
O (cid:18)n(cid:19) holds for any (cid:15) > 0 as n → ∞ and n 20 log n n → 0 .
(cid:3)
A. P ROPOSITION 1 F OR S PARSE G RAPHS
To adapt the dense graph model, graphon, to sparse graphs, the sampling procedure is modiﬁed to G ij |
U
i , U j ∼ Bernoulli ( ρ w ( U i , U j )) where ρ depends only on n .
With this modiﬁcation, equation 14 will be changed to
[Equation]
20) and the proof involves modiﬁcation to Lemma 2 of Chan & Airoldi (2014).
We state the modiﬁed lemma and brieﬂy sketch the proof below.
Modiﬁed Lemma 2 of Chan & Airoldi (2014).
Let σ ( i ) be the oracle permutation such that
U σ (1) <
U σ (2) < . . .
<
U σ ( n ) .
Then, if (cid:12)(cid:12)(cid:12)(cid:12) σ ( i ) n [Equation] (cid:12)(cid:12)(cid:12)(cid:12) < [Equation]
[Equation] (cid:114) log n n then (cid:12)(cid:12) [Equation] i )
[Equation]
) (cid:12)(cid:12) < (cid:114) [Equation] n (21) with probability at least 1 − 8 exp (cid:18) −18 Llog n ρ(cid:19) . Conversely, if (21) holds with probability at least 1 − 8 exp (cid:18)
−18 Llog n ρ(cid:19) then (cid:12)(cid:12)(cid:12)(cid:12) σ ( i )
[Equation]
with probability at least 1 − 40 exp
[Equation]
ρ
[Equation]
Changes to Proof in Chan & Airoldi (2014).
Suppose (cid:12)(cid:12)(cid:12) σ ( i ) n − σ ( j ) n (cid:12)(cid:12)(cid:12) <
δρ for δ > 0 and δ < ρ .
Then, P (cid:16)(cid:12)(cid:12) U σ ( i ) − U σ ( j ) (cid:12)(cid:12) > 3 δρ (cid:17) ≤ 4 exp (cid:16) − 2 n δ 2 ρ 2 (cid:17) .
Consequently,
P
[Equation]
g [Equation] U σ ( i ) )
[Equation]
g
[Equation] U σ ( j ) ) (cid:12)(cid:12) > [Equation] L 1 δ
[Equation]
≤ P
[Equation]
U σ ( i ) − U σ ( j ) (cid:12)(cid:12) > 3 δ ρ
[Equation]
ρ
[Equation]
(cid:19) .
Following their proof, we have
[Equation]
when ρ > 23 L 1 .
Note that there is a small mistake in their proof when Hoeffding’s inequality is applied, where a factor of n 2 is written in place of n .
Putting δ =
L 1 (cid:113) log nn and considering ρ = Ω (cid:18)(cid:113) log nn (cid:19) since ρ depends on n and δ <
ρ , we get
[Equation]
Converse can similarly be proved.
(20) can be derived by substituting the above changed lemma in their proof.
Consequently, Proposition 1 still holds under this formulation, with slight change to the condition as n → ∞ and n 20 ρ log n n → 0 for ρ = Ω (cid:32)(cid:114) log n n (cid:33) .
We will not consider sparsity ρ for analysing consistency of our algorithms in the next sections.
A. D ISTANCE B ASED S PECTRAL C LUSTERING (DSC)
We make Assumptions 1–3 on the K graphons to analyse the Algorithm 1.
We establish the consistency of this algorithm by deriving the number of misclustered graphs |M| through the following steps
We establish deviation bound between the estimated distance matrix (cid:98) D and the ideal distance matrix D (Lemma 4).
We formulate Davis-Kahan theorem in terms of the deviation bound using the result from Mukherjee et al.
(2017) (Lemma 5).
We derive the number of misclustered graphs from Lemma 5.
As stated previously, (cid:98) D in Algorithm 1 is an estimate of D ∈ R m × m , where we deﬁne D ij = (cid:107) w i − w j (cid:107) L 2 .
Note that D is a block matrix with rank K , since D ij = 0 for all G i , G j generated from same graphon, and equals the distance between the graphons i and j otherwise.
We derive the deviation bound for the distance matrix using Lemma 3 and the result is as follows.
Lemma 4 (Distance deviation bound)
As n → ∞ and n 2 0 log n n → 0 , we establish
[Equation]
with probability converging to 1 .
Proof of Lemma 4 (Distance deviation bound).
From Proposition 1 and the deﬁnitions of (cid:98) D ij and
D ij , it is easy to see that (cid:12)(cid:12)(cid:12) (cid:98) D ij − D ij (cid:12)(cid:12)(cid:12) =
O (cid:18)n(cid:19) with probability converging to 1 as n → ∞ and n 20 log n n → 0 .
Using Lemmas 2 and 3, and the deﬁnitions of (cid:98) D ij and D ij , we have
[Equation]
≤
[Equation]
[Equation]
≤
[Equation]
(cid:15) n
[Equation]
+ 4 √ 2 L nwith prob.
→ 1 asymptotically
Thus asymptotically, (cid:12)(cid:12)(cid:12) (cid:98) D ij − D ij (cid:12)(cid:12)(cid:12) =
O (cid:18)n(cid:19) with probability converging to 1.
Hence, (cid:107) (cid:98) D − D
(cid:107) F =
O (cid:18) m n(cid:19)
with probability converging to 1 as n → ∞ and m 2 n 20 log n n
→
A variant of Davis-Kahan theorem (Mukherjee et al., 2017) and the derived deviation bound (24) are used to prove the following lemma.
Lemma 5 (Davis-Kahan theorem)
Let V and (cid:98) V be the m × K matrices whose columns correspond to the leading K eigenvectors of D and (cid:98) D , respectively.
Let γ be the K -th smallest eigenvalue value of D in magnitude.
As n → ∞ and n 20 log n n → 0 , there exists an orthogonal matrix (cid:98) O such that,
[Equation]
with probability converging to 1 .
Proof of Lemma 5 (Davis-Kahan theorem).
A variant of Davis Kahan theorem from Proposition
A.2 of Mukherjee et al.
(2017) states the following for matrix D of rank K .
Let (cid:98) V and V be m ∗ K matrices whose columns correspond to the leading K eigenvectors of (cid:98) D and D , respectively, and γ be the K -th smallest eignenvalue of D in magnitude, then there exists an orthogonal matrix (cid:98) O of size K ∗ K such that,
[Equation]
The number of misclustered graphs is |M| ≤ 8 m T (cid:107) (cid:98) V (cid:98)
O − V (cid:107) 2 F where m T is the maximum number of graphs generated from a single graphon (Mukherjee et al., 2017).
Since m T
= O ( m ) , |M| =
O (cid:18) mγ 2 n(cid:19) by substituting (25) in |M| .
Hence proving Theorem 1.
Proof of Theorem 1.
The number of misclustered graphs |M| ≤ 8 m T (cid:13)(cid:13)(cid:13)(cid:98) V (cid:98) O − V (cid:13)(cid:13)(cid:13) 2 F from
Mukherjee et al.
Thus, we prove the theorem using Lemma 5.
That is, as n → ∞ and m 2 n 2 0 log n n → 0 ,
[Equation]
Proof of Corollary 1.
This corollary deals with a special case where K = 2 and equal number of graphs are generated from the two graphons w and w (cid:48) .
Therefore, m T in the number of misclustered graphs |M| is m/ 2 .
The ideal distance matrix D will be of size m × m with 0 and (cid:107) w − w (cid:48)
(cid:107) L 2 as entries depending on whether the samples are generated from the same graphon or not.
For such a block matrix D , the two non zero eigenvalues are ±
m 2 (cid:107) w − w (cid:48) (cid:107) L 2 .
Therefore, γ is m 2 (cid:107) w − w (cid:48) (cid:107) L 2 .
Corollary 1 can be derived by substituting the derived γ in the number of misclustered graphs |M| in Theorem 1 as shown below.
[Equation]
Let us assume (cid:107) w − w (cid:48) (cid:107) L 2 ≥ C m n 0 where C is a large constant, then as n → ∞ , m 2 n 20 log n n → 0 , |M| → 0 .
[Equation]
A. S IMILARITY B ASED S EMI -D EFINITE P ROGRAMMING (SSDP)
We make Assumptions 1–3 on the K graphons to study the recovery of clusters from Algorithm 2.
The proof structure for cluster recovery stated in Theorem 2 is as follows:
We establish deviation bound between the estimated similarity matrix (cid:98) S and the ideal similarity matrix S (Lemma 6).
We derive the recoverability condition by adapting Proposition 1 of Perrot et al
and the obtained deviation bound (Lemma 7).
The ideal similarity matrix S ∈ R m × m is symmetric with K × K block structure, and S =
Z Σ Z T where Z ∈ { 0 , 1 } m × K be the clustering membership matrix and Σ ∈ R K × K such that Σ ll (cid:48) represents ideal pairwise similarity between graphs from clusters C l and C l (cid:48) .
From the deﬁnition of S ij , Σ ll (cid:48) =
exp
[Equation]
where w l and w l (cid:48) are graphons corresponding to clusters C l and C l (cid:48) , respectively.
(cid:98) S is the estimated similarity matrix of S as mentioned earlier.
Since (cid:98) X ∈ R K × K is the normalised clustering matrix, (cid:98) X
= ZN − 1 Z T where N is a diagonal matrix with 1 |C 1 | , . .
, 1 |C K | .
We derive the deviation bound for the similarity matrices using Lemma 3 and the result is as follows.
Lemma 6 (Similarity deviation bound)
As n → ∞ , n 20 log n n → 0 , we establish
[Equation]
[Equation]
with probability converging to 1 .
Hence, from the result (cid:107) (cid:98) S − S (cid:107) F = O [Equation] n(cid:19) with probability converging to 1 .
Proof of Lemma 6 (Similarity deviation bound).
We derive the bound using Lemmas 2 and
[Equation]
and the deﬁnitions of (cid:98) S ij and S ij .
[Equation]
Consider σ i = σ j = σ
[Equation]
with probability → 1 asymptotically
[Equation]
F n
[Equation]
[Equation]
n
[Equation]
with probability → 1 asymptotically 2 ≤ exp (cid:18) −(cid:107)
[Equation]
(cid:32)
[Equation]
n
[Equation]
(cid:33) exp( x
[Equation]
1 +
[Equation]
x for x >≤
[Equation]
ij (cid:32)
[Equation]
n
[Equation]
1] ≤ S ij + 8( (cid:15) + √
[Equation]
L ) n 0 σ(28)
Thus, from (27) and (28), we get | (cid:98) S ij − S ij | ≤ 8( (cid:15) + √ 2 L ) n 0 σ= O (cid:18)n(cid:19) for any (cid:15) , with probability converging to 1 as n → ∞ and n 20 log n n → 0 .
Hence, (cid:107) (cid:98) S − S (cid:107) F ≤ 8 m ( (cid:15) + √ 2 L ) n 0 σ= O (cid:18) m n(cid:19) , with probability converging to 1 as n → ∞ and m 2 n 20 log n n → 0 .
(cid:3)
The condition for exact recovery of clusters is derived by adapting Proposition 1 of Perrot et al.
The proposition states the recoverability condition for such an SDP deﬁned in (4) in terms of the similarity deviation bound.
Thus, we use the derived bound in Lemma 6 and establish condition on the L 2 -distance to satisfy the proposition from Perrot et al.
First, we state the adapted proposition.
We deﬁne ∆ 1 and ∆ 2 as,
[Equation]
and
[Equation]
Then, the following should be satisﬁed for (cid:98) X to be the unique optimal solution of the SDP in (4):
[Equation]
The minimum cluster size min l |C l | in our case is 1 .
Consequently, the recoverability condition is derived and is as follows.
Lemma 7 (Recoverability of clusters) As n → ∞ , m 2 n 20 log n n → 0 , the
[Equation]
2 should be Ω (cid:18) m n(cid:19) so that (cid:98) X is the unique optimal solution of the SDP (4) .
Proof of Lemma 7 (Recoverability of clusters).
We derive the condition to satisfy the stated proposition.
[Equation]
| .
The minimum cluster size in our case can be 1 .
The analyses of the two cases of the Proposition is as follows.
Case 1.
Let us assume ∆ 2 ≤ ∆ 1 12 , then min (cid:8) ∆ 1 2 , ∆ 1 − 6∆ 2 (cid:9) will be ∆ 1 2 .
Therefore,
[Equation]
Case 2.
Let us assume ∆ 2 > ∆ 1 12 , then min (cid:8) ∆ 1 2 , ∆ 1 − 6∆ 2 (cid:9) will be ∆ 1 − 6∆ 2 .
Therefore, [Equation] L )
[Equation]
n
[Equation]
with probability → 1 aymptotically min l (cid:54) = l
[Equation] (cid:107) w l − w l (cid:48) (cid:107)
Lσ
[Equation]
n
[Equation]
Thus, from (29) and (30), we must satisfy min l (cid:54) = l (cid:48) (cid:107) w l − w l
[Equation]
L
[Equation]
Ω
[Equation] m n [Equation]
(cid:19) for the Proposition to hold.
Consequently, Theorem 2 is the direct reﬂection of this lemma.
(cid:3)
A. G RAPH T WO -S AMPLE T ESTING
Theorem 3 of two-sample testing is proved by deriving the probability of Type-1 and Type-2 errors.
We make Assumptions 1–3 for this case.
Let W 1 and W 2 be the n 0 × n 0 discretized graphons of wand w 2 , respectively, obtained using (6).
Then, the alternate hypothesis H a can be rewritten using
Lemma 2 in the following way,
[Equation]
We derive the probability of the errors using Lemma 3 and is stated in the following lemmas.
Lemma 8 (Probability of Type-1 error)
The probability of Type-1 error, i.e. rejecting the null hypothesis when it is actually true, is
[Equation]
where C depends only on the Lipschitz constants.
Proof of Lemma 8 (Probability of Type-1 error).
The Type-1 error is rejecting H 0 when it is true.
Therefore, in this scenario, (cid:107) W 1 − W 2 (cid:107) F = 0 .
Thus, from Lemma 3, (15) and (17), we have (cid:107) A 1 − A 2 (cid:107) F ≤ 4 (cid:15) with 1 − C (cid:15) 2 n 20 log n n probability.
Therefore, the probability of Type-1 error is,
P
[Equation]
T
[Equation]
1 | H 0 : True ) =
P ( d ( G
[Equation]
G 2 ) ≥ ξ )
[Equation]
(cid:107) F ≥ n 0 ξ ) err only when n 0 ξ ≤ 4 (cid:15) (
[Equation]
nn
[Equation]
n n
[Equation]
Lemma 9 (Probability of Type-2 error)
The probability of Type-2 error, i.e. accepting the null hypothesis when the alternate hypothesis is actually true, is
[Equation]
where C 1 and C 2 depend only on the Lipschitz constants.
Proof of Lemma 9 (Probability of Type-2 error).
The Type-2 error is evaluating to null hypothesis when the alternate hypothesis is true.
Therefore, from (31) (cid:107) W 1 − W 2 (cid:107) F ≥ ρ .
From Lemma 3, (15) and (17),
[Equation]
with probability 1 −
[Equation]
[Equation]
The probability of Type-2 error is,
[Equation]
err only when n 0 ξ ≤ ρ − 4 (cid:15) ; let n 0 ξ = 4 (cid:15)
[Equation]
n
We get the probability by substituting ρ n 0 = φ −√
[Equation] nfrom (31) in the above equation.
Theorem 3 can be proved by asymptotic analysis of Lemmas 8 and 9.
(cid:3)
B DSC AND SSDP A LGORITHMS
The proposed algorithms DSC and SSDP are described as follows:
Algorithm 1: Distance based Spectral Clustering (DSC) input : Adjacency matrices G 1 , ..., G m , histogram size noutput: K clusters C 1 , ..., C K
Construct distance matrix Compute (cid:98) D ∈ R m × m , where (cid:98) D ij =
d ( G i , G j )
Clustering Apply spectral clustering to (cid:98) D with K number of clusters resulting in
C 1 , ..., C K
Algorithm 2: Similarity based Spectral Clustering (SSDP) input : Adjacency matrices G 1 , ..., G m , histogram size noutput: K clusters C 1 , ..., C K
Construct similarity matrix Compute (cid:98) S ∈ R m × m , where (cid:98) S ij = exp (cid:16) − d
G i ,G j ) σ i σ j (cid:17) with σ 1 = . . .
= σ n
Clustering Find (cid:98) X using (4) and apply standard spectral clustering to (cid:98) X resulting in
C 1 , ..., C K .
C E XPERIMENTAL D ETAILS
In this section, we present experimental details and additional experiments.
C. S IMULATED D ATA - H EATMAP OF G RAPHONS
Figure 4 shows the heatmap of the considered four graphons W 1 , W 2 , W 3 and W 4 .
We sample graphs from these graphons for the experiments.
[Figure]
C HOICE OF n
[Figure]
We validate the theoretically deduced bound for n
= O ( (cid:112) n/ log n ) by sampling 5 graphs with a ﬁxed number of nodes n from each of the four graphons, in total 20 graphs, and measuring the performance of DSC and SSDP for different n 0 = { 5 , 10 , 15 , 20 , 25 , 30 } .
We perform three simulations with n = { 50 , 100 , 500 } and ﬁx neighbourhood of one in SSDP.
Figure 5 shows the average error of both the algorithms over 5 independent trials.
Based on the theoretical considerations for n 0 ( (cid:28) (cid:112) n/ log n ), we evaluate n 0 = { 5 , 7 , 15 } for n = { 50 , 100 , 500 } , respectively.
The experimental results show that the derived bound for n 0 serves as a reasonable choice (if not the best) for both DSC and SSDP irrespective of n .
Hence, the choice of n 0 can be deterministic and adaptive with respect to n , thus making our algorithms parameter-free.
C. E XPERIMENTAL RESULTS USING A DJUSTED R AND I NDEX (ARI)
In this section, we provide the results for evaluation of algorithms on simulated and real data under the same setting as described in Section 3.3.
Figure 6 shows the evaluation of all the discussed algorithms using ARI, where the observations made from error rate hold.
[Figure] [Figure]
C. E VALUATION ON LARGE SIMULATED DATA
As mentioned in Section 3.
[Figure]
we evaluate algorithms except GNTK and NCGMM on large graphs sampled from the four graphons W 1 , W 2 , W 3 and W 4 with nodes between 100 and 1000 .
Figureshows the results measured using average error rate and average ARI, respectively.
The proposed algorithm DSC outperforms the others in all the case and SSDP stands second or third best, as observed in simulated data with small graphs.
[Figure]
C. C OMPUTATION T IME OF A LGORITHMS
Table 1 shows the time (measured in seconds) taken for the algorithms on the considered dataset combinations.
Three clusters also show similar behavior to four clusters.
[Table]
TLE
S 1 , S 2 , S1.
TLE
S 1 , S 3 , S1.
TLE
S 2 , S 3 , S1.
TLE
S 1 , S 2 , S 3 , S1.
TLE
C. S CALABILITY EXPERIMENT
We evaluate the scalability of the considered algorithms using simulated data by measuring the time taken for clustering 40 random graphs, 10 sampled from each of the graphons W 1 , W 2 , W 3 and
W 4 .
We did 7 experiments in which the size of the sampled graphs are varied as [50 , max size ] where max size = { 100 , 200 , 300 , 400 , 500 , 600 , 700 } .
Figure 8 shows the experimental results which illustrates high scalability of DSC, SSDP and NCLM over other algorithms.
Note that the experiment shows NCLM as scalable as DSC and SSDP since the sampled graphs are small.
[Figure] [Figure]
In this section, we evaluate the efﬁcacy of the proposed test T with different d ( G 1 , G 2 ) by varying the graph sizes n .
We consider n = { 50 , 100 , 150 } and ﬁx n 0 = 10 from the theoretical bound for evaluating the test T .
The power is computed using the test T for the signiﬁcance level 0 .
05 , and the plots in Figure 9 show the average power computed over 500 trials of bootstrappingsamples generated from all pairs of graphons for d ( G 1 , G 2 ) as our proposed distance, log moments
[Table]
graph ) graph
P r o p o s e d
D i s t a n c e
L o g
M o m e n t s
[Figure]
and MMD, respectively.
From the result for graph sizes (50 , 100) , we observe that the graphon pair ( W 2 , W 3 ) is not easily distinguishable (low H 0 rejection probability), which can be explained by their respective L 2 distance that is shown in the left plot of Figure 3.
This issue does not arise in testing larger graphs as the result shows for graph sizes (100 , 200) and (150 , 300) .
Therefore, test T with the proposed distance can distinguish between pairs of graphons that are quite close provided that the observed graphs are sufﬁciently large, thus proving to be consistent.
On the other hand, log moments and MMD based tests show weakness in distinguishing the graphons, where log moments based test T accepts the null hypothesis in most cases even when the graphons are different for all graph sizes.
For instance, the result for graphon pair W 1 and W 4 is indistinguishable using log moments statistic for any graph size.
On the contrary, MMD based test T rejects the null hypothesis almost always for larger graphs (diagonal values in all graph cases).
Thus, we conclude that the proposed test T in 5 is consistent and this experiment illustrates the efﬁciency of the test T compared to other plausible test statistics.
Subsequently, we evaluate the efﬁcacy of the above tests on the discussed real datasets – Bioinformatics and Social Networks.
We consider graphs from a dataset to belong to a population and hence the objective of the test statistic is to distinguish graphs from different populations, that is, graphs from two different datasets.
Since the populations are not known and the real graphs are treated as representatives of the population, we compute p -value of the test instead of power to measure the efﬁcacy.
The p -value is the evidence for rejecting the null hypothesis which implies that the smaller the p -value, the stronger the evidence that the null hypothesis should be rejected.
Therefore, the pvalue should be high (greater than the signiﬁcance level) for graphs from same population and low ( (cid:39) 0 ) for graphs from different populations.
Figure 10 shows the result for both the dataset cases and different tests.
From the results, it is clear that the log moments and MMD based tests are poor
B
B B
B B
B B
B
Proposed Distance
Log Moments
MMD
B B
B B
B B
B
B 4 B
B
B B
S S
S S
S S
S S
S S
S S
B i o i n f o r m a t
i c s
S o c i a l N e t w o r k s
S
S S
S S
S
S S
B B
B B
S S
S
S and inefﬁcient on real datasets as log moments based test has high acceptance of null hypothesis for almost all the pair of graphs from any population and MMD based test rejects the null hypothesis always except when the graphs are the same.
Whereas, the test using our proposed distance perform well on large graphs from Social Networks datasets, for instance, S 4 with other datasets and within itself.
This test performs well even for small graphs when compared to the other two tests.