T RADING Q UALITY FOR E FFICIENCY OF G RAPH P ARTITIONING : A N I NDUCTIVE M ETHOD ACROSS G RAPHS
Anonymous authors
Paper under double-blind review
A BSTRACT
Many applications of network systems can be formulated as several NP-hard combinatorial optimization problems regarding graph partitioning (GP), e.g., modularity maximization and NCut minimization.
Due to the NP-hardness, to balance the quality and efﬁciency of GP remains a challenge.
Existing methods use advanced machine learning techniques to obtain high-quality solutions but usually have high complexity.
Some fast GP methods use heuristic strategies to ensure low runtime but suffer from quality degradation.
In contrast to conventional transductive GP methods applied to a single graph, we propose an inductive graph partitioning (IGP) framework across multiple associated graphs of a system or scenario to alleviate the NP-hard challenge.
IGP ﬁrst conducts the ofﬂine training of a dual graph neural network on historical graph snapshots to capture properties of the system.
The trained model is then generalized to newly generated graphs for fast high-quality online GP without additional optimization, where a better trade-off between quality and efﬁciency is achieved.
IGP is also a generic framework that can capture the permutation invariant GP ground-truth of historical snapshots in the ofﬂine training and tackle the online GP on graphs with non-ﬁxed number of nodes and clusters.
Experiments on a set of benchmarks demonstrate that IGP achieves competitive quality and efﬁciency over various state-of-the-art baselines.
I NTRODUCTION
For various complex systems, e.g., communication networks, graph is a generic model to describe the entities and their relations using a set of nodes and edges.
Graph partitioning (GP) is a classic inference task that aims to partition the nodes of a graph into several groups (i.e., clusters) with dense linkage distinct from other groups.
Since the extracted clusters are believed to correspond to several real-world substructures of a system, e.g., cells in wireless networks (Dai & Bai, 2017), many network applications are formulated as GP tasks (Qin et al., 2019; Patil & Kulkarni, 2021).
Mathematically, GP can be described as some NP-hard combinatorial optimization problems, e.g., modularity maximization (Newman, 2006) and normalized cut (NCut) minimization (Von Luxburg, 2007).
Due to the NP-hardness, to balance the quality and efﬁciency of GP remains a challenge but some real applications have both requirements of high quality and low runtime, e.g., accurate GP on a wireless cellular network with several thousand nodes in a few seconds (Dai & Bai, 2017).
On the one hand, prior work has demonstrated the ability of machine learning (ML) techniques to achieve high GP quality.
Typical ML-based methods include non-negative matrix factorization (Wang et al., 2011) and generative probabilistic models (Karrer & Newman, 2011).
Graph embedding emerges as a promising technique for GP in recent studies.
These methods ﬁrst learn lowdimensional node representations via random walk (Perozzi et al., 2014) or deep learning (Wang et al., 2016;
Yang et al., 2016) to capture high-order proximities and nonlinear characteristics of a graph.
The GP result is derived by feeding the learned embedding to a downstream clustering module (e.g, K Means).
However, most ML-based methods have to use iterative optimization algorithms (e.g., gradient descent) to learn large-scale parameters for each graph with high complexity.
On the other hand, to reduce the overall runtime and satisfy real-time constraints of some applications is another major focus.
Some fast GP methods apply heuristic strategies to approximate conventional objectives, e.g., greedy modularity maximization (Blondel et al., 2008) and multi-level coarsening for graph-cut minimization (Dhillon et al., 2007).
Several graph embedding approaches also adopt fast approximation to reduce the inference time, e.g., random projection for high-order proximities (Zhang et al., 2018a).
Despite their high efﬁciency (e.g., low runtime), they may suffer from unexpected quality decline due to the inherent information loss of heuristic approximation.
Most state-of-the-art (SOTA) GP methods are transductive , which are optimized to capture structural properties of a static graph and can only tackle GP on such a unique graph.
We try to achieve a better trade-off between quality and efﬁciency from a new inductive perspective across multiple associated graphs of a system or scenario .
An inductive graph partitioning (IGP) framework is proposed based on the fact that most real-world complex systems generate a set of graphs via common knowledge in terms of several underlying distributions , e.g., power-law distributions.
This hypothesis is also adopted in the simulation of various network systems (Wehrle et al., 2010).
The multiple graphs can be (i) associated snapshots evolving over time or (ii) independent (sub)graphs of a scenario without temporal dependency e.g., multiple ego-nets in social media.
IGP ﬁrst trains a high-quality GP model on historical snapshots in an ofﬂine way, which aims to fully capture properties of the system or scenario regardless of time cost.
The trained model is then generalized to newly generated graphs for online GP without additional optimization, which signiﬁcantly reduces the runtime on these new graphs and is believed to have high-quality GP results.
For instance, we can conduct the ofﬂine training of IGP on existing known ego-nets of a social network and generalize it to new unseen ego-nets.
In real applications, it is usually assumed that we have enough time to prepare a high-quality model using historical known data in an ofﬂine way, which is also one-timeeffort only.
Our focus is to achieve a better trade-off between quality and efﬁciency of online GP on new graphs after deploying the trained model to a system with model parameters ﬁxed.
Technically, IGP follows a graph embedding scheme for unsupervised node-level tasks (i.e., GP) across graphs using the inductive nature of graph neural networks (GNN).
Although some inductive
GNNs have the potential to be generalized to new unseen nodes and graphs (Hamilton et al., 2017;
Veli ˇ ckovi ´ c et al., 2018), they still suffer from the following limitations for GP.
In this study, we consider GP on multiple graphs where topology is the only available information source, i.e., without graph attributes.
As some systems allow the addition and deletion of entities, we assume that the number of nodes N is non-ﬁxed for multiple graphs.
Although existing inductive
GNNs can be applied to graphs with non-ﬁxed N , they were originally designed for attributed graphs and rely on ﬁxed dimensionality of node features.
Our preliminary experiments indicate that some classic settings of GNN for the case without node features (e.g., using a constant matrix with ﬁxed dimensionality as the feature input) may suffer from poor GP quality.
Some methods extract features via dimensionality reduction (e.g., PCA (Nazi et al., 2019)) to map the topology (e.g., adjacency matrix) with non-ﬁxed N to a ﬁxed-dimensional feature space, which is still time-consuming.
Moreover, different system snapshots can be assigned with different number of clusters K .
Some inductive GNN based approaches adopt an end-to-end (E2E) framework to achieve a better approximation to classic GP objectives (e.g., GAP (Nazi et al., 2019) for NCut minimization and ClusNet (Wilder et al., 2019) for modularity maximization), with partitioning results derived via a fullyconnected (FC) output layer.
Despite their high quality, they can only tackle the inductive GP across graphs with ﬁxed K , due to ﬁxed dimensionality of the output layer.
These E2E methods still need to be trained from scratch for new graphs with non-ﬁxed K , which is time-consuming.
Although prior studies have demonstrated the ability of inductive GNNs to tackle (semi-)supervised tasks (e.g., node classiﬁcation) on new unseen nodes and graphs (Hamilton et al., 2017; Veliˇckovi´c et al., 2018), few of them consider unsupervised node-level tasks (e.g., GP) across graphs.
Our experiments also indicate that some standard settings of inductive GNNs for unsupervised tasks (e.g., using classic unsupervised training loss of graph embedding for GNN (Hamilton et al., 2017)) may still lack robustness for the online GP on newly generated graphs.
In addition to obtaining a better trade-off between quality and efﬁciency over conventional transductive GP methods via a novel inductive framework across graphs , we also make the following contributions to address the aforementioned limitations of inductive GNNs.
(i) To tackle the online GP with non-ﬁxed N , we develop an efﬁcient feature extraction module for inductive GNNs via graph coarsening.
(ii)
In contrast to E2E methods, IGP adopts an inductive graph embedding scheme across graphs which can tackle the online GP with non-ﬁxed K .
(iii) Note that GP is an unsupervised node-level task, where the cluster labels are permutation invariant, e.g., label assignments ( l 1 , l 2 , l 3 ) = (1 , 2 , 2) and ( l 1 , l 2 , l 3 ) = (2 , 1 , 1) are the same in terms of GP with l i as the cluster label of node v i .
IGP can further incorporate such label information (e.g., GP ground-truth) of historical graphs to the ofﬂine training by combining several GP objectives (e.g., modularity maximization and NCut minimization) with a novel dual GNN structure which ensures strong robustness for the online GP on new graphs.
P ROBLEM S TATEMENTS AND P RELIMINARIES
In this study, we consider GP on a set of graphs S =
{G 1 , G 2 , · · · , G T } from a common system or scenario.
Each graph G t ∈ S can be represented as G t =
( V t , E t ) with V t
= { v t 1 , . . .
, v tN t } and E t = { ( v ti , v tj ) (cid:12)(cid:12) v ti , v tj ∈ V t } as the sets of nodes and edges.
For each G t , topology is the only available information source without graph attributes.
We use an adjacency matrix A t ∈ (cid:60) N t × N t to describe its topology with N t nodes, where ( A t ) ij =
( A t ) ji = 1 if ( v ti , v tj ) ∈ E t and ( A t ) ij =
( A t ) ji = 0 otherwise.
Since some systems allow addition and deletion of entities, we assume that different snapshots in S can have different node sets, i.e., ∃G t , G s ∈ S s.t.
V t (cid:54) =
V s .
Since there may be no temporal dependency among {G 1 , · · · , G T } in some cases (e.g., multiple ego-nets in social media), we do not consider the node correspondence among {V 1 , · · · , V T } .
We follow the hypothesis adopted in the simulation of various network systems (Wehrle et al., 2010) that graphs in S are independently generated via several underlying distributions of a common system or scenario .
Graph Partitioning.
Given a graph G t , GP aims to partition the node set V t into K t subsets (i.e., clusters) C t =
{ C t 1 , · · · , C tK t } so that (i) within each cluster the linkage is dense but (ii) between different clusters the linkage is relatively loose.
The result C t also satisﬁes C tr
∩ C ts = ∅ for ∀ r (cid:54)
= s .
Mathematically, GP can be formulated as several NP-hard combinatorial optimization problems, e.g., NCut minimization (Von Luxburg, 2007) and modularity maximization (Newman, 2006).
Given a graph G t , NCut minimization aims to get the GP result C t that minimizes the NCut metric:
[Equation]
where ¯ C tr = V t − C tr is the complementary set of C tr ; cut( C tr , ¯ C tr )
= (cid:80) v i ∈ C tr ,v j ∈ ¯ C tr
( A t ) ij is the cut between C tr and ¯ C tr ; vol( C tr )
= (cid:80) v i ,v j ∈ C tr
( A t ) ij is the volume of C tr .
The objective (1) can be equivalently expressed in the following matrix form:
[Equation]
where L t =
I N t − D − 0 .
5 t
A [Equation] D
[Equation] [Equation] is the Laplacian matrix of A t ; D t = diag( d t 1 , · · · , d tN t ) is a diagonal matrix with d ti = (cid:80)
j ( A t ) ij ; I N is an N -dimensional identity matrix.
H t ∈ (cid:60) N t × K t is the membership indicator, where ( H t ) ir =
[ d ti · vol( C tr ) − 1 ] 0 .
5 if v ti ∈ C tr
and ( H t ) ir = 0 otherwise.
Modularity maximization is another conventional NP-hard objective of GP.
Given a graph G t , it aims to ﬁnd a partition C t that maximizes the modularity metric:
[Equation]
where e = (cid:80) i d ti / 2 is the number of edges.
The objective can also be rewritten in a matrix form:
[Equation]
where Q t ∈ (cid:60) N t × N t is the modularity matrix with ( Q t ) ij =
( A t ) ij − d ti d tj / (2 e ) .
H t ∈ (cid:60) N t × K t is the membership indicator, where ( H t ) ir =
if v ti ∈ C tr and ( H t ) ir = 0 otherwise.
Inductive Graph Embedding.
Given a single graph G t , conventional transductive graph embedding aims to learn a function f : { v ti } (cid:55)→ { u ti ∈ (cid:60) 1 × k } that maps each node v ti to a k -dimensional vector u t i .
The key characteristics of G t (e.g., clustering structure) should be preserved in the embedding space, where a node pair ( v ti , v tj ) with similar properties (e.g., in the same cluster) should have similar representations ( u ti , u tj ) (e.g., with close distance in the embedding space).
The learned representations { u ti } are used as the input of several downstream tasks on G t including GP.
To alleviate the NP-hard challenge of GP, we consider a novel inductive graph embedding scheme, where model parameters of f are shared by all the graphs in S .
We divide S into a training set [Figure] [Figure] Γ ⊂ S and a test set Γ (cid:48) =
S − Γ , which represent the sets of historical known and newly generated graphs.
We ﬁrst train f on Γ in an ofﬂine way to fully capture structural properties of historical graphs.
Although graph embedding and GP are unsupervised procedures, we assume the permutation invariant GP result C t (i.e., label information) of each historical graph G t ∈ Γ is available in the ofﬂine training, which can be used to enhance the ofﬂine embedding optimization.
In general, the training label information { C t |G t ∈ Γ } can be from the (i) ground-truth of system applications (e.g., cells in wireless cellular network decomposition) or (ii) results of some high-quality (but usually time-consuming) GP baselines.
It indicates another application that we can use GP results of some strong baselines to regularize the ofﬂine training of IGP.
After the training, we directly obtain the embedding { u t (cid:48) i } from f for each new graph G t (cid:48)
∈ Γ (cid:48) with model parameters ﬁxed.
The online GP on a new graph G t (cid:48) is the downstream task, where we assume the number of clusters K t (cid:48) is given.
The GP result of G t (cid:48) is derived by applying a clustering algorithm (e.g., K Means) to { u t (cid:48) i } .
M ETHODOLOGY
We propose an IGP framework with the overview shown in Fig. 1, including the (i) ofﬂine training of a dual GNN structure on historical graphs and (ii) online generalization to new graphs via an inductive graph embedding scheme.
Fig. 2 gives a running example of the ofﬂine training in IGP.
Model Architecture.
Inspired by the adversarial auto-encoder (AAE) (Makhzani et al., 2015), we introduce a novel dual GNN structure, including a generator G and a discriminator D that are jointly optimized via an adversarial process.
To the best of our knowledge, the dual GNN structure is our original design, which is different from SOTA AAE-based graph embedding methods (Pan et al., 2018) that regularize the embedding via the adversarial process between a probability distribution.
In contrast, our dual GNN structure incorporates the permutation invariant label information of historical graphs {G t ∈ Γ } to the learned embedding via (i) a parameter sharing strategy and (ii) an adversarial process between {G t } and auxiliary label-induced graphs {G ( g ) t } .
Concretely, in the ofﬂine training, we assume the topology and GP result (used as the ‘ground-truth’) of each historical graph G t are available, which are described by an adjacency matrix A t ∈ (cid:60) N t × N t and an indicator matrix R t ∈ (cid:60) N t × K t .
We deﬁne ( R t ) ir = 1 if node v ti is in cluster C tr by the ‘ground-truth’ and ( R t ) ir = 0 otherwise.
Due to the disjoint constraint of GP, only one entry in each row of R t is 1 with other entries in the same row set to 0 .
In addition to the original graph G t , an auxiliary label-induced graph G ( g ) t , with A ( g ) t =
R t R T t as the adjacency matrix describing its topology, is introduced to encode the structure of ‘ground-truth’ based on the following Fact 1 .
( Fact 1 ) For a historical graph G t with K t clusters, its auxiliary label-induced graph G ( g ) t has
K t fully connected components with each component corresponding to one unique cluster of G t .
Namely, there is an edge between nodes v ti and v tj in G ( g ) t only when they are in the same cluster, thus encoding the label information of ‘ground-truth’ (see Appendix A for the proof).
In the example of Fig. 2 with 8 nodes { 1 , 2 , · · · , 8 } , G ( g ) t has 2 fully connected components { 1 , 2 , 3 , 4 } and { 5 , 6 , 7 , 8 } w.r.t. the 2 clusters in the GP result (i.e., ‘ground-truth’) of G t .
The generator G includes a feature encoder and a label-induced encoder , forming a dual GNN structure with shared parameters δ G .
The feature encoder takes A t and a feature matrix X t as inputs, and derives graph embedding U t , i.e., U t =
G ( A t , X t ; δ G ) .
The label-induced encoder takes A ( g ) t and X t as inputs, and outputs auxiliary embedding U ( g ) t , i.e., U ( g ) t =
G ( A ( g ) t , X t ; δ G ) .
GNN is originally designed for attributed graphs, where each node v ti must have a feature vector described by the i -th row of a feature matrix X t .
In this study, we consider GP without node attributes.
Instead of using conventional settings of GNN for graphs without attributes (e.g., use a constant matrix for the input features), we extract additional structural feature X t based on the neighbor similarity encoded in A t .
IGP captures the permutation invariant label information of historical graphs by combining the dual GNN structure with classic GP objectives.
We adopt the objectives of (i) modularity maximization and (ii) NCut minimization as two examples, corresponding to two settings of X t .
For modularity maximization in (4), the modularity matrix Q t encodes the reweighted neighbor similarity based on A t , so we let X t =
Q t .
For NCut minimization in (2), the Laplacian matrix L t gives the primary characteristics of graph structure, where M t = D − 0 .
5 t
A t D − 0 .
5 t is the key component regarding neighbor similarity.
Hence, we let X t =
M t .
Both Q t and M t are the reweighting of A t , where the pair of nodes ( v ti , v tj ) with similar neighbor-induced features (( X t )
i, : , ( X t ) j, : ) are more likely to be partitioned into the same cluster.
Our experiments further demonstrate that the extraction of X t is essential to ensuring the high GP quality of IGP.
However, the feature dimensionality of X t is N t (i.e., the number of nodes) which may not be ﬁxed over t .
We introduce an efﬁcient feature extraction module via heavy-edge matching (HEM) graph coarsening (Hendrickson & Leland, 1995) to map X t ∈ (cid:60) N t × N t to a matrix Z t ∈ (cid:60) N t × L with ﬁxed dimensionality L .
For N t > L , we extract reweighted edges E ( w ) t [Equation]
) = ( X t ) ij , ( A t ) ij = 1 } and apply HEM to E ( w ) t .
It merges the original graph G t with N t nodes into a supergraph G ∗ t with L supernodes via a greedy strategy of continuously merging the node pair with largest weight in E ( w ) t into a supernode, e.g., merging a graph with 8 nodes into 2 supernodes in Fig.
HEM outputs a coarsening matrix C t ∈ (cid:60)
N t × L , where ( C t ) ij = | v t ∗ j | − 0 .
5 if node v ti is merged into supernode v t ∗ j and ( C t ) ij = 0 otherwise.
We then let Z t =
X t C t be the reduced features.
Since C t is a sparse matrix, one can obtain Z t by setting its j -th column to ( Z t ) : ,j = (cid:80) v i ∈ v t ∗ j ( C t ) ij ( X t ) : ,i .
For N t <
L , we let Z t =
[ X t , 0 N t × ( L − N t ) ] .
In contrast to existing methods using HEM to reduce topology complexities (Dhillon et al., 2007), we are the ﬁrst to extract feature input Z t for inductive GNNs (with non-ﬁxed N t ) via HEM.
It is more efﬁcient than dimensionality reduction (e.g., PCA of A t ) in SOTA GNNs (Nazi et al., 2019), since HEM is widely used in fast GP methods (Liang et al., 2021).
The derived Z t is also more informative than the input features in some classic settings of inductive GNNs (e.g., let Z t be a constant matrix (Xu et al., 2019)).
Let F ( l − 1) t and F ( l ) t be the input and output of the l -th layer of feature encoder or label-induced encoder in G with F (0) t =
Z t .
The l -th GNN encoder layer is deﬁned as
[Equation]
where ˆA t =
A (cid:48) t
+ I N t
( A (cid:48)
t ∈ { A t , A ( g ) t } ) is the adjacency matrix with self-edges; ˆD t = diag( ˆ d t 1 , · · · , ˆ d tN t ) is the diagonal degree matrix; W ( l − 1) G is the trainable parameter matrix shared by the two encoders;
( F ( l ) t ) i, : is the latent feature vector of node v ti .
In (5), ( F ( l ) t ) i, : is the nonlinear aggregation of the features of { v ti } ∪ n ( v ti ) in the previous layer, where n ( v ti ) is the neighbor set of v ti .
It is also known as the message passing of GNN, where neighbors of v ti propagate their features to v ti for aggregation.
By Fact 1 , A ( g ) t ensures that message passing of the label-induced encoder only occurs in each connected component w.r.t. each cluster in the ‘ground-truth’ (e.g., thecomponents in Fig. 2), while message passing of the feature encoder occurs in the original topology of G t .
The last layers of the feature and label-induced encoders output the graph embedding U t ∈ (cid:60) N t × k and auxiliary label-induced embedding U ( g ) t ∈ (cid:60) N t × k , where U ( g ) t = G ( A ( g ) t , X t ) preserves more informative label-induced properties than U t
= G ( A t , X t ) .
We further use the non-linear inner product of U t to reconstruct the neighbor-induced features X t : [Equation]
The discriminator D is an auxiliary classiﬁer to distinguish U ( g ) t from U t , while the generator G tries to generate plausible embedding U t to fool D .
Such an adversarial process helps G to generate embedding U t close to U ( g ) t .
We denote D as y t =
D ( S t ; δ D ) , where S t ∈ { U t , U ( g ) t } and δ D are the input and set of model parameters; y t ∈ (cid:60) N t is a column vector with ( y t ) i as the probability that ( S t ) i, : =
( U ( g ) t ) i, : rather than ( S t ) i, : =
( U t ) i, : .
Let P ( l − 1) t and P ( l ) t be the input and output of the l -th layer in D with P (0)
t =
S t .
The l -th layer of D is deﬁned as
[Equation]
where { W ( l − 1) D , b ( l − 1) D } are trainable model parameters.
In particular, we use sigmoid as the activation function of the last layer instead of ReLU .
Model Optimization.
As in Fig. 2, the ofﬂine training of IGP is based on three objectives: (i) adversarial learning (AL), (ii) feature reconstruction (FR), and (iii) clustering regularization (CR).
The AL objective helps incorporate the permutation invariant label information of historical graphs to the unsupervised embedding learning via an adversarial process between D and G .
On the one hand, D tries to distinguish U ( g ) t from U t .
The objective of D w.r.t. a graph G t is
[Equation]
On the other hand, G tries to fool D by minimizing the following loss w.r.t. G t :
[Equation]
Such an adversarial process directs G to output the embedding U t that is close to U ( g ) t , thus enabling
G to capture the permutation invariant label information of historical training graphs.
The FR loss forces G to derive the embedding U t that encodes the key properties of the neighborinduced features X t w.r.t. G t by minimizing the reconstruction error between X t and ˜X t :
[Equation]
In addition to the AL loss, IGP can also capture the permutation invariant label information via the regularization of several GP objectives, e.g., modularity maximization (4) and NCut minimization (2).
We introduce the CR objective for a graph G t that minimizes the following loss:
[Equation]
where H t is the membership indicator encoding the ‘ground-truth’ with the same deﬁnitions in (4) and (2).
Since the constraints on H t , e.g., tr( H Tt H t )
=
N t and H Tt H t
= I K t , are always satisﬁed for the given ‘ground-truth’, we do not need to consider the discrete constraints on H t .
To the best of our knowledge, we are the ﬁrst to use the CR objective to incorporate the permutation invariant label information (encoded by H t of historical training data) to the unsupervised embedding learning, whereas H t is the parameter to be optimized or output in existing GP methods (Wilder et al., 2019).
Finally, we derive the objective of G w.r.t. G t by combining (9), (10), and (11):
[Equation]
where { α, β } are hyper-parameters to balance L FR and L CR .
In the joint ofﬂine optimization of D and G , we use the Xavier method to initialize model parameters { δ D , δ G } .
The Adam optimizer is applied to iteratively update δ D and δ G based on the gradients of loss (8) and (12).
In each epoch, we randomly sample a certain number of historical graphs p from the training set to update { δ D , δ G } .
Finally, we save the model parameters { δ ∗ D , δ ∗ G } that result in the best average GP quality on the validation set within a certain number of epochs n .
After ofﬂine training, we generalize IGP to new graphs for fast online GP.
As in Fig. 1, for each new graph G t (cid:48) , we derive its embedding U t (cid:48) by passing { A t (cid:48) , X t (cid:48) } forward the feature encoder of G with its parameters δ G ﬁxed.
As we assume that the number of clusters K t (cid:48) is given, we apply
K Means to U t (cid:48) , which outputs the GP result w.r.t. the given K t (cid:48) .
Therefore, the runtime of online GP on G t (cid:48) includes (i) the feature extraction of Z t , (ii) one feedforward propagation through the feature encoder , and (iii) the downstream clustering.
Due to the space limit, we conclude the ofﬂine training and online generalization procedures of IGP in Appendix B.
E XPERIMENTS
Datasets.
We evaluate IGP on 2 synthetic benchmarks (with 7 settings) and 4 real datasets.
Statistics of the datasets are depicted in Table 1, where T , N , |E| , and K are the number of graphs, nodes, edges, and clusters.
The cover the cases with (i) ﬁxed N and K , (ii) ﬁxed N but non-ﬁxed K , as well as (iii) non-ﬁxed N and K .
GN-Net (Girvan & Newman, 2002) and LFRNet (Lancichinetti et al., 2008) are synthetic benchmarks that can simulate properties of real-world network systems.
In each benchmark, we generated a set of graphs and their GP ground-truth via several distributions.
GN-Net and LFR-Net use p in and µ to control clustering structures of each graph.
We denote the GN-Net with a setting of p in as GN- p in ( p in ∈ { 0 . 5 , 0 . 4 , 0 . 3 } ).
The 3 datasets of GN-Net are with ﬁxed N and K .
We use L ( c, µ ) ( c ∈ { f, n } , µ ∈ { 0 . 3 , 0 . 6 } ) to represent the LFR-Net with a setting of µ , where f and n denote cases with fixed and non-ﬁxed N .
As p in decreases and µ increases, the clustering structures become increasingly difﬁcult to identify.
Taxi (Piorkowski et al., 2009), Reddit (Yanardag & Vishwanathan, 2015), Enron (Rossi & Ahmed, 2015), and AS (Leskovec et al., 2005) are real datasets from a vehicle network, an email network, a social network, and an autonomous system.
As the 4 real datasets do not provide ground-truth regarding K , to ensure the fairness of comparison, where each graph is with a common K for all the methods to be evaluated , we adopted a widely-used strategy that uses the Louvain algorithm (Blondel et al., 2008) to estimate K for each graph.
Details of the datasets are given in Appendix C.2.
Baselines.
We compared IGP over 15 baselines with 4 categories.
(i) SNMF (Wang et al., 2011) and spectral clustering ( SC ) (Von Luxburg, 2007) are classic transductive GP methods.
(ii) VERSE
(Tsitsulin et al., 2018), MILE (Liang et al., 2021), and PhUSION ( PhN ) (Zhu et al., 2021) are graph embedding approaches for high-quality representations, while NPR (Yang et al., 2020), RandNE (Zhang et al., 2018a), and ProNE (Zhang et al., 2019) are efﬁcient embedding methods with fast approximation.
All these embedding methods are transductive .
(iii) Metis (Hendrickson & Leland, 1995), hMetis (Selvakkumaran & Karypis, 2006), and GraClus (Dhillon et al., 2007) are fast transductive GP baselines with heuristic approximation.
(iv) GraSAGE ( GSAGE ) (Hamilton et al., 2017) and GAT
(Veli ˇ ckovi ´ c et al., 2018) are SOTA inductive GNNs, while GAP (Nazi et al., 2019) and ClusNet (Wilder et al., 2019) are E2E methods based on inductive GNNs.
Besides the 15 methods, we also include other strong baselines (e.g., node2vec ) with details given in Appendix C.3.
Both variants of IGP were evaluated.
For simplicity, we use IGP-M and IGP-C to denote the variants adopting the objectives of modularity maximization and NCut minimization, respectively.
Quantitative Evaluation.
In our quantitative evaluation, we consider GP on graphs where K is given.
On all the datasets, we adopted quality metrics of modularity and NCut as in (3) and (1).
On datasets with ground-truth (i.e., GN-Net and LFR-Net ), we also used quality metrics of NMI and accuracy ( AC )
(Cui et al., 2018) to measure the correspondence between GP results and groundtruth.
Usually, larger modularity , NMI and AC , as well as smaller NCut imply better quality.
Moreover, the total runtime of a method to derive its ﬁnal GP result was used as the efﬁciency metric with lower runtime indicating higher efﬁciency.
Most conventional baselines (e.g., SC ) are transductive , which can only be applied to a static graph, but IGP alleviates the NP-hard challenge of GP in an inductive framework across graphs.
To illustrate such superiority of IGP beyond transductive baselines, for each dataset, we used the ﬁrst 80% of the graphs as the training set Γ T with the remaining 10% and 10% deployed as the validation set Γ V and test set Γ (cid:48) .
In this setting, Γ T ∪ Γ V and Γ (cid:48) represent the sets of historical and newly generated graphs, respectively.
We conducted the ofﬂine training of IGP on Γ T and generalized it to Γ (cid:48) for online GP.
In the ofﬂine training, we save the model parameters of IGP with the best average quality on Γ V , where we use NMI and modularity as the validation quality metric for datasets with and without ground-truth, respectively.
Since we focus on the trade-off between quality and efﬁciency on new graphs, the evaluation was conducted on Γ (cid:48) , where we had to apply transductive baselines to each test graph in Γ (cid:48) from scratch due to their transductive nature.
For inductive GNN baselines (i.e., GraSAGE and GAT ), we use the same training and evaluation settings with IGP, i.e., ofﬂine training on Γ T and online generalization to Γ (cid:48) .
Since we consider GP on graphs without attributes, we use a constant matrix 1 N t × L (with ﬁxed feature dimensionality L ), where each entry is set to 1 , as the input features of these GNNs, which is a widely-used setting of inductive GNNs for the case without attributes (Xu et al., 2019).
In the ofﬂine training of inductive
GNN baselines, we adopted the widely-used unsupervised loss of GraSAGE (Hamilton et al., 2017) to train them on Γ T , since most existing inductive GNNs (even with a supervised loss like crossentropy) cannot directly integrate the permutation invariant label information from Γ T .
[Table] [Table]
For the E2E GNN methods (i.e., GAP and ClusNet ), we could only conduct the ofﬂine training and fast online generalization on datasets with ﬁxed K (i.e., on GN-Net ), but we had to train them from scratch on the test set of other datasets with non-ﬁxed K , due to the ﬁxed dimensionality of their E2E output layers.
As recommended in (Nazi et al., 2019) and (Wilder et al., 2019), we used PCA and node2vec to map the adjacency matrices { A t } to a feature space with ﬁxed dimensionality for the input features of GAP and ClusNet .
Moreover, we used their own unsupervised training loss, which approximates the objectives of NCut minimization and modularity maximization, respectively.
For all the embedding-based methods (including the transductive embedding baselines, inductive
GNNs, and IGP), we used K Means as the downstream clustering module.
We ran K Means 10 times to avoid its limitation of getting locally optimal solutions, which is included in the total runtime.
To ensure the fairness of comparison, the embedding dimensionality of all the embedding-based methods was set to be the same on each dataset.
For other E2E baselines (e.g., Metis , GAP , and
ClusNet ), we directly derived the GP results from their outputs.
Since Taxi , Reddit , Enron , and AS do not provide the GP ground-truth (in terms of the membership C t ) on each historical training snapshot of these datasets, we used the GP result of the baseline with best average modularity on Γ V to derive the membership indicator H t for IGP, which is the second source of historical label information deﬁned in Section 2.
Note that we do not need to derive H t in the evaluation phase when generalizing the trained model to test set Γ (cid:48) .
We tuned parameters of all the methods on Γ V of each dataset to report best quality metrics.
Both the mean and standard deviation of all the quality and efﬁciency metrics on Γ (cid:48) of each dataset were recorded.
Evaluation results in terms of average NMI , modularity , and runtime are visualized in Fig.
Due to the space limit, we give (i) the visualization results w.r.t.
AC and NCut , (ii) mean and standard deviation of all the metrics, and (iii) parameter settings in Appendix C.5.
In most cases of Fig. 3, IGP-M (red circle) and IGP-C (blue circle) are in the top group with best quality, even though we did not train them on test set Γ (cid:48) , where other transductive baselines were fully trained.
Surprisingly, IGP even has the best or second-best quality metrics in some cases.
Moreover, the runtime of IGP-M and IGP-C is also competitive to some fast GP baselines (e.g., Metis ).
Although the runtime of inductive GNN baselines (with fast online generalization) is slightly faster than IGP, they still suffer from poor quality, which implies that the feature extraction module and hybrid training loss of IGP are essential to ensure its high quality.
From the view of alleviating the NP-hard challenge of online GP on newly generated graphs, IGP achieves a better trade-off between quality and efﬁciency.
According to the visualization results in Fig. 3, we introduce the trade-off score in Appendix C.5, which can quantitatively evaluate the trade-off between quality and efﬁciency by computing the area covered by an induced rectangle of each data point in the visualized 2D space.
Ablation Study.
For IGP-M and IGP-C, we also validated the effectiveness of (i) AL loss, (ii) FR loss, (iii) CR loss, (iv) feature input Z t , and (v) GNN by respectively excluding the corresponding components from the original model.
In case (iv), we used a constant matrix 1 N t × L with all entries set to 1 to replace Z t while we use an FC network that only takes Z t as input to replace GNNs in G with same layer conﬁgurations.
Example results on L ( n, .
6) in terms of NMI , AC , modularity , and NCut are shown in Table 2.
According to Table 2, the FR loss, Z t , and GNNs are key components to ensure the high quality of IGP, since there is signiﬁcant quality decline for case (ii), (iv), and (v).
The AL and CR losses further enhance the quality by incorporating permutation invariant label information of historical graphs.
Other detailed ablation studies are given in Appendix C.5.
Extended Applications.
Besides the aforementioned experiments, we consider two extended applications.
First, when the number of clusters K of a new graph is not given, IGP also has the potential
[Figure]
in terms of quality metrics of NMI ( ↑ ) and modularity ( ↑ ) ( y -axis), as well as efﬁciency metric of runtime ( ↓ ) ( x -axis).
The number records, evaluation results of AC and NCut , as well as analysis of the trade-off between quality and efﬁciency are given in Appendix C.5.
to estimate K , where we apply an auxiliary model selection method (e.g., XMeans) to the derived embedding U t .
Second, in contrast to the hypothesis that all the given snapshots are from a common system or scenario following several underlying distributions , it is promising that one conducts the ofﬂine training on historical data of one scenario but generalizes the model to other scenarios for fast online GP.
Especially, we consider the application where we train IGP on synthetic graphs but generalize it to real datasets.
The two extended applications are elaborated in Appendix D.
C ONCLUSION
In this paper, we proposed a novel IGP framework to alleviate the NP-hard challenge of GP, obtaining a better trade-off between quality and efﬁciency.
In IGP, we ﬁrst conduct ofﬂine training of a novel dual GNN structure on historical known graphs and generalize the model to newly generated graphs for fast high-quality online GP.
IGP is a generic framework that can incorporate the permutation invariant label information of GP to the ofﬂine training by combining unsupervised GP objectives (e.g., modularity maximization and NCut minimization) with the dual GNN structure.
Since we introduced an inductive graph embedding scheme and an efﬁcient feature extraction module, IGP can also tackle online GP on graphs with non-ﬁxed number of nodes and clusters. In
Appendix E, we give discussions of possible future work, e.g., further reducing the runtime, scaling IGP up to graphs with large number of nodes, and transferring IGP across scenarios.
A P ROOF OF F ACT
By the disjoint constraint of GP, i.e. C tr ∩ C ts = ∅ for ∀ r (cid:54) =
s , each node can only belong to one unique cluster.
For the membership indicator matrix R t , only one entry in each row of R t must be 1 with the rest entries in the same row set to 0 .
For the auxiliary adjacency matrix A ( g ) t = R t R Tt , consider each entry ( A ( g ) t ) ij =
(cid:80)
K t r =
( R t ) ir ( R t ) jr ] with respect to the pair of nodes ( v ti , v tj ) .
If nodes ( v ti , v tj ) are in the same cluster C ts , i.e., v ti , v tj ∈ C ts , we have R is =
R js = 1 .
We can further derive ( A ( g ) t ) ij =
( R t ) is ( R t ) js = 1 .
It indicates that there is an edge between ( v ti , v tj ) with weight 1 , when they are in the same cluster.
If nodes ( v ti , v tj ) are not in the same cluster, i.e., v ti ∈ C tr and v tj ∈ C ts ( r (cid:54)
= s ), we have R ir =and R js = 1 , but ( A ( g ) t ) ij = 0 .
Namely, there is no edge between nodes ( v ti , v tj ) when they are in different clusters.
In summary, there is an edge between ( v ti , v tj ) in the auxiliary label-induced graph G ( g ) t only if they are partitioned into the same cluster according to the given partitioning result C t
= { C t 1 , · · · , C tK t } (used as the ‘ground-truth’).
Especially, each node v t i ∈
C t r (i) must have edges connected to all the other nodes in the same cluster C tr and (ii) do not have edges connected to nodes in different clusters C ts ( r (cid:54)
= s ), thus forming a fully connected component with respect to cluster C tr .
Hence, for a graph G t with K t clusters, its corresponding auxiliary label-induced graph G ( g ) t has K t fully connected components, with each component corresponding to one unique cluster of G t .
B D ETAILED A LGORITHMS OF IGP
In contrast to conventional transductive GP methods applied to a single static graph, IGP is an inductive framework that explores the structure properties across a set of graphs S = {G 1 , · · · , G T } .
In IGP, we ﬁrst train a novel dual GNN structure on historical system snapshots in an ofﬂine way (regardless of the training time cost).
We then generalize the trained model to newly generated snapshots of the same system for fast high-quality online GP without additional optimization, where a better trade-off between quality and efﬁciency can be achieved.
For simplicity, we divide S into a training set Γ T , a validation set Γ V , and a test set Γ (cid:48) , where Γ T ∪ Γ V and Γ (cid:48) represent the set of historical and newly generated system snapshots, respectively.
Namely, snapshots in Γ T ∪ Γ V should occur before those in Γ (cid:48) . ( k +1) t
V ( k +1) t ← V ( k +1) t ∪ V ( k ) t
E ( k +1) t ← E ( k +1) t ∪ E ( k ) t
k ← k + 1 /
Update level index k 24 return based on merging membership of V ( k ) t
B. F EATURE E XTRACTION
For a set of graph snapshots S = {G 1 , . . .
G T } with non-ﬁxed node set {V 1 , . . .
, V T } (i.e., with non-ﬁxed number of nodes N t ), we introduce an efﬁcient feature extraction strategy via HEM graph coarsening, which maps the neighbor-induced feature X t ∈ (cid:60) N t × N t
( X t = Q t or X t = M t ) to a reduced feature matrix Z t ∈ (cid:60) N t × L with ﬁxed dimensionality L .
Algorithm 1 and Algorithmsummarize the procedures of HEM graph coarsening and the feature extraction module.
For each graph G t ∈ S , if L < N t , Algorithm 1 ﬁrst merges G t into a supergraph G ∗ t (with L supernodes)
based on reweighted edges E ( w ) t = { w ( v ti , v tj ) | w ( v ti , v tj ) =
( X t ) ij , ( A t ) ij = 1 } and outputs a coarsening matrix C t ∈ (cid:60) N t × L that describes the merging membership.
Concretely, ( C t ) ij = | v t ∗ j | − 0 .
5 if node v ti is merged into supernode v t ∗ j (i.e., v ti ∈ v t ∗ j ) and ( C t ) ij = 0 , otherwise.
Based on C t given by Algorithm 1, Algorithm 2 further derives a reduced feature matrix Z t via
Z t = X t C t .
By the sparsity of C t , we can set its j -th column to ( Z t ) : ,j = (cid:80) v i ∈ v t ∗ j ( C t ) ij ( X t ) : ,i .
In contrast, when L > N t , Algorithm 2 directly derives Z t =
[ X t , 0 N t × ( L − N t ) ] via a simple padding strategy.
B. O FFLINE T RAINING ON H ISTORICAL S YSTEM S NAPSHOTS
In the ofﬂine training of IGP, we assume that the partitioning result C t = { C t 1 , · · · , C tK t } (i.e., ‘ground-truth’) of each historical graph
G t ∈ Γ T ∪ Γ V is available, which is permutation invariant in terms of cluster labels.
For simplicity, we correlate each historical graph G t ∈ Γ T ∪ Γ V with a 2 -tuple ( A t , C t ) .
The ofﬂine training procedure of IGP is summarized in Algorithm 3, where L is ﬁxed dimensionality for the reduced features Z t ; { α, β } are hyper-parameters of the training loss
Input: input graph G t ; neighbor-induced features X t ; number of nodes N t ; reduced dimensionality L Output: reduced features Z t 1 if L ≥ N t then let Z t =
[ X t , 0 N t × ( L − N t ) ] //Pad
Z t with zeros 3 else if L < N t then derive the coarsening matrix C t ∈ (cid:60) N t × L via Algorithm derive the merging membership V ∗ t based on C t for each supernode v ∗ j ∈ V ∗ t do ( Z t ) : ,j ← 0 N t × 1 //Initialize the j -th column of Z t for each node v i ∈ v ∗
j do ( Z t ) : , j ← ( Z t ) : ,j + ( C t ) ij ( X t ) : ,i Input n Output best model parameters { δ
, δ
D } 1 initialize the variable ¯ q metric 2 initialize model parameters { δ G , δ D } via the Xavier algorithm 3 for epoch from 1 to n do for sample count from 1 to p do randomly sample a graph G t ∈ Γ T get the associated 2-tuple ( A t , C t ) of G t derive the neighbor-induced features X t based on adjacency matrix
A t derive the reduced features Z t from { A t , X t } via Algorithm derive the membership indicator H t based on the ‘ground-truth’ C t for update count from 1 to m do derive the loss function L D via equation (8) ﬁxed
L G ; p is the number of historical snapshots sampled in each epoch; { η D , η G } are learning rates of the optimizers updating { δ D , δ G } ; n is the maximum number of epochs.
For a selected training sample in each epoch, repeatedly updating model parameters m ( m ≥ 1 ) times can sometimes lead to better partitioning quality (on the validation and test set).
Algorithm 3 ﬁnally saves the model parameters { δ ∗ D , δ ∗ G } with the best quality metric on the validation set Γ V . [Table]
B. O NLINE G ENERALIZATION TO N EWLY G ENERATED S YSTEM S NAPSHOTS
In the online generalization of IGP, we ﬁx the learned model parameters { δ ∗ D , δ ∗ G } and directly derive graph embedding U t (cid:48) with respect to each newly generated graph G t (cid:48) ∈ Γ (cid:48) from feature encoder of G .
Since we assume that the number of clusters K t (cid:48) of each graph G t (cid:48) ∈ Γ (cid:48) is given in the online GP, we apply K Means to U t (cid:48) to derive the partitioning result C (cid:48) t (cid:48) with K t (cid:48) clusters.
In this case, the total runtime of online GP on each newly generated graph G t (cid:48)
∈ Γ (cid:48) includes (i) the efﬁcient feature extraction of Z t (cid:48) , (ii) one feedforward propagation of { A t (cid:48) , Z t (cid:48) } through feature encoder , and (iii) the downstream ( K Means) clustering.
In conclusion, we summarize the online generalization procedure of IGP in Algorithm 4.
B. C OMPLEXITY A NALYSIS
For a given graph G t , we use the efﬁcient matrix multiplication to derive the neighbor-induced features Q t and M t .
Concretely, we set Q t =
A t − d Tt d t (cid:14) (2 w ) and M t = D − 1/
A t D − 1/2 , where d t =
[ d t 1 , · · · , d tN t ] and D t = diag( d t 1 , · · · , d tN t ) are the degree vector and degree diagonal matrix of G t .
Although the complexities of computing Q t and M t are both at most O ( N 2 t ) , they can be easily paralleled via GPUs, which signiﬁcantly reduces the runtime.
Moreover, the complexity of the HEM graph coarsening (i.e., Algorithm 1) is no more than O ( N t |E t | log |E t | ) , with |E t | denoting the number of edges in G t and |E t | (cid:28) N 2 t .
The complexity to derive Z t via X t and C t (i.e.,
Algorithm 2) is no more than O ( N t Ln ∗ ) , where n ∗ is the maximum number of nodes merged into a supernode; L is the number of supernodes (which is also the reduced feature dimensionality of
Z t ).
Hence, the complexity to derive the reduced feature matrix Z t based on X t is no more than
O ( N t |E t | log |E t | + N t Ln ∗ ) .
Let L ( l − 1) be the feature dimensionality of the l -th GNN layer of the feature encoder or auxiliary label-induced encoder in G , where we have L (0) =
L .
We used the efﬁcient sparse-dense matrix multiplication to implement the graph convolutional operation described in equation (5), where the sparsity of A t is considered.
The complexity of one feedforward propagation through the feature encoder is no more than O ( |E t | L (0) L (1) + |E t | L (1) L (2) + · · · ) =
O ( |E t | LL (cid:48) ) , where L (cid:48)
=
L (1) and L ( l ) <
L ( l − 1) .
Hence, the overall time complexity of one feedforward propagation through the feature encoder is linear in the number of edges of the input graph G t .
Let k be the dimensionality of graph embedding.
Usually, we have k (cid:28) N t .
For a graph with K t clusters, the complexity of K Means is no more than O ( N t K t kI ) for I iterations.
In particular, the runtime of the downstream clustering (i.e., K Means algorithm) can also be signiﬁcantly reduced via parallel implementations.
C E XPERIMENT D ETAILS C. E VALUATION M ETRICS
In our experiments, we used the (unsupervised) metrics of modularity and NCut to evaluate the partitioning quality of each method on all the datasets.
The formal deﬁnitions of modularity and
NCut are given in equation (3) and equation (1) (see Section 2).
For datasets with partitioning ground-truth (i.e., GN-Net and LFR-Net ), we also used the metrics of normalized mutual information (NMI) and accuracy (AC)
(Cui et al., 2018), which can measure the [Table] correspondence between the partitioning result (given by the method to be evaluated) and groundtruth (given by the dataset).
In general, better correspondence indicates better partitioning quality
[Table] For a graph G t with K t clusters, we use H t
= { H t 1 , · · · , H tK t } and C t = { C t 1 , · · · , C tK t } to denote the ground-truth and partitioning result, where H tr ⊆ V t and C tr ⊆ V t represent node member sets of the r -th cluster.
Given H t and C t , the deﬁnition of NMI is as follow:
[Equation]
where n r = | H tr | is the number of nodes in the r -th cluster of ground-truth; n s = | C ts | is the number of nodes in the s -th cluster of partitioning result; n r,s = | H tr ∩ C ts | is the number of nodes that are simultaneously partitioned into the r -th cluster of ground-truth and the s -th cluster of partitioning result.
Let L t =
( l t 1 , · · · , l tN t ) and R t =
( r t 1 , · · · , r tN t ) be label sequences of the partitioning result and ground-truth with respect to node sequence ( v t 1 , · · · , v tN t ) , where l ti and r ti are cluster labels of node v ti .
The deﬁnition of AC is as follow:
[Equation]
where map( · ) is the Kuhn-Munkres mapping function that gives the best membership map from L t to R t ; δ ( a, b ) is the Kronecker delta function with δ ( a, b ) =
when a = b and δ ( a, b ) = 0 otherwise.
Usually, larger modularity, NMI, and AC, as well as smaller NCut indicate better partitioning quality of a given GP method.
C. D ATASET D ETAILS
The detailed statistics of the 11 datasets (with 7 synthetic benchmarks and 4 real datasets) used in our experiments are depicted in Table 3, where T , N , |E| , and K are the number of snapshots, nodes, edges, and clusters, respectively.
GN-Net (Girvan & Newman, 2002) is a widely-used synthetic benchmark for GP, in which the graph topology and GP ground-truth of each graph can be generated via the stochastic block model (Abbe, 2017).
We ﬁxed N = 5 , 000 and evenly partitioned the nodes into K = 250 clusters.
For two uniformly selected nodes v i and v j , if they’re assigned in the same cluster (i.e., v i , v j ∈ C r ), the probability to generate an edge ( v i , v j ) is p in .
Otherwise (i.e., v i ∈ C r and v j ∈ C s ( r (cid:54)
= s )), the probability to generate an edge ( v i , v j ) is (1 − p in )/
( K − 1) .
In particular, we set p in ∈ { 0 . 5 , 0 . 4 , 0 . 3 } to generate 3 datasets.
For each dataset, we independently generated 2 ,graphs.
In each graph, we also randomly shufﬂed the node indices to ensure that there is no index correspondence between any two graphs in the dataset (as we deﬁned in Section 2).
Especially, with the decrease of p in , the clustering structures become increasingly difﬁcult to identify .
For simplicity, we denote the GN-Net with a speciﬁc setting of p in as GN- p in (e.g., GN- 0.5, GN- 0.4, and GN- 0.3).
Note that all the datasets of GN-Net are with ﬁxed N and K .
LFR-Net (Lancichinetti et al., 2008) is a more challenging synthetic benchmark that can simulate power-law properties of node degree and cluster size of real-world network systems.
It uses ( k, k max , c min , c max , µ ) to independently generate a single graph, where k and k max are the average and maximum node degree; c min and c max are the minimum and maximum cluster size; µ is the mixing ratio between the external degree and total degree of each node v i with respect to the cluster v i belongs to.
In particular, we ﬁxed ( k, k max , c min , c max ) =
and set µ ∈ { 0 . 3 , 0 . 6 } .
With the increase of µ , the clustering structure is increasingly difﬁcult to identify .
We consider both cases with ﬁxed and non-ﬁxed N , where we respectively ﬁxed N = 5 , 000 and set N to a random integer within [5000 , 6000] to generate 2 , 000 graphs for each case.
Similar to GN-Net , we also independently generated each graph in a dataset and ensure that there is no index correspondence between any two graphs via the random shufﬂe of node indices.
For simplicity, we use L ( c, µ ) ( c ∈ { f, n } ) to represent the dataset with a speciﬁc setting of µ , where f and n denote the case with fixed and non-ﬁxed N (e.g., L ( f , 0.3), L ( f , 0.6), L ( n , 0.3), and L ( n , 0.6)).
Moreover, K is non-ﬁxed in LFR-Net .
Taxi 1 (Piorkowski et al., 2009) is a real GPS dataset including trajectories of 1 , 279 taxis in Beijing, which forms a mobile ad hoc network system.
We sampled 3 , 000 graph snapshots from the dataset in chronological order.
In each graph, we treated each taxi as a node and constructed the topology based on the top- 50 neighbors with the closest distance for each node.
Since Taxi does not provide ground-truth regarding K for GP, we used Louvain algorithm (Blondel et al., 2008) to estimate the K value for each graph in this dataset.
Hence, it is with ﬁxed N and non-ﬁxed K .
Reddit 2 (Yanardag & Vishwanathan, 2015) is a social network dataset extracted from Reddit.
Each graph snapshot in the dataset corresponds to an online discussion thread.
We extracted 1 , 870 graphs with N ≥ 500 from the dataset.
Different graphs are with different number of users (i.e., nodes), varying from 501 to 3 , 648 .
The comment interactions between users in each discussion thread are described as topology of the corresponding graph.
Since Reddit does not provide ground-truth regarding K , we also used Louvain to estimate K for each graph.
Hence, Reddit is with non-ﬁxed N and K .
Note that different snapshots have different node sets (i.e., with non-ﬁxed N ) and node indices in each snapshot start from 0.
It indicates that there is no correspondence between node indices of any two snapshots in the dataset.
Enron 3 (Rossi & Ahmed, 2015) is a real interaction network extracted from the email system of
Enron company, which contains email records from 1980-01-01 to 2004-02-04.
We extractedsnapshots with N ≥ 500 from the records in chronological order, in which we treated each user as a unique node and constructed the topology based on email interactions.
Note that different graphs have different sets of users that have interaction behaviors.
Moreover, we also used Louvain algorithm to estimate K for each graph, since Enron does not provide ground-truth regarding K .
Therefore, it is with non-ﬁxed N and K .
We also ensure that there is no node index correspondence between any two graphs by permuting node indices of each graph from 0.
AS 4 (Leskovec et al., 2005) is a real dataset extracted from the border gateway protocol (BGP) logs of a communication network, which describe the topology among a set of autonomous systems (AS).
It contains 733 daily instances spanning from 1997-08-11 to 2000-01-02.
Each BGP router is abstracted as a unique node with communication between routers extracted as the graph topology.
As the dataset allows addition and deletion of routers, different graphs are with different set of nodes.
Since AS does not provide the ground-truth K , we also applied Louvain to determine K for each snapshot.
Therefore, it is with non-ﬁxed N and K , where there is no index correspondence between any two graphs.
In summary, the 3 datasets of GN-Net are with ﬁxed N and K .
The 4 datasets of LFR-Net cover both the cases of (i) ﬁxed N but non-ﬁxed K as well as (ii) non-ﬁxed N and K .
Taxi is a real dataset with ﬁxed N but non-ﬁxed K , while Reddit , Enron , and AS are with non-ﬁxed N and K .
Note that both GN-Net and LFR-Net are synthetic benchmarks that can simultaneously generate graph topology and GP ground-truth of each snapshot.
Taxi , Reddit , Enron , and AS are real datasets without ground-truth (regarding K ), so we applied Louvain algorithm to these datasets to determine the K value of each single snapshot, which ensures the fairness of comparison, i.e., each snapshot shoule be assigned with a common K for all the methods to be evaluated.
C. B ASELINE M ETHODS
In addition to the 15 baseline methods introduced in Section 4, we also evaluated partitioning quality and runtime of other 5 strong baselines, which include degree-corrected stochastic block model ( DCSBM ), IncSNA (Su et al., 2020), (Karrer & Newman, 2011), DeepWalk (Perozzi et al., 2014), node2vec (Grover & Leskovec, 2016), LINE (Tang et al., 2015), TIMERS (Zhang et al., 2018b) and GIN (Xu et al., 2019).
In summary, we have in total evaluated the quality and efﬁciency ofbaselines with 4 categories.
• First, SNMF , DCSBM , and SC are classic transductive GP methods based on several GP objectives (e.g., SC for NCut minimization and DCSBM for modularity maximization), which aim to achieve high-quality partitioning results (but may have high complexity).
Moreover, IncSNA is an incrmental GP baselines that can utilize the node correspondence between graphs to incremental derive the GP results for multiple associated graphs.
• Second, DeepWalk , node2vec , LINE , VERSE , MILE , PhUSION , NPR , RandNE , and ProNE are unsupervised transductive graph embedding baselines.
In particular, LINE , NPR , RandNE , and ProNE are efﬁcient embedding methods with several fast approximation strategies for the embedding learning (but may have low partitioning quality).
In contrast, DeepWalk , node2vec , VERSE , MILE , and PhUSION are baselines that aim to derive high-quality representations (but may have high complexity)
Besides, TIMERS is a typical incremental graph embedding method that can incremental update the learned embedding for graphs with node correspondence.
• Third, Metis , hMetis , and GraClus are classic fast transductive GP baselines that aim to achieve low runtime with the heuristic approximation of classic GP objectives (which may suffer from quality degradation due to the information loss of heuristic approximation).
Fourth, GraSAGE , GAT , and GIN , GAP , and ClusNet are inductive GNN baselines.
In particular, GraSAGE , GAT , and GIN are SOTA inductive GNN following the graph embedding scheme, i.e., learning node representations to support downstream tasks including GP.
Note that we do not consider supervised training loss of these inductive GNNs (e.g., using the cross-entropy loss for a softmax output layer combined with the GNN), because GP is a typical unsupervised task and the cluster labels are permutation invariant.
In contrast, GAP and ClusNet are E2E GP baselines based on inductive nature of GNN, where an FC output layer is integrated into the GNN to directly output the partitioning results.
Note that the ﬁrst three types of methods are inherently transductive , which only focus on GP on a single static graph but cannot be generalized to other new graphs.
Hence, we had to apply these transductive baselines to each newly generated system snapshot (i.e., each snapshot in test set Γ (cid:48) ) from scratch.
For the fourth type of baselines (i.e., inductive GNN approaches), we used the same settings with IGP (i.e., ofﬂine training on training set Γ T and online generalization to test set Γ (cid:48) ).
In particular, we can only use the inductive nature of E2E GNN methods (i.e., GAP and ClusNet ) on datasets with ﬁxed number of clusters K (i.e., on GN-Net ).
However, we have to use the same settings with transductive baselines (i.e., training GAP and ClusNet from scratch for each new test graph) on rest datasets with non-ﬁxed K , due to the ﬁxed dimensionality of their output layers.
Note that not all the datasets have node correspondence between graphs but the incremental GP and graph embedding baselines (i.e., IncNSA and TIMERS ) can only be applied to datasets with node correspondence.
To include IncNSA and TIMERS in our evaluation, we applied them to the original version of Taxi and AS that have node correspondence.
Some other experiment settings regarding the baseline methods have been already introduced in Section 4.
C. I MPLEMENTATION D ETAILS
We used PyTorch to implement the dual GNN structure of IGP.
The feature extraction module (i.e.,
Algorithm 1 and Algorithm 2) was implemented via NumPy and PyTorch, while we used the implementation of K Means of scikit-learn as the downstream clustering module.
All the experiments were conducted on a server with Intel Xeon CPU (E5-2650v4@2.20GHz, 48 cores), 1 Tesla V
GPU, 61GB memory, as well as the Ubuntu Linux operating system.
In this setting, modules implemented by PyTorch were speeded up via GPU.
[Table] For all the embedding based methods, including transductive embedding baselines (i.e., second type of the baseline), inductive GNNs (i.e., GraSAGE , GAT , and GIN ), and IGP, we used K Means as the downstream clustering module.
In contrast, we can directly obtain the partitioning results from outputs of the rest baselines.
Note that K Means for graph embedding based methods and optimization algorithms of several classic GP baselines (i.e., SNMF , DCSBM , Metis , hMetis , and
GraClus ) have the limitation of obtaining locally optimal solutions.
We adopted the strategy to run
K Means of embedding based methods and optimization algorithms of classic baselines 10 times, which was included in the total runtime of corresponding methods.
The partitioning membership with minimum loss function value (e.g., the sum of squared distance between data and cluster center for K Means) was reported as the ﬁnal partitioning result for evaluation.
When testing the runtime of each method, we ensured that only the method to be evaluated was running on the experiment server, i.e., there were no multiple methods simultaneously running on the test server.
On the validation set Γ V of each dataset, we tuned parameters and determine proper layer conﬁgurations for the two variants of IGP.
Concretely, we adjusted α ∈ { 1 , 10 } and m ∈ { 1 , 2 , · · · 5 } for both
IGP-M and IGP-C
, while we tuned β ∈ { 0 . 1 , 1 , 2 , · · · 5 } and β ∈ { 0 . 1 , 1 , 100 , 1000 } for IGP-M and IGP-C, respectively.
The recommended parameter settings and layer conﬁgurations of IGP-M and IGP-C on each dataset are depicted in Table 4, where dimensionality of the ﬁrst and last layer of G are set to be L (i.e., dimensionality of reduced feature) and k (i.e., dimensionality of graph embedding), respectively.
Note that we use the same layer conﬁgurations for IGP-M and IGP-C on each dataset.
Moreover, we set the maximum number of epochs n = 100 for all the datasets.
C. D ETAILED E VALUATION R ESULTS AND D ISCUSSIONS C.5. Q UANTITATIVE E VALUATION
In addition to the visualization results of NMI , Modularity , and runtime in Fig. 3, we also visualized the evaluation results in terms of AC , NCut , and runtime on the 11 datasets in Fig. 4.
In Fig.
4, one can have the observation consistent with Fig.
In most cases, both IGP-M and IGP-C are in the top groups with best quality metric.
The total runtime of IGP-M and IGP-C is also competitive to the fast transductive GP baselines.
In the experiments, we recorded the mean value µ and standard deviation σ of all the evaluation metrics and runtime for all the methods (on test set Γ (cid:48) of each dataset).
In particular, we use format ‘ µ ( σ ) ’ to represent each evaluation result.
The quantitative evaluation results on GN-Net and LFRNet in terms of NMI and AC are depicted in Table 5 and Table 6.
The evaluation results on all the datasets in terms of modularity and NCut are illustrated in Table 7, Table 8, Table 9, and Table 10.
Moreover, Table 11 and Table 12 give the runtime of all the methods.
For both IGP-M and IGP-C, we also recorded the time of (i) feature extraction (i.e., Algorithm 2), (ii) one feedforward propagation (through the feature encoder of G ), and (iii) downstream clustering (i.e., 10 independent runs of
K Means), which together contribute to the total runtime of IGP.
Runtime of the three modules is denoted as ’Feat’, ’Prop’, and ’Clus’, respectively.
According to evaluation results in Table 5, Table 6, Table 7, Table 8, Table 9, Table 10 Table 11, and Table 12, which are consistent with the visualization results in shown Fig.
3 (see Section 4), we have the following major observations. [Figure]
in terms of quality metrics of AC ( ↑ ) and NCut ( ↓ ) ( y -axis), as well as efﬁciency metric of runtime ( ↓ ) ( x -axis).
The number records are given in Table 7, 8, 9, 10, 11, and 12.
• For all the datasets, IGP-M and IGP-C are in the top group with best quality metrics.
Moreover, their runtime is also competitive to fast GP baselines (e g., Metis , hMetis , GraClus , RandNE , etc.) and other inductive GNNs with fast online generalization (e g., GraSAGE , GAT , and GIN ).
From the perspective of alleviating the NP-hard challenge of GP, IGP can achieve a signiﬁcant trade-off between quality and efﬁciency of online GP on newly generated system snapshots.
We further quantitatively evaluate the signiﬁcant trade-off obtained by IGP via a novel trade-off score in Appendix C.5.2.
• Surprisingly, in some cases, IGP even has the best or second-best quality metrics.
Note that IGP is a novel inductive framework, in which we do not conduct additional optimization on the test set of each dataset, but other conventional transductive baselines (e.g., SP , DeepWalk , etc.) can fully explore the structural properties of each test graph.
It indicates the potential of IGP to capture additional characteristics shared by multiple historical snapshots beyond conventional transductive approaches applied to one single new graph.
• On GN-Net and LFR-Net , partitioning quality of all the methods declines with the decrease of p in and increase of µ , where both IGP-M and IGP-C are always in the top group with best quality metrics.
It demonstrates the strong robustness of IGP, especially when the intrinsic clustering structure of the given graph is not distinct.
[Table]
[Table]
• Similar to IGP, the E2E GNN baselines (i
[Table]
e., GAP and ClusNet ) can also apply fast online generalization to datasets with ﬁxed K (i.e., on GN-Net ).
Namely, we also trained GAP and ClusNet on training set of GN-Net and directly generalized them to corresponding test sets for online GP, in which we directly derived the partitioning results from their output layers.
However, the online GP of GAP and ClusNet is with poor quality, which indicates that the online generalization of the graph embedding scheme in IGP is more robust than that of the E2E approaches, especially when clustering structure of a dataset is not distinct.
Furthermore, we still need to train GAP and ClusNet from scratch for each new test graph on datasets with non-ﬁxed K due to the ﬁxed dimensionality of their output layers, which is time-consuming.
In contrast, the graph embedding scheme of IGP can still tackle fast online GP with non-ﬁxed K .
In contrast to the reduced structural feature Z t in IGP, we used a constant matrix 1 N t × L as feature inputs of inductive GNN baselines (i.e., GraSAGE , GAT , adn GIN ), which is a standard setting of most inductive GNNs for graphs without available node attributes (Xu et al., 2019).
The widely-used unsupervised training loss of GraSAGE (Hamilton et al., 2017) was also adopted in the ofﬂine training of these inductive GNNs.
Although the inductive GNN baselines have slightly faster runtime over IGP, they still suffer from poor partitioning quality on all the datasets.
The results further validate that the extracted
[Table]
[Table]
feature Z t as well as hybrid training loss of IGP (which can incorporate the permutation invariant label information of historical training snapshots) are essential to ensure its high partitioning quality.
• For the total runtime of IGP on all the datasets, the downstream clustering (e.g., K Means with 10 independent runs) is a major bottleneck.
However, the downstream clustering module is also the key component that enables IGP to deal with online GP with non-ﬁxed K .
In our future work, we intend to further reduce the runtime of IGP by replacing the downstream clustering module with a generic E2E module that can directly derive the partitioning result with respect to a speciﬁc (non-ﬁxed) K via an output layer.
• In most cases, IGP-M and IGP-C have similar quality metrics.
On datasets in which the clustering structures are more difﬁcult to identify (e.g., L ( f , 0.6) and L ( n , 0.6)), IGP-M has better partitioning quality than that of IGP-C. It implies that the modularity matrix Q t is more informative than the normalized adjacency matrix M t to be the neighbor similarity feature X t in IGP.
Moreover, the neighbor-induced feature X t is also the key component to ensure the strong robustness of IGP, especially when clustering structures of a given graph are indistinct. [Table] [Table]
C.5. T RADE -O FF A NALYSIS
In this paper [Table] we focus on the trade-off between quality and efﬁciency of online GP on newly generated graphs from a system.
To the best of our knowledge, to evaluate the trade-off between two aspects that conﬂict with each other using a comprehensive metric remains an open issue.
We propose a trade-off score (TOS) to quantitatively evaluate the trade-off between quality and efﬁciency of a given GP method.
Concretely, the calculation of TOS is based on the normalized area covered by an induced rectangle of a data point in the quality-efﬁciency visualization result (like Fig. 3).
Fig.
5 gives two running examples regarding how to calculate TOS for a given GP method.
Our goal is to let larger TOS indicate a better trade-off between quality and efﬁciency.
In our evaluation, we use NMI, AC, modularity, and NCut as the quality metrics, while we evaluate the efﬁciency of a speciﬁc method via its total runtime.
Let Q and E be the value of a quality metric and the runtime with respect to a GP method to be evaluated.
Given the tuple ( E, Q ) , each method used in our experiments can be visualized as a data point in a 2-D space as illustrated in Fig.
When we use NMI or AC as the quality metric Q , we further derive the normalized metric via
[Equation]
[Equation] [Table] [Table] since the value range of NMI or AC is Q ∈ [0 , 1] and larger NMI or AC indicate better partitioning quality.
Furthermore, when we use modularity as the quality metric, the value range is Q ∈ [ − 1 , 1] , in which larger modularity indicates better quality.
Hence, we derive the normalized metric as
[Equation]
In contrast, smaller NCut metric implies better partitioning quality with value range Q ∈ [0 , + ∞ ) .
Therefore, when using NCut as the quality metric Q , we normalize it via
[Equation]
[Figure] [Table] with Q m as the maximum NCut value through all the methods to be evaluated.
For efﬁciency metric
E (i.e., runtime), smaller value of E indicates higher efﬁciency with value range E ∈ [0 , + ∞ ) , so we further normalize E via
[Equation]
where E m denotes the maximum runtime through all the methods to be evaluated.
Finally, we deﬁne
TOS with respect to the given quality and efﬁciency metrics ( Q, E ) as the (normalized) area covered by the induced rectangle illustrated in Fig.
Namely, we have
[Equation]
in which larger TOS implies a better trade-off between quality and efﬁciency.
The TOS values with respect to the quality metrics of NMI, AC, modularity, and NCut are depicted in Table 13, Table 14, Table 15, and Table 16, respectively.
In our experiments, we saved model parameters of IGP with best metrics in terms of NMI and modularity on validation sets of datasets with and without ground-truth, respectively .
Namely, we used NMI and modularity as validation metrics for datasets with and without ground-truth .
For TOS metrics with respect to NMI and modularity (see Table 13 and Table 15), both IGP-M and IGP-C have the best and second-best
TOS values in most cases, indicating that IGP can achieve the best trade-off when using NMI and modularity as the (validation) quality metrics.
Furthermore, IGP-M and IGP-C can also achieve the top- 5 TOS metrics with respect to AC and NCut on all the datasets.
In summary, IGP can achieve a better trade-off between quality and efﬁciency of the online GP over various baseline methods. [Table] [Table] Taxi
Reddit
AS Enron SNMF     0. 0.7970 DCSBM 0. 0.4398 SC 0.
0.8454 IncSNA - - - - - - - 0.
- DeepWalk 0.
0.6271 node2vec 0.
0.6972 LINE 0.
0.7965 VERSE 0.
0 MILE 0.
0.7653 PhUSION 0.
0.7338 NRP 0.
0.4963 RandNE 0.
0.5998 ProNE 0.
0.7971 TIMERS - - - - - - - - 0.
- Metis 0.
0.8213 hMetis 0.
0.8245 GraClus 0.
0.8044 GraSAGE 0.
0.5423 GAT 0.
[Table]
C.5. R OBUSTNESS A NALYSIS
We further test the robustness of IGP across multiple graphs using a more challenging setting of the LFR-Net .
Different from the quantitative evaluation in Appendix C.5, where we used ﬁxed settings of parameters ( k, k max , , c min , c max ) and µ , we randomly set µ ∈ [0 . 3 , 0 . 6] , k ∈ [10 , 20] , k max ∈ [100 , 200] , c min ∈ [10 , 20] , c max ∈ [100 , 200] , and N ∈ [500 , 10000] to independently generate 1 , 000 graphs with GP ground-truth.
In this case, the difference between synthetic graphs (in terms of their underlying distributions) is much larger than that in Appendix C.5.
We adopted the same experiment settings and evaluation protocols of Appendix C.5 to compare the quality, efﬁciency, and trade-off between the two aspects of all the methods on this new dataset.
The robustness analysis results in terms of NMI , AC , modularity , NCut , runtime , and TOS values between each quality metric (from the four metrics) and one efﬁciency metric are depicted in
Table 17.
In most cases, IGP-M can achieve the best TOS values.
The TOS values of IGP-C are also competitive to those of IGP-M.
In summary, the robustness analysis results demonstrate that our
[Table]
MILE
[Table]
PhUSION
[Table]
[Table]
[Table]
[Table]
IGP method is robust to the large distribution difference of graphs in the same system or scenario, while ensuring its ability to achieve signiﬁcant trade-off between quality and efﬁciency.
C.5. C ONVERGENCE A NALYSIS
We also analyzed the convergence of the ofﬂine training procedure of IGP (i [Table] e., Algorithm 3).
In each epoch of the ofﬂine training on a dataset, we recorded the average partitioning quality (in terms of NMI, AC, modularity [Table] and NCut) with respect to the learned embedding { U t } (i) on training set Γ T and validation set Γ V (i.e., historical system snapshots).
Moreover, we also evaluated the average partitioning quality of the generalized embedding { U t } on test set Γ (cid:48)
(i.e., new snapshots) in each epoch.
To further validate the effectiveness of auxiliary label-induced embedding { U ( g ) t } , [Figure] [Figure] the average partitioning quality of { U ( g ) t } on training set was also recorded in each epoch during the ofﬂine training.
Note that we do not need to derive { U ( g ) t } for Γ V and Γ (cid:48) .
We use evaluation records on the challenging L ( n , 0.6) dataset, whose clustering structures are difﬁcult to identify, as an example.
The convergence curves (within the ﬁrst 50 epochs) of the average
NMI, AC, modularity, and NCut with respect to (i) { U ( g ) t } on training set Γ T , (ii) { U t } on training set Γ T , (iii) { U t } on validation set Γ V , and (iv) { U t } on test set Γ (cid:48) for IGP-M and IGP-C are illustrated in Fig.
6 and Fig. 7, respectively.
[Table]
[Table]
One can have similar observations in Fig.
6 and Fig.
Concretely, the partitioning quality (in terms of NMI, AC, modularity, and NCut) of { U t } continuously improves in the ﬁrst several epochs on the training, validation, and test set.
More importantly, the average NMI and AC of { U ( g ) t } on training set Γ T are always 1 in all the epochs for both IGP-M and IGP-C.
It indicates that the partitioning results derived from the auxiliary label-induced embedding { U ( g ) t } have perfect mapping to the ground-truth.
For both IGP-M and IGP-C, the average modularity and NCut of { U ( g ) t } also keep at values that are much better than metrics of { U t } on the training, validation, and test set.
The convergence curves with respect to { U ( g ) t } further verify that the introduced label-induced embedding { U ( g ) t } can capture much more information regarding ‘ground-truth’ of the training set (i.e., historical system snapshots) than the learned embedding { U t } , which enables IGP to capture the permutation invariant label information during the ofﬂine training.
It also ensures the strong robustness of IGP beyond existing baselines.
In summary, the convergence curves in Fig.
6 and Fig.
7 validate the effectiveness of the ofﬂine training procedure and the dual GNN structure of IGP.
C.5. A BLATION S TUDY
In addition to example results shown in Table 2, we conducted additional ablation studies on GN -0.3,
L ( f ,0.6), and L ( n ,0.6), in which the clustering structures are more difﬁcult to identify compared with other datasets.
In each case of the ablation study (see Section 4), the mean value µ and standard deviation σ of all the quality metrics (on test set Γ (cid:48) ) were recorded.
The detailed ablation study results on L ( f ,0.6), L ( n ,0.6), and GN -0.3 (in the format µ ( σ ) ) are illustrated in Table 18, Table 19, and Table 20, respectively.
Note that Table 20, Table 18, and Table 19 have consistent results.
The FR loss, feature input Z t , and GNN are key components to ensure the high quality of IGP, because there are signiﬁcant quality declines in cases without the three components.
Moreover, the AL and CR losses are the components to further enhance the partitioning quality, as they can incorporate the permutation invariant label information of historical snapshots to the ofﬂine training of IGP.
D E XTENDED A PPLICATIONS D. M ODEL S ELECTION
As deﬁned in Section 2, we assume that the number of clusters K is given in the downstream GP.
For online GP in which K is not given, IGP also has the potential to estimate the K value for a given graph.
As a demonstration, we applied XMeans, a model selection method, to the derived embedding { U t } of both IGP-M and IGP-C to estimate K for each snapshot in the test sets of
L ( f, . 3) , L ( f, . 6) , L ( n, . 3) and L ( n, . 6) .
In the 4 datasets, ground-truth regarding the K value is [Table] [Figure] 11e3(
[Figure] [Figure] [Figure] [Figure]
Test Set    # C l u s t e r s
L ( f ,.3):Ground-Truth
L ( f ,.3):IGP-M
L ( f ,.3):IGP-C
L ( f ,.6):Ground-Truth
L ( f ,.6):IGP-M
L ( f ,.6):IGP-C given by the dataset and K is non-ﬁxed for multiple test graphs.
With the increase of µ , the clustering structures of L ( f, µ ) and L ( n, µ ) are increasingly difﬁcult to identify.
In particular, we use the same parameter settings and layer conﬁgurations with that in Table 4.
The range of K of XMeans for each dataset was set according to the maximum and minimum number of clusters as illustrated in Table 3.
The model selection results and ground-truth with respect to the test sets of L ( f, µ ) and L ( n, µ ) are depicted in Fig.
8 and Fig 9, respectively.
The model selection results in Fig.
8 and Fig 9 are similar.
When µ = 0 .
3 (with distinct clustering structures ), the model selection results of both IGP-M and IGP-C
ﬁt well to ground-truth on most test snapshots of L ( f, 0 . 3) and L ( n, 0 . 3) .
Note that the potential of model selection of IGP is still largely inﬂuenced by the intrinsic clustering structures of a dataset.
The estimated K values are still far from ground-truth when µ = 0 .
6 (with indistinct clustering structures ) on L ( f, 0 . 6) and L ( n, 0 . 6) .
In summary, IGP has the potential to estimate the number of clusters K of newly generated graph snapshots with distinct clustering structures.
D. T RANSFERRING T EST
To test the potential of IGP to transfer from one (training) scenario to other different (generalization) scenarios, we consider a challenging application in which we ﬁrst conduct ofﬂine training of IGP on completely synthetic graphs and directly generalize the trained model to the real datasets for fast online GP (without additional optimization).
In particular, we expect the synthetic graphs used for ofﬂine training cover various structural properties of real-world network systems.
We adopt LFR-Net (see Appendix C.2) as the model to generate synthetic training graphs.
In contrast to the parameter settings of LFR-Net in our quantitative evaluation (see Appendix C.2), we randomly set N ∈ [500 , 6000] , k ∈ [10 , 100] , k max ∈ [50 , 200] , c min ∈ [5 , 100] , c max ∈ [50 , 400] , and µ ∈ [0 . 1 , 0 . 7] to independently generate 2 , 000 synthetic snapshots, where we ensure that k max > k and c max > c min for each snapshot.
In our experiments, we conducted ofﬂine training on the aforementioned synthetic snapshots and generalized the trained model to test sets of the four real datasets used in our quantitative evaluation (i.e., Taxi , Reddit , AS , and Enron as introduced in Appendix C.2).
We used the same layer conﬁgurations as in Table 4 with respect to each real dataset and tried different parameter settings of IGP-M and IGP-C to report their best average quality.
The recommended parameter settings of IGP-M and
IGP-C are shown in Table 21.
Evaluation results on test sets of the four real datasets (in terms of modularity and NCut) are depicted in Table 22, where ’Real’ and ’Syn.’
represent the results with respect to (i) training on training set of the original real datasets (same as the results in Table 8 and [Figure]
[Figure] Table 10) and (ii) training on the aforementioned synthetic snapshots, respectively.
According to
Table 22, we have the following key observations.
• In some cases (e.g., on Taxi and Reddit ), the quality metrics of training on synthetic graphs are close to those of training on the original real datasets for both IGP-M and IGP-C, which are also competitive to the strong baselines as illustrated in Table 8 and Table
It preliminarily validates the potential of IGP to be transferred from completely synthetic graphs to real datasets.
However, in other cases, there are still signiﬁcant quality declines for both the variants of IGP, which indicate that the transferring capability of IGP is largely affected by the domain similarity between the synthetic training set and real test set.
The negative transferring can be alleviated by carefully generating or selecting synthetic snapshots with the assistant of validation information from the target real datasets.
We leave the study of this application in our future work.
• Compared with IGP-C, IGP-M has better transferring quality metrics in most cases.
The quality declines of IGP-C are also more signiﬁcant than those of IGP-M. It further validates that the modularity matrix Q t used in IGP-M is more informative than the normalized adjacency matrix M t used in IGP-C.
Note that IGP has similar motivations with SOTA pre-training methods for graphs, e.g., PretrainGNN (Hu et al., 2020) and GCC (Qiu et al., 2020), in which we pre-train a GNN-based model on historical graphs (in an ofﬂine way) and generalize it to new snapshots.
The major difference between IGP and SOTA pre-training methods is that a ﬁne-tuning procedure with another optimization algorithm (based on a certain ﬁne-tuning loss) on new test graphs is required for most pretraining methods after the ofﬂine (pre-)training.
In particular, the ﬁne-tuning procedure may still be time-consuming for new test graphs, where several iterative optimization algorithms (e.g., gradient descent) are applied.
In this study, we focus on the NP-hard challenge of GP and try to achieve a better trade-off between quality and efﬁciency.
To some extent, the proposed IGP framework sacriﬁces some opportunities to ﬁne-tune the (pre-)trained model, which ensures the low runtime on new test graphs.
In our future research, we intend to consider a promising application that we pre-train a high-quality GP model on completely synthetic graphs and fast ﬁne-tune the (pre-)trained model on new graphs (via some efﬁcient ﬁne-tuning strategies), with the guarantee of high partitioning quality and low runtime. [Table]
E P OSSIBLE F UTURE W ORK
We summarize several promising future research directions as follows.
Further Reducing the Runtime.
According to the results in Table 11 and Table
[Table] the downstream clustering module (i [Table] e., K Means with 10 independent runs) consumes most of the runtime of IGP.
Although replacing the clustering module with an FC output layer (trained in an E2E way) to directly output the partitioning results can further reduce the total runtime, the downstream clustering module is a key component enables IGP to tackle online GP with non-ﬁxed K .
In contrast, existing E2E GP methods still need to be trained from scratch for new graphs with non-ﬁxed K .
In our future research, we intend to further reduce the runtime of IGP by exploring a generic E2E scheme that can directly derive the GP results (e.g., via an output layer) and tackle the online GP with non-ﬁxed
K . One possible solution is to simultaneously train multiple output layers (with different output dimensionality) to derive multiple partitioning results with respect to all the possible values of K .
Another possible strategy is to output partitioning results in a hierarchical structure (that covers the results with respect to all the possible values of K ) based on the idea of hierarchical clustering. Dealing with Distribution Shifts.
As discussed in Appendix D.2, our IGP framework has similar motivations with SOTA pre-training methods for graphs (Hu et al., 2020; Qiu et al., 2020).
Namely, we conduct the ofﬂine (pre-)training of inductive GNNs (regardless of time cost) but sacriﬁce some opportunities to ﬁne-tune IGP on new test graphs for a better trade-off between quality and efﬁciency of the online GP.
The ﬁne-tuning on new graphs of targeted scenarios can avoid the negative transferring by utilizing validation information from new targeted graphs.
Moreover, to select the proper set of pre-training datasets is still an open issue in recent researches regarding the pre-training and ﬁne-tuning scheme.
In our future work, we intend to conduct the ofﬂine (pre-)training of IGP on completely synthetic graphs (that covers properties of various real network systems) and explore an efﬁcient strategy to ﬁne-tune the model on real new system snapshots (of other scenarios) with the guarantee of high partitioning quality and low runtime.
Scaling Up to Large Graphs. Scaling GNNs up to graphs with large number of nodes is a signiﬁcant direction in recent researches, where several sampling strategies are used to split a large graph into multiple subgraphs (Chen et al., 2018;
Huang et al., 2018; Zeng et al., 2020), i.e., mini-batches with small number of nodes.
Note that IGP is an inductive framework across graphs, which can also be trained on and generalized to multiple subgraphs sampled from a large graph (in terms of large number of nodes).
Most of existing GNNs that can be scaled up to large graphs usually focus on (semi-)supervised node-level tasks (e.g., node classiﬁcation) with available node attributes, while we consider challenging inductive unsupervised node-level task (i.e., GP) across graphs without graph attributes in this study.
In our future work, we intend to further explore efﬁcient (i) feature extraction and (ii) subgraph sampling strategies for IGP on large graphs without available attributes.
In particular, we can ﬁrst split a large graph into multiple small subgraphs via a sampling module and extract the neigh-induced features { X t } for each sampled graph.
IGP can then be applied to multiple sampled graphs and derive graph embedding { U t } with respect to each subgraph.
One possible challenge in this application is to ensure the consistency of embeddings { U t } of multiple subgraphs sampled from a common large graph (e.g., embeddings of all the subgraphs are mapped into a common latent space) without any auxiliary information given by the node attributes. Theoretical Bound Analysis.
Existing theoretical analysis regarding the combinatorial nature of GP (Newman, 2006; Von Luxburg, 2007) is usually based on one single graph.
However, we considered the GP across multiple graphs using the inductive nature of GNN.
The GP quality on a test graph is not only related to the expressive ability of inductive GNNs, but also related to the similarity between the training and test graphs.
To give strict theoretical bound analysis of IGP (e.g., for modularity maximization and NCut minimization) is a signiﬁcant direction of our feature work.
We intend to extend the theoretical analysis of GP on one single graph to multiple associated graphs with the consideration of (i) the expressive ability of GNN and (ii) similarity between graphs in terms of their underlying distributions.
Application to Attributed Graphs.
In this study, we considered inductive GP without attributes.
Our IGP method can also be easily extended to include node attributes by concatenating the reduced neighbor-induced features Z t with the attribute matrix describing node attributes.
However, prior research (Qin et al., 2018; Chunaev et al., 2020; Qin & Lei, 2021) has demonstrated that there exist complicated correlations between graph topology and attributes.
On the one hand, attributes may provide some complementary information beyond topology that can further improve the quality of some tasks including GP.
On the other hand, attributes may also carry some noise and mismatched characteristics, which result in unexpected quality declines , e.g., in terms of modularity and NCut for
GP.
In our feature work, we intend to consider an adaptive incorporation scheme between these two heterogeneous sources for inductive GP.
Concretely, when attributes carry consistent characteristics with topology, they can fully play their role in improving the GP quality.
In contrast, when topology ‘mismatches’ with attributes, we need to control the contribution of attributes to avoid unexpected quality decline.