T RADING Q UALITY FOR E FFICIENCY OF G RAPH P ARTITIONING : A N I NDUCTIVE M ETHOD ACROSS G RAPHS
Anonymous authors
Paper under double-blind review
A BSTRACT
Many applications of network systems can be formulated as several NP-hard combinatorial optimization problems regarding graph partitioning (GP), e.g., modularity maximization and NCut minimization.
Due to the NP-hardness, to balance the quality and efﬁciency of GP remains a challenge.
Existing methods use machine learning techniques to obtain high-quality solutions but usually have high complexity.
Some fast GP methods adopt heuristic strategies to ensure low runtime but suffer from quality degradation.
In contrast to conventional transductive GP methods applied to a static graph, we propose an inductive graph partitioning (IGP) framework across multiple evolving graph snapshots to alleviate the NPhard challenge.
IGP ﬁrst conducts the ofﬂine training of a novel dual graph neural network on historical snapshots to capture the structural properties of a system.
The trained model is then generalized to newly generated snapshots for fast highquality online GP without additional optimization, where a better trade-off between quality and efﬁciency is achieved.
IGP is also a generic framework that can capture the permutation invariant partitioning ground-truth of historical snapshots in the ofﬂine training and tackle the online GP on graphs with non-ﬁxed number of nodes and clusters.
Experiments on a set of benchmarks demonstrate that IGP achieves competitive quality and efﬁciency to various state-of-the-art baselines.
I NTRODUCTION
For various complex systems, e.g., communication networks, graph is a generic model to describe the entities and their relations using a set of nodes and edges.
Graph partitioning (GP) is a classic inference task that aims to partition the nodes of a graph into several groups (i.e., clusters) with dense linkage distinct from other groups.
Since the extracted clusters are believed to correspond to several real-world substructures of a system, e.g., cells in wireless networks (Dai & Bai, 2017), many network applications are formulated as GP tasks (Qin et al., 2019; Patil & Kulkarni, 2021).
Mathematically, GP can be described as some NP-hard combinatorial optimization problems, e.g., modularity maximization (Newman, 2006) and normalized cut (NCut) minimization (Von Luxburg, 2007).
Due to the NP-hardness, to balance the quality and efﬁciency of GP remains a challenge, but some real applications have both requirements of high quality and low runtime, e.g., accurate GP on a wireless cellular network with several thousand nodes in a few seconds (Dai & Bai, 2017).
On the one hand, prior work has demonstrated the ability of machine learning (ML) techniques to achieve high partitioning quality.
Typical ML-based methods include non-negative matrix factorization (Wang et al., 2011) and generative probabilistic models (Karrer & Newman, 2011).
Graph embedding emerges as a promising technique for GP in recent studies.
These methods ﬁrst learn low-dimensional node representations via random walk (Perozzi et al., 2014) or deep learning (Wang et al., 2016; Yang et al., 2016) to capture high-order proximities and nonlinear characteristics of a graph.
The GP result is derived by feeding the learned embedding to a downstream clustering module (e.g, K Means).
However, most ML-based methods have to use several iterative optimization algorithms (e.g., gradient descent) to learn large-scale parameters with inevitably high complexity.
On the other hand, to reduce the overall runtime and satisfy real-time constraints of some applications is another major focus.
Some fast GP methods apply heuristic strategies to approximate conventional objectives, e.g., greedy modularity maximization (Blondel et al., 2008) and multi-level coarsening for graph-cut minimization (Dhillon et al., 2007).
Several graph embedding approaches also adopt fast approximation to reduce the inference time, e.g., random projection for high-order proximities (Zhang et al., 2018).
Despite their high efﬁciency (e.g., low runtime), they may still suffer from unexpected quality decline due to the inherent information loss of heuristic approximation.
Most state-of-the-art (SOTA) GP methods are inherently transductive , which explore structural properties of a static graph and can only tackle the GP on such a unique graph.
We try to achieve a better trade-off between quality and efﬁciency from a new inductive perspective across multiple associated graphs of a system .
An inductive graph partitioning (IGP) framework is proposed based on the fact that most real-world complex systems generate a set of snapshots via common knowledge in terms of several underlying distributions , e.g., power-law distributions in social networks.
Such a hypothesis is also widely adopted in the simulation of various network systems (Wehrle et al., 2010).
IGP ﬁrst trains a high-quality GP model on historical system snapshots in an ofﬂine way, which aims to fully capture the inherent structural properties of a system (regardless of time cost).
The trained model is then generalized to newly generated snapshots for online GP without additional optimization, which signiﬁcantly reduces the runtime on these new graphs and is believed to have high-quality GP results.
In some real applications, it’s usually assumed that we have enough time to prepare a high-quality model using historical data in an ofﬂine way, which is also one-time-effort only.
Our main focus is to achieve a better trade-off between quality and efﬁciency of the online GP on new graphs after deploying the trained model to a system (with model parameters ﬁxed).
Technically, IGP follows a graph embedding scheme for the unsupervised node-level task (i.e., GP) across multiple graphs based on the inductive nature of graph neural networks (GNN).
Although
SOTA inductive GNNs have the potential to be generalized to new unseen nodes and graphs (Hamilton et al., 2017; Veli ˇ ckovi ´ c et al., 2017), they still suffer from the following limitations for GP.
In this study, we consider GP on multiple graphs where topology is the only available information source, i.e., without graph attributes.
As some systems allow the addition and deletion of entities, we assume that the number of nodes N is non-ﬁxed for multiple graphs.
Although existing inductive GNNs can be applied to graphs with non-ﬁxed N , they were originally designed for attributed graphs and rely on ﬁxed dimensionality of node features.
Our preliminary experiments indicate that some classic settings of GNN for the case without node features (e.g., using a constant matrix with ﬁxed dimensionality as the feature input) may suffer from poor GP quality.
Some methods extract features via dimensionality reduction (e.g., PCA (Nazi et al., 2019)) to map the topology (e.g., adjacency matrix) with non-ﬁxed N to a ﬁxed-dimensional feature space, which is still time-consuming.
Moreover, different system snapshots can be assigned with different number of clusters K .
Some inductive GNN based approaches adopt an end-to-end (E2E) framework to achieve a better approximation to conventional GP objectives (e.g., GAP (Nazi et al., 2019) for NCut minimization and ClusNet (Wilder et al., 2019) for modularity maximization), with partitioning results derived via a fully-connected (FC) output layer.
Despite their high quality, they can only tackle the inductive GP across graphs with ﬁxed K , due to the ﬁxed dimensionality of the output layer.
These E2E methods still need to be trained from scratch for new graphs with non-ﬁxed K , which is time-consuming.
Although prior studies have demonstrated the ability of inductive GNNs to tackle (semi-)supervised tasks (e.g., node classiﬁcation) on new unseen nodes and graphs (Hamilton et al., 2017; Veliˇckovi´c et al., 2017), few of them consider unsupervised node-level tasks (e.g., GP) across graphs.
Our experiments also indicate that some standard settings of inductive GNNs for unsupervised tasks (e.g., using classic unsupervised training loss of graph embedding for GNN (Hamilton et al., 2017)) may still lack robustness for the online GP on newly generated graphs.
In addition to obtaining a better trade-off between quality and efﬁciency over conventional transductive GP methods via a novel unsupervised node-level inductive framework across graphs (i.e., IGP), we also make the following contributions to address the aforementioned limitations of inductive GNNs.
(i) To tackle the online GP with non-ﬁxed N , we develop an efﬁcient feature extraction strategy for inductive GNNs via graph coarsening.
(ii)
In contrast to E2E methods, IGP adopts an inductive graph embedding scheme across graphs , which can tackle the online GP with non-ﬁxed
K .
(iii) Note that GP is an unsupervised node-level task, where the cluster labels are permutation invariant, e.g., label sequences (0 , 1 , 1) and (1 , 0 , 0) are the same in terms of GP.
IGP can further incorporate such label information (e.g., GP ground-truth) of historical graphs to the ofﬂine training by combining conventional GP objectives (e.g., modularity maximization and NCut minimization) with a novel dual GNN structure , which ensures strong robustness for online GP on new graphs.
P ROBLEM S TATEMENTS AND P RELIMINARIES
In this study, we consider GP on a set of graphs S =
{G 1 , G 2 · · · , G T } with T snapshots from a common system.
Each snapshot G t ∈ S can be represented as a 2-tuple G t
= ( V t , E t ) with V t
= { v t 1 , . . . , v tN t } and E t = { ( v ti , v tj ) (cid:12)(cid:12) v ti , v tj ∈ V t } as the sets of nodes and edges, respectively.
For each G t , topology is the only available information source without any graph attributes.
We use an adjacency matrix A t ∈ (cid:60) N t × N t to describe its topology with N t nodes, where ( A t ) ij =
( A t ) ji =if ( v ti , v tj ) ∈ E t and ( A t ) ij =
( A t ) ji = 0 otherwise.
Since some systems allow addition and deletion of entities, we assume that different snapshots in S can have different node sets, i.e., ∃G t , G s ∈ S s.t. V t (cid:54) =
V s .
For simplicity, the temporal dependency of S (i.e., node index correspondence among node sets {V 1 , · · · , V T } ) is not considered in this study.
We follow the hypothesis widely adopted in the simulation of various network systems (Wehrle et al., 2010) that snapshots in S are independently generated via several underlying distributions of a common system .
Graph Partitioning.
Given a graph G t , GP aims to partition the node set V t into K t subsets (i.e., clusters) C t =
{ C t 1 , · · · , C tK t } so that (i) within each cluster the linkage is dense but (ii) between different clusters the linkage is relatively loose.
The result C t also satisﬁes C tr
∩ C ts = ∅ for ∀ r (cid:54)
= s .
Mathematically, GP can be formulated as several NP-hard combinatorial optimization problems, e.g., NCut minimization (Von Luxburg, 2007) and modularity maximization (Newman, 2006).
Given a graph G t , NCut minimization aims to get the GP result C t that minimizes the NCut metric:
[Equation]
where ¯ C tr = V t − C tr is the complementary set of C tr ; cut( C tr , ¯ C tr )
= (cid:80) v i ∈ C tr ,v j ∈ ¯ C tr
( A t ) ij is the cut between C tr and ¯ C tr ; vol( C tr )
= (cid:80) v i ,v j ∈ C tr
( A t ) ij is the volume of C tr .
The objective (1) can be equivalently expressed in the following matrix form:
[Equation]
where L t =
I N t − D − 0 .
5 t
A t D − 0 .
5 t is the Laplacian matrix of A t ; D t = diag( d t 1 , · · · , d tN t ) is a diagonal matrix with d ti = (cid:80) j
( A t ) ij ; I N is an N -dimensional identity matrix.
H t ∈ (cid:60) N t × K t is the membership indicator, where ( H t ) ir =
[ d ti · vol( C tr ) − 1 ] 0 .
5 if v ti ∈ C tr
and ( H t ) ir = 0 otherwise.
Modularity maximization is another conventional NP-hard objective of GP.
Given a graph G t , it aims to ﬁnd a partition C t that maximizes the modularity metric:
[Equation]
where e = (cid:80) i d ti / 2 is the number of edges.
The objective can also be rewritten in a matrix form:
[Equation]
where Q t ∈ (cid:60) N t × N t is the modularity matrix with ( Q t ) ij =
( A t ) ij − d ti d tj / (2 e ) .
H t ∈ (cid:60) N t × K t is the membership indicator, where ( H t ) ir =
if v ti ∈ C tr and ( H t ) ir = 0 otherwise.
Inductive Graph Embedding.
Given a single graph G t , conventional transductive graph embedding aims to learn a function f : { v ti } (cid:55)→ { u ti ∈ (cid:60) 1 × k } that maps each node v ti to a k -dimensional vector u ti .
The key characteristics of G t (e.g., clustering structure) should be preserved in the embedding space, where the node pair ( v ti , v tj ) with similar properties (e.g., in the same cluster) should have similar representations ( u ti , u tj ) (e.g., with close distance in the embedding space).
The learned representations { u ti } are used as the input of several downstream tasks on G t including GP.
To alleviate the NP-hard challenge of GP, we consider a novel inductive graph embedding scheme, where model parameters of f are shared by all the snapshots in S .
We divide S into a training set Γ ⊂ S and a test set Γ (cid:48)
=
S − Γ , which represent the sets of historical and newly generated snapshots, i.e., snapshots in Γ should occur before those in Γ (cid:48) .
We ﬁrst train f on Γ in an ofﬂine way to fully capture the structural properties of historical snapshots.
Although graph embedding and GP are unsupervised procedures, we assume that the partitioning result C t (i.e., label information) of each historical graph G t ∈ Γ is available in the ofﬂine training.
Note that cluster labels in C t are
[Figure]
[Figure]
vv 7 vv 1 vv 3 v 4 vvv 7 v 8 vvv 3 v 4 vvv 7 v
In Feature , message passing occurs in
[Figure]
permutation invariant, e.g., label sequences (0 , 1 , 1) and (1 , 0 , 0) are the same for GP.
We consider the integration of such label information in the ofﬂine training.
The label information described by historical partitioning results { C t |G t ∈ Γ } can be from the (i) ground-truth of system applications or (ii) results of some high-quality (but usually time-consuming) GP baselines. It indicates another application that we can use the partitioning results of some strong baselines to regularize the ofﬂine training of IGP.
After the ofﬂine training, we directly obtain the embedding { u t (cid:48) i } from f for each new graph G t (cid:48)
∈ Γ (cid:48) with model parameters ﬁxed.
The online GP on each new graph G t (cid:48) is the downstream task, where we assume the number of clusters K t (cid:48) is given.
The partitioning result of G t (cid:48) is derived by applying a clustering algorithm (e.g., K Means) to the embedding { u t (cid:48) i } .
M ETHODOLOGY
We propose a novel IGP framework with the overview shown in Fig. 1, including the (i) ofﬂine training of a dual GNN structure on historical graphs and (ii) online generalization to new graphs for fast high-quality GP via an inductive graph embedding scheme.
A simple running example (with 8 nodes and 2 clusters) regarding the ofﬂine training of the dual GNN structure is given in Fig.
Model Architecture.
Inspired by adversarial auto-encoder (AAE) (Makhzani et al., 2015), we introduce a novel dual GNN structure, including a generative network G and a discriminative network
D that are jointly optimized via an adversarial process.
To the best of our knowledge, the dual GNN structure is our original design, which is different from SOTA AAE-based graph embedding methods (Pan et al., 2018) that regularize the embedding via the adversarial process between a probability distribution.
In contrast, our dual GNN structure incorporates the permutation invariant label information of historical graphs {G t ∈ Γ } to the learned embedding via (i) a parameter sharing strategy and (ii) an adversarial process between {G t } and auxiliary label-induced graphs {G ( g ) t } .
Concretely, in the ofﬂine training, we assume the topology and GP result (used as the ‘ground-truth’) of each historical graph G t are available, which are described by an adjacency matrix A t ∈ (cid:60) N t × N t and an indicator matrix R t ∈ (cid:60) N t × K t .
We deﬁne ( R t ) ir = 1 if node v ti is in cluster C tr by the ‘ground-truth’ and ( R t ) ir = 0 otherwise.
Due to the disjoint constraint of GP, only one entry in each row of R t is 1 with other entries in the same row set to 0 .
In addition to the original graph G t , an auxiliary label-induced graph G ( g ) t , with A ( g ) t =
R t R T t as the auxiliary adjacency matrix describing its topology, is introduced to encode the structure of ‘ground-truth’ based on Theorem 1.
( Theorem 1 ) For a historical graph G t with K t clusters, its auxiliary label-induced graph G ( g ) t has
K t fully connected components with each component corresponding to one unique cluster of G t .
Namely, there is an edge between nodes v ti and v tj in G ( g ) t only when they are in the same cluster, thus encoding the label information of ‘ground-truth’.
In the example of Fig. 2 (with 8 nodes), G ( g ) t has 2 fully connected components { v 1 , v 2 , v 3 , v 4 } and { v 5 , v 6 , v 7 , v 8 } with respect to the 2 clusters in the GP result (i.e., ‘ground-truth’) of G t .
The proof of Theorem 1 is given in Appendix A.
The generative network G includes a feature encoder and a label-induced encoder , forming a dual GNN structure with shared parameters δ G .
The feature encoder takes A t and a feature matrix X t as inputs, and derives graph embedding U t , i.e., U t =
G ( A t , X t ; δ G ) .
The label-induced encoder takes A ( g ) t and X t as inputs, and outputs auxiliary embedding U ( g ) t , i.e., U ( g ) t =
G ( A ( g ) t , X t ; δ G ) .
GNN is originally designed for attributed graphs, where each node v ti must have a feature vector described by the i -th row of a feature matrix X t .
In this study, we consider GP without node attributes.
Instead of using conventional settings of GNN for graphs without attributes (e.g., use a constant matrix for the input features), we extract additional structural feature X t based on the neighbor similarity encoded in A t .
IGP captures the (permutation invariant) label information of historical graphs by combining the dual GNN structure with classic GP objectives.
We adopt the objectives of (i) modularity maximization and (ii) NCut minimization as two examples, corresponding to two settings of X t .
For modularity maximization in (4), the modularity matrix Q t encodes the reweighted neighbor similarity based on A t , so we let X t =
Q t .
For NCut minimization in (2), the Laplacian matrix L t gives the primary characteristics of graph structure, where M t = D − 0 .
5 t
A t D − 0 .
5 t is the key component regarding neighbor similarity.
Hence, we let X t =
M t .
Both Q t and M t are the reweighting of A t , where the pair of nodes ( v ti , v tj ) with similar neighbor-induced features (( X t )
i, : , ( X t ) j, : ) are more likely to be partitioned into the same cluster.
Our experiments further demonstrate that the extraction of X t is essential to ensuring high partitioning quality of IGP.
However, the feature dimensionality of X t is N t (i.e., the number of nodes) which may not be ﬁxed over t .
We introduce an efﬁcient feature extraction module via heavy-edge matching (HEM) graph coarsening (Hendrickson & Leland, 1995) to map X t ∈ (cid:60) N t × N t to a matrix Z t ∈ (cid:60) N t × L with ﬁxed dimensionality L .
For N t > L , we extract reweighted edges E ( w ) t = { w ( v ti , v tj ) | w ( v ti , v tj ) =
( X t ) ij , ( A t ) ij = 1 } and apply HEM to E ( w ) t .
It merges the original graph G t with N t nodes into a supergraph G ∗ t with L supernodes via a greedy strategy of continuously merging the node pair with largest weight in E ( w ) t into a supernode, e.g., merging a graph with 8 nodes into 2 supernodes in Fig.
HEM outputs a coarsening matrix C t ∈ (cid:60)
N t × L , where ( C t ) ij = | v t ∗ j | − 0 .
5 if node v ti is merged into supernode v t ∗ j and ( C t ) ij = 0 otherwise.
We then let Z t =
X t C t be the reduced features.
Since C t is a sparse matrix, one can obtain Z t by setting its j -th column to ( Z t ) : ,j = (cid:80) v i ∈ v t ∗ j ( C t ) ij ( X t ) : ,i .
For N t <
L , we let Z t =
[ X t , 0 N t × ( L − N t ) ] .
In contrast to existing methods using HEM to reduce topology complexities (Dhillon et al., 2007), we’re the ﬁrst to extract feature input Z t for inductive GNNs (with non-ﬁxed N t ) via HEM.
It’s more efﬁcient than dimensionality reduction (e.g., PCA of A t ) used in SOTA GNNs (Nazi et al., 2019), since HEM is widely used in fast GP methods (Liang et al., 2018).
The derived Z t is also more informative than the input features in some classic settings of inductive GNNs (e.g., let Z t be a constant matrix (Xu et al., 2018)).
Let F ( l − 1) t and F ( l ) t be the input and output of the l -th layer of feature encoder or label-induced encoder in G with F (0) t =
Z t .
The l -th GNN encoder layer is deﬁned as
[Equation]
where ˆA t =
A (cid:48) t
+ I N t
( A (cid:48)
t ∈ { A t , A ( g ) t } ) is the adjacency matrix with self-edges; ˆD t = diag( ˆ d t 1 , · · · , ˆ d tN t ) is the diagonal degree matrix; W ( l − 1) G is the trainable parameter matrix shared by the two encoders; ( F ( l ) t ) i, : is the latent feature vector of node v ti .
In (5), ( F ( l ) t ) i, : is the nonlinear aggregation of the features of { v ti } ∪ n ( v ti ) in the previous layer, where n ( v ti ) is the neighbor set of v ti .
It’s also known as the message passing of GNN, where neighbors of v ti propagate their features to v ti for aggregation.
By Theorem 1, A ( g ) t ensures that message passing of the label-induced encoder only occurs in each connected component with respect to each cluster in the ‘groundtruth’ (e.g., the 2 components in Fig. 2), while message passing of the feature encoder occurs in the original topology of G t .
The last layers of the feature and label-induced encoders output the graph embedding U t ∈ (cid:60) N t × k and auxiliary label-induced embedding U ( g ) t ∈ (cid:60) N t × k , where
U ( g ) t = G ( A ( g ) t , X t ) preserves more informative label-induced properties than U t
= G ( A t , X t ) .
We further use the non-linear inner product of U t to reconstruct the neighbor-induced features X t : ˜X t = tanh( U t U T t ) .
The discriminative network D is an auxiliary classiﬁer that aims to distinguish U ( g ) t from U t , while the generative network G tries to generate plausible embedding U t to fool D .
Such an adversarial process helps G to generate embedding U t close to U ( g ) t .
We denote D as y t =
D ( S t ; δ D ) , where
S t ∈ { U t , U ( g ) t } and δ D are the input and set of model parameters; y t ∈ (cid:60) N t is a column vector with ( y t ) i as probability that ( S t ) i, : =
( U ( g ) t ) i, : rather than ( S t ) i, : =
( U t ) i, : .
Let P ( l − 1) t and P ( l ) t be the input and output of the l -th layer in D with P (0)
t =
S t .
The l -th layer of D is deﬁned as
[Equation]
where { W ( l − 1) D , b ( l − 1) D } are trainable model parameters.
In particular, we use sigmoid as the activation function of the last layer instead of ReLU .
Model Optimization.
As in Fig. 2, the ofﬂine training of IGP is based on three objectives: (i) adversarial learning (AL), (ii) feature reconstruction (FR), and (iii) clustering regularization (CR).
The AL objective helps incorporate the permutation invariant label information of historical graphs to the unsupervised embedding learning via an adversarial process between D and G .
On the one hand, D tries to distinguish U ( g ) t from U t .
The objective of D with respect to a graph G t is
[Equation]
On the other hand, G tries to derive plausible embedding U t to fool D by minimizing the following loss with respect to G t :
[Equation]
[Equation]
Such an adversarial process directs G to output the embedding U t that is close to U ( g ) t , thus enabling
G to capture the permutation invariant label information of historical graphs.
The FR loss forces G to derive the embedding U t that encodes the key properties of the neighborinduced features X t with respect to G t by minimizing the reconstruction error between X t and ˜X t :
[Equation]
In addition to the AL loss, IGP can also capture the (permutation invariant) label information via the regularization of conventional GP objectives, e.g., modularity maximization (4) and NCut minimization (2).
We introduce the CR objective for a graph G t that minimizes the following loss:
[Equation]
where H t is the membership indicator encoding the ‘ground-truth’ with the same deﬁnitions in (4) and (2).
Since the constraints on H t , e.g., tr( H Tt H t )
=
N t and H Tt H t
= I K t , are always satisﬁed for the given ‘ground-truth’, we don’t need to consider the discrete constraints on H t .
To the best of our knowledge, we’re the ﬁrst to use the CR objective to incorporate the permutation invariant label information (encoded by H t from historical training data) to the unsupervised embedding learning, whereas H t is the parameter to be optimized or output in existing GP methods (Wilder et al., 2019).
Finally, we drive the objective of G with respect to G t by combining (9), (10), and (11):
[Equation]
where { α, β } are hyper-parameters to balance L FR and L CR .
In the joint ofﬂine optimization of D and G , we use the Xavier method to initialize model parameters { δ D , δ G } .
The Adam optimizer is applied to iteratively update δ D and δ G based on the gradients of loss (8) and (12).
In each epoch, we randomly sample a certain number of historical graphs p from the training set to update { δ D , δ G } .
Finally, we save the model parameters { δ ∗ D , δ ∗ G } that result in the best average quality on the validation set within a certain number of epochs n .
After ofﬂine training, we generalize IGP to new system snapshots for fast online GP.
As in Fig. 1, for each new graph G t (cid:48) , we derive its embedding U t (cid:48) by passing { A t (cid:48) , X t (cid:48) } forward the feature encoder of G with its parameters δ G ﬁxed.
As we assume that the number of clusters K t (cid:48) is given, we apply
K Means to U t (cid:48) , which outputs the partitioning result with respect to the given K t (cid:48) .
Therefore, the runtime of online GP on G t (cid:48) includes (i) the efﬁcient feature extraction of Z t , (ii) one feedforward propagation through the feature encoder , and (iii) the downstream clustering.
Due to the space limit, we conclude the ofﬂine training and online generalization procedures of IGP in Appendix B.
E XPERIMENTS
Datasets.
We evaluate the effectiveness of IGP on 2 synthetic benchmarks (with 7 settings) and 4 real datasets.
Statistics of the datasets are depicted in Table 1, where T , N , |E| , and K are the number of graphs, nodes, edges, and clusters.
The 11 datasets cover the cases with (i) ﬁxed N and
K , (ii) ﬁxed N but non-ﬁxed K , as well as (iii) non-ﬁxed N and K .
GN-Net (Girvan & Newman, 2002) and LFR-Net (Lancichinetti et al., 2008) are widely-used synthetic benchmarks that can simulate properties of real-world network systems.
In each benchmark, we generated a set of graphs and their partitioning ground-truth via several distributions as a simulation of real systems.
GN-Net and LFR-Net use p in and µ to control the clustering structures of each snapshot.
We denote the GN-Net with a setting of p in as GN- p in ( p in ∈ { 0 . 5 , 0 . 4 , 0 . 3 } ).
Thedatasets of GN-Net are with ﬁxed N and K .
We use L ( c, µ ) ( c ∈ { f, n } , µ ∈ { 0 . 3 , 0 . 6 } ) to represent the LFR-Net with a setting of µ , where f and n denote the cases with fixed and non-ﬁxed N .
As p in decreases and µ increases, the clustering structures become increasingly difﬁcult to identify.
Taxi (Piorkowski et al., 2009), Reddit (Yanardag & Vishwanathan, 2015), Enron (Rossi & Ahmed, 2015), and AS (Leskovec et al., 2005) are real datasets from a vehicle network, an email network, a social network, and an autonomous system.
As the 4 real datasets don’t provide ground-truth regarding the K value of each graph, to ensure the fairness of comparison, where each graph is with a common K for all the methods, we applied the widely-used Louvain algorithm (Blondel et al., 2008) to determine K for each graph.
Details of the datasets are elaborated in Appendix C.2.
Baselines.
We compared IGP over 15 baselines with 4 categories.
(i) Symmetry NMF ( SNMF )
(Wang et al., 2011) and spectral clustering ( SC ) (Von Luxburg, 2007) are classic transductive GP methods.
(ii) VERSE
(Tsitsulin et al., 2018), MILE (Liang et al., 2018), and PhUSION (Zhu et al., 2021) are SOTA graph embedding approaches for high-quality representations, while NPR (Yang et al., 2020), RandNE (Zhang et al., 2018), and ProNE (Zhang et al., 2019) are efﬁcient embedding methods with fast approximation.
All these embedding methods are transductive .
(iii) Metis (Hendrickson & Leland, 1995), hMetis (Selvakkumaran & Karypis, 2006), and GraClus (Dhillon et al., 2007) are classic fast ( transductive ) baselines with heuristic approximation.
(iv) GraSAGE (with mean aggregator)
(Hamilton et al., 2017) and GAT (Veli ˇ ckovi ´
c et al., 2017) are SOTA inductive
GNNs, while GAP (Nazi et al., 2019) and ClusNet (Wilder et al., 2019) are E2E methods based on inductive GNNs.
Besides the aforementioned methods, our comparison also includes some other strong baselines (e.g., node2vec (Grover & Leskovec, 2016)), with details given in Appendix C.3.
Both variants of IGP were evaluated.
For simplicity, we use IGP-M and IGP-C to denote the variants adopting the objectives of modularity maximization and NCut minimization, respectively.
Quantitative Evaluation.
We consider GP on multiple graph snapshots, where K is given for each graph.
For all the datasets, we adopted the quality metrics of modularity and NCut as in (3) and (1).
For datasets with ground-truth (i.e., GN-Net and LFR-Net ), we also used normalized mutual information (NMI) and accuracy (AC)
(Cui et al., 2018) to measure the correspondence between the partitioning result and ground-truth.
Usually, larger modularity, NMI and AC, as well as smaller
NCut indicate better quality.
Moreover, the total runtime of each method was also recorded.
Most conventional baselines (e.g., SC ) are transductive , which can only be applied to a static graph, but IGP alleviates the NP-hard challenge of GP in an inductive framework across graphs.
To illustrate such superiority of IGP beyond transductive baselines, for each dataset, we used the ﬁrst 80% of the graphs as the training set Γ T with the remaining 10% and 10% deployed as the validation set Γ V and test set Γ (cid:48) .
In this setting, Γ T ∪ Γ V and Γ (cid:48) represent the sets of historical and newly generated graphs, respectively.
We conducted the ofﬂine training of IGP on Γ T and generalized it to Γ (cid:48) for online GP.
In the ofﬂine training, we save the model parameters of IGP with the best average quality on Γ V , where we use NMI and modularity as the validation quality metric for datasets with and without ground-truth, respectively.
Since we focus on the trade-off between quality and efﬁciency on new graphs of a system, the evaluation was conducted on Γ (cid:48) , where we had to apply transductive baselines to each test graph in Γ (cid:48) from scratch due to their transductive nature.
For inductive GNN baselines (i.e., GraSAGE and GAT ), we use the same training and evaluation settings with IGP, i.e., ofﬂine training on Γ T and online generalization to Γ (cid:48) .
Since we consider GP on graphs without attributes, we use a constant matrix 1 N t × L (with ﬁxed feature dimensionality L ), where each entry is set to 1 , as the input features of these GNNs, which is a widely used setting of inductive GNNs for the case without attributes (Xu et al., 2018).
In the ofﬂine training of inductive GNN baselines, we adopted the widely used unsupervised loss of GraSAGE (Hamilton et al., 2017) to train them on Γ T , since most existing inductive GNNs (even with a supervised loss like crossentropy) cannot integrate the (permutation invariant) label information from Γ T .
For the E2E GNN methods (i.e., GAP and ClusNet ), we could only conduct the ofﬂine training and fast online generalization on datasets with ﬁxed K (i.e., on GN-Net ), but we had to train them from
[Table] [Table]
scratch on the test set of other datasets with non-ﬁxed K , due to the ﬁxed dimensionality of their E2E output layers.
As recommended in (Nazi et al., 2019) and (Wilder et al., 2019), we used PCA and node2vec to map the adjacency matrices { A t } to a feature space with ﬁxed dimensionality for the input features of GAP and ClusNet .
Moreover, we used their own unsupervised training loss, which approximates the objectives of NCut minimization and modularity maximization, respectively.
For all the embedding-based methods (including the transductive embedding baselines, inductive
GNNs, and IGP), we used K Means as the downstream clustering module.
We ran K Means 10 times to avoid its limitation of getting locally optimal solutions, which is included in the total runtime.
To ensure the fairness of comparison, the embedding dimensionality of all the embedding-based methods was set to be the same on each dataset.
For other E2E baselines (e.g., Metis , GAP , and ClusNet ), we directly derived the partitioning results from their outputs.
Since Taxi , Reddit , Enron , and AS don’t provide the partitioning ground-truth (in terms of the partitioning membership C t ) on each historical training snapshot of these datasets, we used the partitioning result of the baseline with the best average modularity (on the validation set Γ V ) to derive the membership indicator H t for IGP (i.e., the second source of historical label information deﬁned in Section 2).
Note that we don’t need to derive H t in the evaluation phase when generalizing the trained model to test set Γ (cid:48) .
We tuned the parameters of all the methods on Γ V of each dataset to report the best average quality metrics.
Both the mean and standard deviation of all the quality metrics and runtime on Γ (cid:48) of each dataset were recorded.
Due to the space limit, we report the results in mean and standard deviation as well as the parameter settings in Appendix C.5 and C.4.
The evaluation results in terms of average NMI, AC, modularity, NCut, and runtime are illustrated in Fig.
In most cases of Fig. 3, IGP-M (red circle) and IGP-C (blue circle) are in the top group with the best quality, even though we didn’t train them on the test set Γ (cid:48) , where other transductive baselines were fully trained.
Surprisingly, IGP even performs the best or second-best in some cases.
Moreover, the runtime of IGP-M and IGP-C is also competitive to the fast GP baselines.
Although the runtime of inductive GNN baselines (with the same fast online generalization setting) is slightly faster than IGP, they still suffer from poor quality, which implies that the neighbor-induced features Z t and hybrid training loss of IGP are key components to ensure its high quality.
From the view of alleviating the NP-hard challenge of online
GP on newly generated system snapshots, the inductive framework of IGP achieves a better trade-off between quality and efﬁciency.
In Appendix C.5, we further quantitatively evaluate the signiﬁcant trade-off of IGP beyond other baselines by introducing a novel trade-off score and also give some other discussions as well as analysis of the evaluation results.
Ablation Study.
For both IGP-M and IGP-C, we also validated the effectiveness of (i) AL loss, (ii) FR loss, (iii) CR loss, (iv) reduced features Z t , and (v) GNN by respectively excluding the corresponding component from the original model.
In case (iv), we used a constant matrix 1 N t × L (with all entries set to 1 ) to replace Z t , while we use an FC network that only takes Z t as the input to replace the GNN encoder of G (with the same layer conﬁgurations).
Example results on L ( n, .
6) in terms of average NMI, AC, modularity, and NCut are shown in Table 2.
According to Table 2, the FR loss, features Z t , and GNN are key components to ensure the high quality of IGP, since there is signiﬁcant quality decline for case (ii), (iv), and (v).
The AL and CR losses further enhance the quality by incorporating the permutation invariant label information of historical snapshots.
Due to the space limit, we leave details of some other ablation studies to Appendix C.5.
Extended Applications.
Besides the aforementioned experiments, we consider two extended applications.
First, when the number of clusters K of a new graph is not given, IGP also has the potential to estimate the K value, where we apply an auxiliary model selection method (e.g., XMeans) to the derived embedding U t .
Second, in contrast to the hypothesis that all the given snapshots are
[Figure]
from the same system (following several underlying distributions), it’s promising that one conducts ofﬂine training on the historical data of one scenario but generalizes the model to other scenarios for fast online GP.
We consider the challenging application where we train IGP on synthetic graphs but generalize it to real datasets.
The two extended applications are elaborated in Appendix D.
C ONCLUSION
In this paper, we proposed a novel IGP framework to alleviate the NP-hard challenge of GP, obtaining a better trade-off between quality and efﬁciency.
In IGP, we ﬁrst conduct ofﬂine training of a novel dual GNN structure on historical system snapshots and generalize the model to newly generated graphs for fast high-quality online GP.
IGP is a generic framework that can incorporate the permutation invariant label information of GP to the ofﬂine training by combining unsupervised GP objectives (modularity maximization and NCut minimization) with the dual GNN structure.
Since we introduced an inductive graph embedding scheme and an efﬁcient feature extraction module, IGP can also tackle online GP on graphs with non-ﬁxed number of nodes and clusters.
Due to the space limit, we give discussions of possible future work (e.g., further reducing the runtime, scaling IGP up to graphs with large number of nodes, and transferring IGP across scenarios)
A P ROOF OF T HEOREM
By the disjoint constraint of GP, i.e., C tr ∩ C ts = ∅ for ∀ r (cid:54) =
s , each node can only belong to one unique cluster.
For the membership indicator matrix R t , only one entry in each row of R t must be 1 with the rest entries in the same row set to 0 .
For the auxiliary adjacency matrix A ( g ) t = R t R Tt , consider each entry ( A ( g ) t ) ij =
(cid:80)
K t r =
( R t ) ir ( R t ) jr ] with respect to the pair of nodes ( v ti , v tj ) .
If nodes ( v ti , v tj ) are in the same cluster C ts , i.e., v ti , v tj ∈ C ts , we have R is =
R js = 1 .
We can further derive ( A ( g ) t ) ij =
( R t ) is ( R t ) js = 1 .
It indicates that there is an edge between ( v ti , v tj ) with weight 1 , when they are in the same cluster.
If nodes ( v ti , v tj ) are not in the same cluster, i.e., v ti ∈ C tr and v tj ∈ C ts ( r (cid:54)
= s ), we have R ir =and R js = 1 , but ( A ( g ) t ) ij = 0 .
Namely, there is no edge between nodes ( v ti , v tj ) when they are in different clusters.
In summary, there is an edge between ( v ti , v tj ) in the auxiliary label-induced graph G ( g ) t only if they are partitioned into the same cluster according to the given partitioning result C t
= { C t 1 , · · · , C tK t } (used as the ‘ground-truth’).
Especially, each node v ti ∈ C tr
(i) must have edges connected to all the other nodes in the same cluster C tr and (ii) don’t have edges connected to nodes in different clusters
C ts ( r (cid:54) = s ), thus forming a fully connected component with respect to cluster C tr .
Hence, for a graph G t with K t clusters, its corresponding auxiliary label-induced graph G ( g ) t has K t fully connected components, with each component corresponding to one unique cluster of G t .
B D ETAILED A LGORITHMS OF IGP
In contrast to conventional transductive GP methods applied to a single static graph, IGP is an inductive framework that explores the structure properties across a set of graphs S = {G 1 , · · · , G T } .
In IGP, we ﬁrst train a novel dual GNN structure on historical system snapshots in an ofﬂine way (regardless of the training time cost).
We then generalize the trained model to newly generated snapshots of the same system for fast high-quality online GP without additional optimization, where a better trade-off between quality and efﬁciency can be achieved.
For simplicity, we divide S into a training set Γ T , a validation set Γ V , and a test set Γ (cid:48) , where Γ T ∪ Γ V and Γ (cid:48) represent the set of historical and newly generated system snapshots, respectively.
Namely, snapshots in Γ T ∪ Γ V should occur before those in Γ (cid:48) .
B. F EATURE E XTRACTION
For a set of graph snapshots S = {G 1 , . . .
G T } with non-ﬁxed node set {V 1 , . . .
, V T } (i.e., with non-ﬁxed number of nodes N t ), we introduce an efﬁcient feature extraction strategy via HEM graph coarsening, which maps the neighbor-induced feature X t ∈ (cid:60) N t × N t
( X t = Q t or X t = M t ) to a reduced feature matrix Z t ∈ (cid:60) N t × L with ﬁxed dimensionality L .
Algorithm 1 and Algorithmsummarize the procedures of HEM graph coarsening and the feature extraction module.
For each graph G t ∈ S , if L < N t , Algorithm 1 ﬁrst merges G t into a supergraph G ∗ t (with L supernodes)
based on reweighted edges E ( w ) t = { w ( v ti , v tj ) | w ( v ti , v tj ) =
( X t ) ij , ( A t ) ij = 1 } and outputs a coarsening matrix C t ∈ (cid:60) N t × L that describes the merging membership.
Concretely, ( C t ) ij = | v t ∗ j | − 0 .
5 if node v ti is merged into supernode v t ∗ j (i.e., v ti ∈ v t ∗ j ) and ( C t ) ij = 0 , otherwise.
Based on C t given by Algorithm 1, Algorithm 2 further derives a reduced feature matrix Z t via Z t = X t C t .
By the sparsity of C t , we can set its j -th column to ( Z t ) : ,j = (cid:80) v i ∈ v t ∗ j ( C t ) ij ( X t ) : ,i .
In contrast, when L > N t , Algorithm 2 directly derives Z t =
[ X t , 0 N t × ( L − N t ) ] via a simple padding strategy.
( k +1) t
V ( k +1) t ← V ( k +1) t ∪ V ( k ) t
E ( k +1) t ← E ( k +1) t ∪ E ( k ) t
k ← k
+ 1 //Update
the level index k 24 return C t based on the merging membership of V ( k ) t
Input: input graph G t ; neighbor-induced features X t ; number of nodes N t ; reduced dimensionality L Output: reduced features Z t 1 if L ≥ N t then let Z t =
[ X t , 0 N t × ( L − N t ) ] //Pad
Z t with zeros 3 else if L < N t then derive the coarsening matrix C t ∈ (cid:60) N t × L via Algorithm derive the merging membership V ∗ t based on C t for each supernode v ∗ j ∈ V ∗ t do ( Z t ) : ,j ← 0 N t × 1 //Initialize the j -th column of Z t for each node v i ∈ v ∗ j do ( Z t ) : ,j ← ( t ) : ,j + C t ) ij ( X ) : ,i
B. O FFLINE T RAINING ON H ISTORICAL S YSTEM S NAPSHOTS
In the ofﬂine training of IGP, we assume that the partitioning result C t = { C t 1 , · · · , C tK t } (i.e., ‘ground-truth’) of each historical graph G t ∈ Γ T ∪ Γ V is available, which is permutation invariant in terms of cluster labels.
For simplicity, we correlate each historical graph G t ∈ Γ T ∪ Γ V with a 2 -tuple ( A t , C t ) .
The ofﬂine training procedure of IGP is summarized in Algorithm 3, where L is ﬁxed dimensionality for the reduced features Z t ; { α, β } are hyper-parameters of the training loss L G ; p is the number of historical snapshots sampled in each epoch; { η D , η G } are learning rates of the optimizers updating { δ D , δ G } ; n is the maximum number of epochs.
For a selected training sample in each epoch, repeatedly updating model parameters m ( m ≥ 1 ) times can sometimes lead to better partitioning quality (on the validation and test set).
Algorithm 3 ﬁnally saves the model parameters { δ ∗ D , δ ∗ G } with the best quality metric on the validation set Γ V .
B. O NLINE G ENERALIZATION TO N EWLY G ENERATED S YSTEM S NAPSHOTS
In the online generalization of IGP, we ﬁx the learned model parameters { δ ∗ D , δ ∗ G } and directly derive graph embedding U t (cid:48) with respect to each newly generated graph G t (cid:48) ∈ Γ (cid:48) from feature encoder of G .
Since we assume that the number of clusters K t (cid:48) of each graph G t (cid:48) ∈ Γ (cid:48) is given in the online GP, we apply K Means to U t (cid:48) to derive the partitioning result C (cid:48) t (cid:48) with K t (cid:48) clusters.
In this case, the total runtime of online GP on each newly generated graph G t (cid:48)
∈ Γ (cid:48) includes (i) the efﬁcient feature extraction of Z t (cid:48) , (ii) one feedforward propagation of { A t (cid:48) , Z t (cid:48) } through feature encoder , and (iii) the downstream ( K Means) clustering.
In conclusion, we summarize the online generalization procedure of IGP in Algorithm 4.
B. C OMPLEXITY A NALYSIS
For a given graph G t , we use the efﬁcient matrix multiplication to derive the neighbor-induced features Q t and M t .
Concretely, we set Q t =
A t − d Tt d t (cid:14) (2 w ) and M t = D − 1/
A t D − 1/2 , where d t =
[ d t 1 , · · · , d tN t ] and D t = diag( d t 1 , · · · , d tN t ) are the degree vector and degree diagonal matrix of G t .
Although the complexities of computing Q t and M t are both at most O ( N 2 t ) , they can be easily paralleled via GPUs, which signiﬁcantly reduces the runtime.
Moreover, the complexity of the HEM graph coarsening (i.e., Algorithm 1) is no more than O ( N t |E t | log |E t | ) , with |E t | denoting
[Table]
the number of edges in G t and |E t | (cid:28) N 2 t .
The complexity to derive Z t via X t and C t (i.e.,
Algorithm 2) is no more than O ( N t Ln ∗ ) , where n ∗ is the maximum number of nodes merged into a supernode; L is the number of supernodes (which is also the reduced feature dimensionality of
Z t ).
Hence, the complexity to derive the reduced feature matrix Z t based on X t is no more than
O ( N t |E t | log |E t | + N t Ln ∗ ) .
Let L ( l − 1) be the feature dimensionality of the l -th GNN layer of the feature encoder or auxiliary label-induced encoder in G , where we have L (0) =
L .
We used the efﬁcient sparse-dense matrix multiplication to implement the graph convolutional operation described in equation (5), where the sparsity of A t is considered.
The complexity of one feedforward propagation through the feature encoder is no more than O ( |E t | L (0) L (1) + |E t | L (1) L (2) + · · · ) =
O ( |E t | LL (cid:48) ) , where L (cid:48)
=
L (1) and L ( l ) <
L ( l − 1) .
Hence, the overall time complexity of one feedforward propagation through the feature encoder is linear in the number of edges of the input graph G t .
Let k be the dimensionality of graph embedding.
Usually, we have k (cid:28) N t .
For a graph with K t clusters, the complexity of K Means is no more than O ( N t K t kI ) for I iterations.
In particular, the runtime of the downstream clustering (i.e., K Means algorithm) can also be signiﬁcantly reduced via parallel implementations.
C E XPERIMENT D ETAILS C. E VALUATION M ETRICS
In our experiments, we used the (unsupervised) metrics of modularity and NCut to evaluate the partitioning quality of each method on all the datasets.
The formal deﬁnitions of modularity and NCut are given in equation (3) and equation (1) (see Section 2).
For datasets with partitioning ground-truth (i.e., GN-Net and LFR-Net ), we also used the metrics of normalized mutual information (NMI) and accuracy (AC)
(Cui et al., 2018), which can measure the correspondence between the partitioning result (given by the method to be evaluated) and groundtruth (given by the dataset).
In general, better correspondence indicates better partitioning quality.
For a graph G t with K t clusters, we use H t
= { H t 1 , · · · , H tK t } and C t = { C t 1 , · · · , C t K t } to denote the ground-truth and partitioning result, where H tr ⊆ V t and C tr ⊆ V t represent node member sets of the r -th cluster.
Given H t and C t , the deﬁnition of NMI is as follow:
[Equation]
13) where n r = | H tr | is the number of nodes in the r -th cluster of ground-truth; n s = | C ts | is the number of nodes in the s -th cluster of partitioning result; n r,s = | H tr ∩ C ts | is the number of nodes that are simultaneously partitioned into the r -th cluster of ground-truth and the s -th cluster of partitioning result.
Let L t =
( l t 1 , · · · , l tN t ) and R t =
( r t 1 , · · · , r tN t ) be label sequences of the partitioning result and ground-truth with respect to node sequence ( v t 1 , · · · , v t N t ) , where l t i and r t i are cluster labels of node v ti .
The deﬁnition of AC is as follow:
[Equation]
[Equation]
where map( · ) is the Kuhn-Munkres mapping function that gives the best membership map from L t to R t ; δ ( a, b ) is the Kronecker delta function with δ ( a, b ) =
when a = b and δ ( a, b ) = 0 otherwise.
[Table]
Usually, larger modularity, NMI, and AC, as well as smaller NCut indicate better partitioning quality of a given GP method.
C. D ATASET D ETAILS
The detailed statistics of the 11 datasets (with 7 synthetic benchmarks and 4 real datasets) used in our experiments are depicted in Table [Table] where T , N , |E| , and K are the number of snapshots, nodes, edges, and clusters, respectively.
GN-Net (Girvan & Newman, 2002) is a widely-used synthetic benchmark for GP, in which the graph topology and partitioning ground-truth of each snapshot can be generated via the stochastic block model (Abbe, 2017).
We ﬁxed N = 5 , 000 and evenly partitioned the nodes into K =clusters.
For two uniformly selected nodes v i and v j , if they’re assigned in the same cluster (i.e., v i , v j ∈ C r ), the probability to generate an edge ( v i , v j ) is p in .
Otherwise (i.e., v i ∈ C r and v j ∈ C s ( r (cid:54)
= s )), the probability to generate an edge ( v i , v j ) is (1 − p in )/
( K − 1) .
In particular, we set p in ∈ { 0 . 5 , 0 . 4 , 0 . 3 } to generate 3 datasets.
For each dataset, we independently generated 2 , 000 snapshots.
In each snapshot, we also randomly shufﬂed the node indices to ensure that there’s no index correspondence between any two snapshots in the dataset (as we deﬁned in Section 2).
Especially, with the decrease of p in , the clustering structures become increasingly difﬁcult to identify .
For simplicity, we denote the GN-Net with a speciﬁc setting of p in as GN- p in (e.g., GN- 0.5, GN- 0.4, and GN- 0.3).
Note that all the datasets of GN-Net are with ﬁxed N and K .
LFR-Net (Lancichinetti et al., 2008) is a more challenging synthetic benchmark that can simulate power-law properties of node degree and cluster size of real-world network systems.
It uses ( k, k max , c min , c max , µ ) to independently generate a single graph, where k and k max are the average and maximum node degree; c min and c max are the minimum and maximum cluster size; µ is the mixing ratio between the external degree and total degree of each node v i with respect to the cluster v i belongs to.
In particular, we ﬁxed ( k, k max , c min , c max ) =
and set µ ∈ { 0 . 3 , 0 . 6 } .
With the increase of µ , the clustering structure is increasingly difﬁcult to identify .
We consider both cases with ﬁxed and non-ﬁxed N , where we respectively ﬁxed N = 5 , 000 and set N to a random integer within [5000 , 6000] to generate 2 , 000 graphs for each case.
Similar to GN-Net , we also independently generated each snapshot in a dataset and ensure that there’s no index correspondence between any two snapshots via the random shufﬂe of node indices.
For simplicity, we use L ( c, µ ) ( c ∈ { f, n } ) to represent the dataset with a speciﬁc setting of µ , where f and n denote the case with fixed and non-ﬁxed N (e.g., L ( f , 0.3), L ( f , 0.6), L ( n , 0.3), and L ( n , 0.6)).
Moreover, K is non-ﬁxed in LFR-Net .
Taxi 1 (Piorkowski et al., 2009) is a real GPS dataset including trajectories of 1 , 279 taxis in Beijing, which forms a mobile ad hoc network system.
We sampled 3 , 000 snapshots from the dataset in chronological order.
In each snapshot, we treated each taxi as a node and constructed the topology based on the top- 50 neighbors with the closest distance for each node in a snapshot.
Since Taxi doesn’t provide ground-truth regarding the number of clusters K for GP, we used Louvain algorithm (Blondel et al., 2008) to determine the K value for each snapshot in this dataset.
Hence, it’s with ﬁxed N and non-ﬁxed K .
Reddit 2 (Yanardag & Vishwanathan, 2015) is a social network dataset extracted from Reddit.
Each graph snapshot in the dataset corresponds to an online discussion thread.
We extracted 1 , 870 graphs with N ≥ 500 from the dataset.
Different snapshots are with different number of users (i.e., nodes), varying from 501 to 3 , 648 .
The comment interactions between users in each discussion thread are described as topology of the corresponding graph.
Since Reddit doesn’t provide ground-truth regarding the K value, we also used Louvain to determine K for each graph.
Hence, Reddit is with non-ﬁxed N and K .
Note that different snapshots have different node sets (i.e., with non-ﬁxed N ) and node indices in each snapshot start from 0. It indicates that there’s no correspondence between node indices of any two snapshots in the dataset.
Enron 3 (Rossi & Ahmed, 2015) is a real interaction network extracted from the email system of
Enron company, which contains email records from 1980-01-01 to 2004-02-04.
We extractedsnapshots with N ≥ 500 from the records in chronological order, in which we treated each user as a unique node and constructed the topology based on email interactions.
Note that different snapshots have different sets of users that have interaction behaviors.
Moreover, we also used Louvain algorithm to estimate the value of K for each snapshot, since Enron doesn’t provide ground-truth regarding K .
Therefore, it’s with non-ﬁxed N and K .
We also ensure that there’s no node index correspondence between any two graphs by permuting node indices of each graph from 0.
AS 4 (Leskovec et al., 2005) is a real dataset extracted from the border gateway protocol (BGP) logs of a communication network, which describe the topology among a set of autonomous systems (AS).
It contains 733 daily instances spanning from 1997-08-11 to 2000-01-02.
Each BGP router is abstracted as a unique node with communication between routers extracted as the graph topology.
As the dataset allows addition and deletion of routers, different snapshots are with different set of nodes.
Since AS doesn’t provide ground-truth regarding the K value, we also applied Louvain to determine K for each snapshot.
Therefore, it’s with non-ﬁxed N and K , where there’s no index correspondence between any two snapshots.
In summary, the 3 datasets of GN-Net are with ﬁxed N and K .
The 4 datasets of LFR-Net cover both the cases of (i) ﬁxed N but non-ﬁxed K as well as (ii) non-ﬁxed N and K .
Taxi is a real dataset with ﬁxed N but non-ﬁxed K , while Reddit , Enron , and AS are with non-ﬁxed N and K .
Note that both GN-Net and LFR-Net are synthetic benchmarks that can simultaneously generate graph topology and partitioning ground-truth of each snapshot.
Taxi , Reddit , Enron , and AS are real datasets without partitioning ground-truth (regarding K ), so we applied Louvain algorithm to these datasets to determine the K value of each single snapshot, which ensures the fairness of comparison, i.e., each snapshot is with a common K for all the GP methods to be evaluated.
C. B ASELINE M ETHODS
In addition to the 15 baseline methods introduced in Section 4, we also evaluated partitioning quality and runtime of other 5 strong baselines, which include degree-corrected stochastic block model ( DCSBM ) (Karrer & Newman, 2011), DeepWalk (Perozzi et al., 2014), node2vec (Grover &
Leskovec, 2016), LINE (Tang et al., 2015), and GIN (Xu et al., 2018).
In summary, we have in total evaluated the quality and efﬁciency of 20 baselines with 4 categories.
• First, SNMF , DCSBM , and SC are classic transductive GP methods based on several GP objectives (e.g., SC for NCut minimization and DCSBM for modularity maximization), which aim to achieve high-quality partitioning results (but may have high complexity).
• Second, DeepWalk , node2vec , LINE , VERSE , MILE , PhUSION , NPR , RandNE , and ProNE are unsupervised transductive graph embedding baselines.
In particular, LINE , NPR , RandNE , and ProNE are efﬁcient embedding methods with several fast approximation strategies for the embedding learning (but may have low partitioning quality).
In contrast, DeepWalk , node2vec , VERSE , MILE , and PhUSION are baselines that aim to derive high-quality representations (but may have high complexity).
• Third, Metis , hMetis , and GraClus are classic fast transductive GP baselines that aim to achieve low runtime with the heuristic approximation of classic GP objectives (which may suffer from quality degradation due to the information loss of heuristic approximation).
• Fourth, GraSAGE , GAT , and GIN , GAP , and ClusNet are inductive GNN baselines.
In particular, GraSAGE , GAT , and GIN are SOTA inductive GNN following the graph embedding scheme, i
e., learning node representations to support downstream tasks including GP.
Note that we don’t consider supervised training loss of these inductive GNNs (e.g., using the cross-entropy loss for a softmax output layer combined with the GNN), because GP is a typical unsupervised task and the cluster labels are permutation invariant.
In contrast, GAP and ClusNet are E2E GP baselines based on inductive nature of GNN, where an FC output layer is integrated into the GNN to directly output the partitioning results
Note that the ﬁrst three types of methods are inherently transductive , which only focus on GP on a single static graph but cannot be generalized to other new graphs.
Hence, we had to apply these transductive baselines to each newly generated system snapshot (i.e., each snapshot in test set Γ (cid:48) ) from scratch.
For the fourth type of baselines (i.e., inductive GNN approaches), we used the same settings with IGP (i.e., ofﬂine training on training set Γ T and online generalization to test set Γ (cid:48) ).
In particular, we can only use the inductive nature of E2E GNN methods (i.e., GAP and ClusNet ) on datasets with ﬁxed number of clusters K (i.e., on GN-Net ).
However, we have to use the same settings with transductive baselines (i.e., training GAP and ClusNet from scratch for each new test graph) on rest datasets with non- ﬁxed K , due to the ﬁxed dimensionality
of their output layers.
Some other experiment settings regarding the baseline methods have been already introduced in Section 4.
C. I MPLEMENTATION D ETAILS
We used PyTorch to implement the dual GNN structure of IGP.
The feature extraction module (i.e., Algorithm 1 and Algorithm 2) was implemented via NumPy and PyTorch, while we used the implementation of K Means of scikit-learn as the downstream clustering module.
All the experiments were conducted on a server with Intel Xeon CPU (E5-2650v4@2.20GHz, 48 cores), 1 Tesla V GPU, 61GB memory, as well as the Ubuntu Linux operating system.
In this setting, modules implemented by PyTorch were speeded up via GPU.
For all the embedding based methods, including transductive embedding baselines (i.e., second type of the baseline), inductive GNNs (i.e., GraSAGE , GAT , and GIN ), and IGP, we used K Means as the downstream clustering module.
In contrast, we can directly obtain the partitioning results from outputs of the rest baselines.
Note that K Means for graph embedding based methods and optimization algorithms of several classic GP baselines (i.e., SNMF , DCSBM , Metis , hMetis , and
GraClus ) have the limitation of obtaining locally optimal solutions.
We adopted the strategy to run K Means of embedding based methods and optimization algorithms of classic baselines 10 times, which was included in the total runtime of corresponding methods.
The partitioning membership with minimum loss function value (e.g., the sum of squared distance between data and cluster center for K Means) was reported as the ﬁnal partitioning result for evaluation.
When testing the runtime of each method, we ensured that only the method to be evaluated was running on the experiment server, i.e., there were no multiple methods simultaneously running on the test server.
On the validation set Γ V of each dataset, we tuned parameters and determine proper layer conﬁgurations for the two variants of IGP.
Concretely, we adjusted α ∈ { 1 , 10 } and m ∈ { 1 , 2 , · · · 5 } for both IGP-M and IGP-C , while we tuned β ∈ { 0 . 1 , 1 , 2 , · · · 5 } and β ∈ { 0 . 1 , 1 , 100 , 1000 } for IGP-M and IGP-C, respectively.
The recommended parameter settings and layer conﬁgurations of IGP-M and IGP-C on each dataset are depicted in Table 4, where dimensionality of the ﬁrst and last layer of G are set to be L (i.e., dimensionality of reduced feature) and k (i.e., dimensionality of graph embedding), respectively.
Note that we use the same layer conﬁgurations for IGP-M and IGP-C on each dataset.
Moreover, we set the maximum number of epochs n = 100 for all the datasets.
C. D ETAILED E VALUATION R ESULTS AND D ISCUSSIONS C.5. Q UANTITATIVE E VALUATION
In the experiments, we recorded the mean value µ and standard deviation σ of all the evaluation metrics and runtime for all the methods (on test set Γ (cid:48) of each dataset).
In particular, we use format
[Table] [Table]
‘ µ ( σ ) ’ to represent each evaluation result. The quantitative evaluation results on GN-Net and LFRNet in terms of NMI and AC are depicted in Table 5 and Table 6.
The evaluation results on all the datasets in terms of modularity and NCut are illustrated in Table 7, Table 8, Table 9, and Table 10.
Moreover, Table 11 and Table 12 give the runtime of all the methods.
For both IGP-M and IGP-C, we also recorded the time of (i) feature extraction (i.e., Algorithm 2), (ii) one feedforward propagation (through the feature encoder of G ), and (iii) downstream clustering (i.e., 10 independent runs of K Means), which together contribute to the total runtime of IGP.
Runtime of the three modules is denoted as ’Feat’, ’Prop’, and ’Clus’, respectively.
According to evaluation results in Table 5, Table 6, Table 7, Table 8, Table 9, Table 10 Table 11, and Table 12, which are consistent with the visualization results in shown Fig.
3 (see Section 4), we have the following major observations
• For all the datasets, IGP-M and IGP-C are in the top group with best quality metrics.
Moreover, their runtime is also competitive to fast GP baselines (e.g., Metis , hMetis , GraClus , RandNE , etc.) and other inductive GNNs with fast online generalization (e.g., GraSAGE , GAT , and GIN ).
From the perspective of alleviating the NP-hard challenge of GP, IGP can achieve a signiﬁcant trade-off between quality and efﬁciency of online GP on newly generated system snapshots.
We further quantitatively evaluate the signiﬁcant trade-off obtained by IGP via a novel trade-off score in Appendix C.5.2.
• Surprisingly, in some cases, IGP even has the best or second-best quality metrics.
Note that IGP is a novel inductive framework, in which we don’t conduct additional optimization on the test set of each dataset, but other conventional transductive baselines (e.g., SP , DeepWalk , etc.) can fully explore the structural properties of each test graph.
It indicates the potential of IGP to capture additional characteristics shared by multiple historical snapshots beyond conventional transductive approaches applied to one single new graph.
• On GN-Net and LFR-Net , partitioning quality of all the methods declines with the decrease of p in and increase of µ , where both IGP-M and IGP-C are always in the top group with
[Table]
[Table] [Table]
best quality metrics. It demonstrates the strong robustness of IGP
[Table]
especially when the intrinsic clustering structure of the given graph isn’t distinct. • Similar to IGP, the E2E GNN baselines (i.e., GAP and ClusNet ) can also apply fast online generalization to datasets with ﬁxed K (i.e., on GN-Net ).
Namely, we also trained GAP and ClusNet on training set of GN-Net and directly generalized them to corresponding test sets for online GP, in which we directly derived the partitioning results from their output layers.
However, the online GP of GAP and ClusNet is with poor quality, which indicates that the online generalization of the graph embedding scheme in IGP is more robust than that of the E2E approaches, especially when clustering structure of a dataset isn’t distinct.
Furthermore, we still need to train GAP and ClusNet from scratch for each new test graph on datasets with non-ﬁxed K due to the ﬁxed dimensionality of their output layers, which is time-consuming.
In contrast, the graph embedding scheme of IGP can still tackle fast online GP with non-ﬁxed K .
In contrast to the reduced structural feature Z t in IGP, we used a constant matrix 1 N t × L as feature inputs of inductive GNN baselines (i.e., GraSAGE , GAT , adn GIN ), which is a standard setting of most inductive GNNs for graphs without available node attributes (Xu et al., 2018).
The widely-used unsupervised training loss of GraSAGE (Hamilton et al., 2017) was also adopted in the ofﬂine training of these inductive GNNs.
Although the inductive GNN baselines have slightly faster runtime over IGP, they still suffer from poor partitioning quality on all the datasets.
The results further validate that the extracted feature Z t as well as hybrid training loss of IGP (which can incorporate the permutation [Table]
[Table]
[Table]
invariant label information of historical training snapshots) are essential to ensure its high partitioning quality. • For the total runtime of IGP on all the datasets, the downstream clustering (e
[Table]
g., K Means with 10 independent runs) is a major bottleneck.
However, the downstream clustering module is also the key component that enables IGP to deal with online GP with non-ﬁxed K .
In our future work, we intend to further reduce the runtime of IGP by replacing the downstream clustering module with a generic E2E module that can directly derive the partitioning result with respect to a speciﬁc (non-ﬁxed) K via an output layer.
• In most cases, IGP-M and IGP-C have similar quality metrics.
On datasets in which the clustering structures are more difﬁcult to identify (e.g., L ( f , 0.6) and L ( n , 0.6)), IGP-M has better partitioning quality than that of IGP-C. It implies that the modularity matrix Q t is more informative than the normalized adjacency matrix M t to be the neighbor similarity feature X t in IGP.
Moreover, the neighbor-induced feature X t is also the key component to ensure the strong robustness of IGP, especially when clustering structures of a given graph are indistinct.
C.5. T RADE -O FF A NALYSIS
In this paper, we focus on the trade-off between quality and efﬁciency of online GP on newly generated graphs from a system.
To the best of our knowledge, to evaluate the trade-off between two aspects that conﬂict with each other using a comprehensive metric remains an open issue.
We propose a trade-off score (TOS) to quantitatively evaluate the trade-off between quality and efﬁciency [Table] [Table] of a given GP method.
Concretely, the calculation of TOS is based on the normalized area covered by an induced rectangle of a data point in the quality-efﬁciency visualization result (like Fig. 3).
Fig.
4 gives two running examples regarding how to calculate TOS for a given GP method.
Our goal is to let larger TOS indicate a better trade-off between quality and efﬁciency. In our evaluation
[Table]
we use NMI, AC, modularity
[Table]
and NCut as the quality metrics, while we evaluate the efﬁciency of a speciﬁc method via its total runtime.
Let Q and E be the value of a quality metric and the runtime with respect to a GP method to be evaluated.
Given the tuple ( E, Q ) , each method used in our experiments can be visualized as a data point in a 2-D space as illustrated in Fig.
When we use NMI or AC as the quality metric Q , we further derive the normalized metric via
[Equation]
) since the value range of NMI or AC is Q ∈ [0 , 1] and larger NMI or AC indicate better partitioning quality.
Furthermore, when we use modularity as the quality metric, the value range is Q ∈ [ − 1 , 1] , in which larger modularity indicates better quality.
Hence, we derive the normalized metric as
[Equation] [Equation]
[Table] [Figure] In contrast, smaller NCut metric implies better partitioning quality with value range Q ∈ [0 , + ∞ ) .
Therefore, when using NCut as the quality metric Q , we normalize it via
[Equation]
(17) with Q m as the maximum NCut value through all the methods to be evaluated.
For efﬁciency metric E (i.e., runtime), smaller value of E indicates higher efﬁciency with value range E ∈ [0 , + ∞ ) , so we further normalize E via
[Equation]
where E m denotes the maximum runtime through all the methods to be evaluated.
Finally, we deﬁne
TOS with respect to the given quality and efﬁciency metrics ( Q, E ) as the (normalized) area covered by the induced rectangle illustrated in Fig.
Namely, we have
[Equation]
in which larger TOS implies a better trade-off between quality and efﬁciency.
The TOS values with respect to the quality metrics of NMI, AC, modularity, and NCut are depicted in Table 13, Table 14, Table 15, and Table 16, respectively.
In our experiments, we saved model parameters of IGP with best metrics in terms of NMI and modularity on validation sets of datasets with and without ground-truth, respectively .
Namely, we used NMI and modularity as validation metrics for datasets with and without ground-truth .
For TOS metrics with respect to NMI and modularity (see Table 13 and Table 15), both IGP-M and IGP-C have the best and second-best TOS values in most cases, indicating that IGP can achieve the best trade-off when using NMI and [Table]
[Table]
modularity as the (validation) quality metrics.
Furthermore, IGP-M and IGP-C can also achieve the top- 5 TOS metrics with respect to AC and NCut on all the datasets.
In summary, IGP can achieve a better trade-off between quality and efﬁciency of the online GP over various baseline methods.
C.5. C ONVERGENCE A NALYSIS
We also analyzed the convergence of the ofﬂine training procedure of IGP (i.e., Algorithm 3).
In each epoch of the ofﬂine training on a dataset, we recorded the average partitioning quality (in terms of NMI, AC, modularity, and NCut) with respect to the learned embedding { U t } (i) on training set Γ T and validation set Γ V (i.e., historical system snapshots).
Moreover, we also evaluated the average partitioning quality of the generalized embedding { U t } on test set Γ (cid:48) (i.e., new snapshots) in each epoch.
To further validate the effectiveness of auxiliary label-induced embedding { U ( g ) t } , the average partitioning quality of { U ( g ) t } on training set was also recorded in each epoch during the ofﬂine training.
Note that we don’t need to derive { U ( g ) t } for Γ V and Γ (cid:48) .
We use evaluation records on the challenging L ( n , 0.6) dataset, whose clustering structures are difﬁcult to identify, as an example.
The convergence curves (within the ﬁrst 50 epochs) of the average
NMI, AC, modularity, and NCut with respect to (i) { U ( g ) t } on training set Γ T , (ii) { U t } on train- [Table] [Table] ing set Γ T , (iii) { U t } on validation set Γ V , and (iv) { U t } on test set Γ (cid:48) for IGP-M and IGP-C are illustrated in Fig.
5 and Fig. 6, respectively.
One can have similar observations in Fig.
5 and Fig.
Concretely, the partitioning quality (in terms of NMI, AC, modularity, and NCut) of { U t } continuously improves in the ﬁrst several epochs on the training, validation, and test set.
More importantly, the average NMI and AC of { U ( g ) t } on training set Γ T are always 1 in all the epochs for both IGP-M and IGP-C.
It indicates that the partitioning results derived from the auxiliary label-induced embedding { U ( g ) t } have perfect mapping to the ground-truth.
For both IGP-M and IGP-C, the average modularity and NCut of { U ( g ) t } also keep at values that are much better than metrics of { U t } on the training, validation, and test set.
The convergence curves with respect to { U ( g ) t } further verify that the introduced label-induced embedding { U ( g ) t } can capture much more information regarding ‘ground-truth’ of the training set (i.e., historical system snapshots) than the learned embedding { U t } , which enables IGP to capture the permutation invariant label information during the ofﬂine training.
It also ensures the strong robustness of IGP beyond existing baselines.
In summary, the convergence curves in Fig.
5 and Fig.
6 validate the effectiveness of the ofﬂine training procedure and the dual GNN structure of IGP. [Figure] [Figure]
C.5. A BLATION S TUDY
In addition to example results shown in Table 2, we conducted additional ablation studies on GN -0.3, L ( f ,0.6), and L ( n ,0.6), in which the clustering structures are more difﬁcult to identify compared with other datasets.
In each case of the ablation study (see Section 4), the mean value µ and standard deviation σ of all the quality metrics (on test set Γ (cid:48) ) were recorded.
The detailed ablation study results on L ( f ,0.6), L ( n ,0.6), and GN -0.3 (in the format µ ( σ ) ) are illustrated in Table 17, Table 18, and Table 19, respectively.
[Table]
[Table]
Note that Table 19, Table 17, and Table 18 have consistent results.
The FR loss, feature input Z t , and GNN are key components to ensure the high quality of IGP, because there are signiﬁcant quality declines in cases without the three components.
Moreover, the AL and CR losses are the components to further enhance the partitioning quality, as they can incorporate the permutation invariant label information of historical snapshots to the ofﬂine training of IGP.
D E XTENDED A PPLICATIONS D. M ODEL S ELECTION
As deﬁned in Section 2, we assume that the number of clusters K is given in the downstream GP. For online GP in which K isn’t given, IGP also has the potential to estimate the K value for a given graph.
As a demonstration, we applied XMeans, a model selection method, to the derived embedding { U t } of both IGP-M and IGP-C to estimate K for each snapshot in the test sets of L ( f, . 3) , L ( f, . 6) , L ( n, . 3) and L ( n, . 6) .
In the 4 datasets, ground-truth regarding the K value is given by the dataset and K is non-ﬁxed for multiple test graphs.
With the increase of µ , the clustering structures of L ( f, µ ) and L ( n, µ ) are increasingly difﬁcult to identify.
In particular, we use the same parameter settings and layer conﬁgurations with that in Table 4.
The range of K of XMeans for each dataset was set according to the maximum and minimum number of clusters as illustrated in Table 3.
The model selection results and ground-truth with respect to the test sets of L ( f, µ ) and L ( n, µ ) are depicted in Fig.
7 and Fig 8, respectively.
The model selection results in Fig.
7 and Fig 8 are similar.
When µ = 0 .
3 (with distinct clustering structures ), the model selection results of both IGP-M and IGP-C
ﬁt well to ground-truth on most test snapshots of L ( f, 0 . 3) and L ( n, 0 . 3) .
Note that the potential of model selection of IGP is still largely inﬂuenced by the intrinsic clustering structures of a dataset.
The estimated K values are still far from ground-truth when µ = 0 .
6 (with indistinct clustering structures ) on L ( f, 0 . 6) and L ( n, 0 . 6) .
In summary, IGP has the potential to estimate the number of clusters K of newly generated graph snapshots with distinct clustering structures.
D. T RANSFERRING T EST
To test the potential of IGP to transfer from one (training) scenario to other different (generalization) scenarios, we consider a challenging application in which we ﬁrst conduct ofﬂine training of IGP on completely synthetic graphs and directly generalize the trained model to the real datasets for fast online GP (without additional optimization).
In particular, we expect the synthetic graphs used for ofﬂine training cover various structural properties of real-world network systems.
We adopt LFR-Net (see Appendix C.2) as the model to generate synthetic training graphs.
In contrast to the parameter settings of LFR-Net in our quantitative evaluation (see Appendix C.2), we randomly set N ∈ [500 , 6000] , k ∈ [10 , 100] , k max ∈ [50 , 200] ,
[Table]
12.99(0.50) 33.66(1.20) 4.53(0.19) 11.80(0.32) 6.11e3(646.8439) 996.6852(439.4099)
Test Set    # C l u s t e r s
L ( f ,. [Figure]
[Figure] L ( f ,.3):IGP [Figure] M
L ( f ,.3) [Figure] IGP [Figure]
C
L ( f ,.6): [Figure] L ( f ,.6) [Figure]
IGP [Figure] L ( f [Figure]
[Figure] ):IGP [Figure] c min ∈ [5 , 100] , c max ∈ [50 , 400] , and µ ∈ [0 . 1 , 0 . 7] to independently generate 2 , 000 synthetic snapshots, where we ensure that k max > k and c max > c min for each snapshot.
In our experiments, we conducted ofﬂine training on the aforementioned synthetic snapshots and generalized the trained model to test sets of the four real datasets used in our quantitative evaluation (i.e., Taxi , Reddit , AS , and Enron as introduced in Appendix C.2).
We used the same layer conﬁgurations as in Table 4 with respect to each real dataset and tried different parameter settings of IGP-M and IGP-C to report their best average quality.
The recommended parameter settings of IGP-M and
IGP-C are shown in Table 20.
Evaluation results on test sets of the four real datasets (in terms of modularity and NCut) are depicted in Table 21, where ’Real’ and ’Syn.’
represent the results with respect to (i) training on training set of the original real datasets (same as the results in Table 8 and
Table 10) and (ii) training on the aforementioned synthetic snapshots, respectively.
According to Table 21, we have the following key observations
• In some cases (e.g., on Taxi and Reddit ), the quality metrics of training on synthetic graphs are close to those of training on the original real datasets for both IGP-M and IGP-C, which are also competitive to the strong baselines as illustrated in Table 8 and Table 10.
It preliminarily validates the potential of IGP to be transferred from completely synthetic graphs to real datasets.
However, in other cases, there’re still signiﬁcant quality declines for both the variants of IGP, which indicate that the transferring capability of IGP is largely affected by the domain similarity between the synthetic training set and real test set.
The negative transferring can be alleviated by carefully generating or selecting synthetic snapshots with the assistant of validation information from the target real datasets.
We leave the study of this application in our future work.
• Compared with IGP-C, IGP-M has better transferring quality metrics in most cases.
The quality declines of IGP-C are also more signiﬁcant than those of IGP-M. It further validates that the modularity matrix Q t used in IGP-M is more informative than the normalized adjacency matrix M t used in IGP-C.
Note that IGP has similar motivations with SOTA pre-training methods for graphs, e.g., PretrainGNN (Hu et al., 2019) and GCC (Qiu et al., 2020), in which we pre-train a GNN-based model on historical graphs (in an ofﬂine way) and generalize it to new snapshots.
The major difference between IGP and SOTA pre-training methods is that a ﬁne-tuning procedure with another optimization algorithm (based on a certain ﬁne-tuning loss) on new test graphs is required for most pretraining methods after the ofﬂine (pre-)training. In particular, the ﬁne-tuning procedure may still be [Figure]
[Figure] time-consuming for new test graphs, where several iterative optimization algorithms (e [Figure] g., gradient descent) are applied
[Figure] In this study [Figure] we focus on the NP-hard challenge of GP and try to achieve a better trade-off between quality and efﬁciency.
To some extent, the proposed IGP framework sacriﬁces some opportunities to ﬁne-tune the (pre-)trained model, which ensures the low runtime on new test graphs.
In our future research, we intend to consider a promising application that we pre-train a high-quality GP model on completely synthetic graphs and fast ﬁne-tune the (pre-)trained model on new graphs (via some efﬁcient ﬁne-tuning strategies), with the guarantee of high partitioning quality and low runtime.
E P OSSIBLE F UTURE W ORK
We summarize several promising future research directions as follows
According to the results in Table 11 and Table 12, the downstream clustering module (i.e., K Means with 10 independent runs) consumes most of the runtime of IGP.
Although replacing the clustering module with an FC output layer (trained in an E2E way) to directly output the partitioning results can further reduce the total runtime, the downstream clustering module is a key component enables IGP to tackle online GP with non-ﬁxed K .
In contrast, existing E2E GP methods still need to be trained from scratch for new graphs with non-ﬁxed K .
In our future research, we intend to further reduce the runtime of IGP by exploring a generic E2E scheme that can directly derive the GP results (e.g., via an output layer) and tackle the online GP with non-ﬁxed K .
One possible solution is to simultaneously train multiple output layers (with different output dimensionality) to derive multiple partitioning results with respect to all the possible values of K .
Another possible strategy is to output partitioning results in a hierarchical structure (that covers the results with respect to all the possible values of K ) based on the idea of hierarchical clustering.
• As discussed in Appendix D.2, our IGP framework has similar motivations with SOTA pre-training methods for graphs (Hu et al., 2019; Qiu et al., 2020). Namely, we conduct the ofﬂine (pre-)training of inductive GNNs (regardless of the time cost) but sacriﬁce some opportunities to ﬁne-tune IGP on new test graphs for a better trade-off between quality and efﬁciency of the online GP.
The ﬁne-tuning on new graphs (from other scenarios) can avoid the negative transferring by utilizing validation information from new targeted graphs.
Moreover, to select the proper set of pre-training datasets is still an open issue in recent researches regarding the pre-training and ﬁne-tuning scheme.
In our future work, we intend to conduct the ofﬂine (pre-)training of IGP on completely synthetic graphs (that covers properties of various real network systems) and explore an efﬁcient strategy to ﬁne- [Table] tune the model on real new system snapshots (of other scenarios) with the guarantee of high partitioning quality and low runtime.
• Scaling GNNs up to graphs with large number of nodes is a signiﬁcant direction in recent researches, where several sampling strategies are used to split a large graph into multiple subgraphs (Chen et al., 2018; Huang et al., 2018; Zeng et al., 2019), i.e., mini-batches with small number of nodes.
Note that IGP is an inductive framework across graphs, which can also be trained on and generalized
to multiple subgraphs sampled from a large graph (in terms of large number of nodes).
Most of existing GNNs that can be scaled up to large graphs usually focus on (semi-)supervised node-level tasks (e.g., node classiﬁcation) with available node attributes, while we consider challenging inductive unsupervised node-level task (i.e., GP) across graphs without graph attributes in this study.
In our future work, we intend to further explore efﬁcient (i) feature extraction and (ii) subgraph sampling strategies for IGP on large graphs without available attributes.
In particular, we can ﬁrst split a large graph into multiple small subgraphs via a sampling module and extract the neigh-induced features { X t } for each sampled graph.
IGP can then be applied to multiple sampled graphs and derive graph embedding { U t } with respect to each subgraph.
One possible challenge in this application is to ensure the consistency of embeddings { U t } of multiple subgraphs sampled from a common large graph (e.g., embeddings of all the subgraphs are mapped into a common latent space) without any auxiliary information given by the node attributes.