A Variational Edge Partition Model for Supervised Graph Representation Learning
Anonymous Author(s)
Abstract
Graph neural networks (GNNs), which propagate the node features through theedges and learn how to transform the aggregated features under label supervision,have achieved great success in supervised feature extraction for both node-level andgraph-level classification tasks.
However, GNNs typically treat the graph structureas given and ignore how the edges are formed.
This paper introduces a graphgenerative process to model how the observed edges are generated by aggregatingthe node interactions over a set of overlapping node communities, each of whichcontributes to the edges via a logical OR mechanism.
Based on this generativemodel, we partition each edge into the summation of multiple community-specificweighted edges and use them to define community-specific GNNs.
A variationalinference framework is proposed to jointly learn a GNN-based inference networkthat partitions the edges into different communities, these community-specific
GNNs, and a GNN-based predictor that combines community-specific GNNs forthe end classification task.
Extensive evaluations on real-world graph datasetshave verified the effectiveness of the proposed method in learning discriminativerepresentations for both node-level and graph-level classification tasks.
Introduction
Many real-world entities are bonded by relations, e.g. , the users in a social network service areconnected by online friendships and the atoms in molecules are held together by chemical bonds.
Representing the entities by nodes and relations by edges, a set of interconnected entities naturallyforms a graph.
Reasoning about these entities and their relations could be conducted under graphneural networks (GNNs)
Instead of isolating the feature transformation for each individualentity, GNNs allow the information to be exchanged between related entities.
Models employing thisstrategy have achieved great success in a wide range of applications involving graph structured data,such as classification [5, 6], link prediction [7], recommendation [8, 9], (overlapping) communitydetection [10, 11] and drug discovery [12].
The essence of many graph-analytic tasks could be summarized as supervised graph representationlearning, where the information on the graph is usually characterized by the node features, adjacencymatrix, and node or graph labels.
A supervised machine learning pipeline is often introduced toembed the nodes or graph under label supervision for a specific classification task.
For most of the
GNN-based methods, the information on nodes and edges are composited through neighborhoodaggregation, i.e. , updating the features of each node by combining the features of the surroundingnodes.
During this process, the graph structure mostly serves as a given source to indicate the nodes’neighborship [13] or provide the weights for aggregation [5].
Treating the graph structure as givenoverlooks the latent structures that control the formation of the graph, which, however, could provide valuable information for graph representation learning.
In other words, failing to consider the latentgraph structures may limit the ultimate potential of this line of work.
One type of latent structure at the hinge of the graph structure and node information is node communities.
Their connection to the graph can be explained under a latent-community based graph generationprocess.
For example, both the mixed-membership stochastic blockmodel (MMSB) [14] and edgepartition model (EPM) [15] explain the formation of edges by node interactions over overlappinglatent communities.
Let us consider person u in social network G , some of her social connectionsmay be established through interactions with colleagues as a machine learning researcher, somemay originate from her interactions with co-members of the same hobby club, and different types ofinteractions could overlap to strengthen certain social connections between the nodes.
The node community structure also sets the aspects of node information.
In the example of socialnetwork G where person u is affiliated with both a research group and a few hobby groups, due to thediverse nature of these different communities, it is likely that she exhibits different characteristicswhen interacting with co-members of different communities.
Namely, the information on u for theresearch group may be related to her research expertise, and the information on u for the hobby clubsmay be related to her hobbies.
Therefore, an ideal solution is to learn a different aspect of nodeproperties for each community that it is affiliated with, and represent the overall node information asan aggregation of all community-specific node properties.
Based on this insight, we develop a variational edge partition model (VEPM), which is a generativegraph representation learning framework.
VEPM models how the edges and labels are generatedfrom overlapping latent communities.
Instead of hard assigning a node to a single community, VEPMencodes each node into a K -dimensional vector of scores measuring the strength that the node isaffiliated with each of the K communities.
From these scores, we compute the intensities of pairwiseinteractions within each community.
Given that information, under the Bernoulli-Poisson link, theedges could be modeled by the logical OR of independently generated binary latent edges [15].
The generation of labels includes community-specific node representation learning and aggregation.
A key premise of the first part is to detect the hidden structure specified for each community.
In ourmodel, it is achieved through edge partition, i.e. , decomposing each edge into a summation of linkstrengths according to the intensities of community-specific interactions.
The edge partition stepisolates the link strengths accumulated through each community.
With the partitioned edges, we learn
K separate GNNs to produce community-specific node representations, then compose them into theoverall node embeddings to generate the labels.
Evaluation shows that the proposed framework canachieve significant performance enhancement on various node and graph classification benchmarks.
We summarize our main contributions as follows:
• We introduce VEPM that utilizes the idea of edge partitioning to extract overlapping
latent community structures, which are used to not only enrich node attributes with nodecommunity affiliation scores, but also define community-specific node feature
aggregations.• We formalize the training of VEPM in a variational inference framework, which is poweredby GNNs in its latent community inference, representation generation, and label prediction.
We further develop an effective pretrain-finetune algorithm for training VEPM.• We analyze how VEPM works and evaluate it over various real-world network datasets.
Empirical results show that the proposed VEPM outperforms many previous methods invarious node- and graph-level classification tasks, especially with limited labels.
Preliminaries and Related Work
Embedding nodes and graphs with GNNs.
Given a node-attributed graph G with N nodes, itsinformation is generally expressed by a design matrix X ∈ R N × F , whose rows represent nodefeatures of dimension F , and an adjacency matrix A of shape N × N .
The nonzero values of Aindicate the weights on the corresponding edges.
GNNs [1–5, 13, 16] are multi-layer parametricmodels that maps ( X , A ) to the embedding space.
The update of the embedding of node u atthe t th GNN layer could be summarized as h ( t ) u
= AGG( f θ ( h ( t − 1) u ) , { f θ ( h ( t − 1) v ) } v ∈ N ( u ) , A u, : )and h (0) u
= x u .
Here AGG denotes the neighborhood aggregation function, N ( u ) denotes theneighbors of u , and f θ ( · ) is a transformation function parameterized with θ .
In VEPM, we adoptthe aggregation function introduced by Kipf and Welling [5], which can be expressed as h ( t ) u
=
[Equation]
) , where ˜ A is the normalized adjacency matrix of G augmented withself-loops.
When the task requires a graph-level representation, all node embeddings would besummarized into a single embedding vector.
This process is called graph pooling [17–19].
In thiswork, we focus on improving the overall performance from GNN’s perspective and adopt a simplegraph pooling method proposed by Xu et al.
In the sequel, we unify the expression of modelsconsisting of GNN layers as GNN θ (cid:0) X , A (cid:1) .
Community-regularized representation learning.
Communities are latent groups of nodes in agraph [20].
The simplest way to use community information to help a supervised learning task isto regularize it with a community-detection related task.
For instance, Hasanzadeh et al.
[21] and
Wang et al.
[22] jointly train a deep stochastic blockmodel and a classification model; a similarapproach could be found in Liu et al.
[23] where the graph reconstruction regularizer is replaced by amodularity metric.
Either way, the node embeddings are expected to reflect patterns of communitystructures and maintain informative to the task at the same time.
These methods generally treatnode-community affiliation as node embeddings and expect them to contain sufficient information forthe downstream task.
However, the affiliation strengths of nodes to a community may oversimplifythe information that such a community can provide for the downstream task.
Unlike these methods,
VEPM embeds node information on each community with a specific GNN, where the communitystructures are reflected by the partitioned adjacency matrices.
Aggregating node features with theseweighted adjacency matrices naturally incorporates community information into task-learning.
Multi-relational data analysis.
Leveraging heterogeneous relations has been extensively studied inthe literature of mining Knowledge Graphs [24–26].
The data of these models is usually organizedby a multigraph, which is related to our perspective of the latent node interactions that generate theobserved graph because the interactions in different communities are potentially heterogeneous, andthey are permitted to coexist between a pair of nodes.
This leads to a shared high-level idea to breakdown the original heterogeneous graph into homogeneous factors and analyze each factor with acustomized model.
However, unlike those multigraphs where the edge types are explicitly annotated,the overlapping heterogeneous node interactions are not observable from the data that VEPM aims tohandle, escalating the difficulty of our optimization problem.
To deal with the latent variables, weformalize VEPM into a generative model and train it via variational inference.
Graph factorization-based models.
The existence of multiple hidden structures in graphs isalso noticed by some previous work [27, 28].
The architecture of these models starts with graphfactorization, followed by a set of modules with each one processing the information from one ofthe graph factors.
In these methods, the only ground-truth information that trains graph factorizationis from label supervision, making them rely on excessive label annotations, which might be hardto suffice in practice and thus potentially hinder their application.
On account of the potentialheterogeneity of graph information on different communities, VEPM adopts a similar pipeline tothese methods, but the factorization of the original graph is driven by not only the supervised task butalso a graph generative model, which greatly reduces our dependency on the amount of labeled data.
Variational Edge Partition Model
Many analytical tasks on node-attributed graphs could boil down to (semi-) supervised classificationproblems, i.e. , given ( X , A ) and observed node or graph labels y o , predicting the unobservedlabels y u .
In VEPM, we formalize the classification task as modeling the predictive distribution asp ( y u | X , A , y o ) , and construct our variational supervised learning pipeline with GNNs.
3.1 Generative task-dependent graph representation learning with latent communities
To model p ( y u | X , A , y o ) , we introduce Φ ∈ R N × K + , a latent non-negative node-communityaffiliation matrix, whose entry at the n th row and k th column is interpreted as the strength that then th node is affiliated with the k th community.
Given Φ , we specify a generative network as
[Equation]
to describe how the observed edges A and labels y o are generated.
Due to the complexity brought bythe deep architecture of the generative network , analytically inferring the posterior of Φ is impractical.
We hence approximate the posterior p θ ( Φ | A , y o ) with a variational distribution, q ω ( Φ | A , X ) ,modeled by a separate GNN-based inference network .
The subscripts θ and ω denote the parameters
GNN GNN
GNN GNN
T X
A (1) A (
2) A (3) A (3) T H
Output (2) T H (1) T H
T X
T 
A community encoder edge partitioner community-GNN bank representation composer
The Inference Network
The Generative Network
T 
T K
T 
KL egen task
GNN in the generative network and inference network , respectively.
We illustrate the architecture of VEPMin Figure 1.
We approximate the posterior predictive distribution using the Monte Carlo method as
[Equation]
(2) where Φ ( s ) iid ∼ q ω ( Φ | A , X ) for s = 1 , . . .
, S .
In what follows, we outline the data generation process and corresponding modules in the generativenetwork , introduce the module in the inference network , and describe how to train both networks.
3.2 Edge generation and label prediction
To model the distribution of the edges given Φ , we adopt the generative process developed in
EPM [15], which explains the generation of the edges under the Bernoulli-Poisson link as A i,j =1 M i,j ≥ 1 , M i,j ∼ Poisson (cid:16)(cid:80)
Kk =1 γ k
Φ i,k Φ j,k (cid:17) .
Here γ k is a positive community activation levelindicator, which measures the member interaction frequency via community k , and in our practice istreated as a trainable parameter shared by both the inference network and generative network .
Weinterpret γ k Φ i,k Φ j,k as the interaction rate between nodes i and j via community k .
The EPM has an alternative representation that partitions each edge into the logical disjunction ( i.e. ,logical OR) of K latent binary edges, expressed as
[Equation]
where p ijk := 1 − e − γ k
Φ i,k Φ j,k .
Thus nodes i and j would be connected as long as their interactionforms up an edge in at least one community.
In other words, they are disconnected only if theirinteractions in all communities fail to generate any edges.
Under the EPM, we have the conditionalprobability as p θ (A i,j = 1 | Φ )
= 1 − e − (cid:80)
Kk =1 γ k Φ i,k Φ j,k .
To complete the model, we set the priordistribution of the node-community affiliation matrix as Φ ∼ (cid:81) Ni
=1 (cid:81)
Kk =
Gam(Φ i,k ; α, β ) .
Given A and Φ , we now describe the predictive process of the labels.
The node-community affiliationimpacts this process from two aspects: for each node, the corresponding row of Φ serves as sideinformation that could enrich node attributes; for each pair of connected nodes, it could be used toderive the node interactions rate via each community, i.e. , { γ k Φ i,k Φ j,k } k =1 ,K , which are furtherused to partition the edges to K communities, carried out by the edge partitioner :
Edge partitioner.
The edge partitioner takes edges A and node-community affiliation matrix Φ asinputs and returns K positive-weighted edges: { A (1) , A (2) , · · · , A ( K ) } , s . t .
(cid:80)
Kk =1 A ( k ) =
A .
The edge partition function is
[Equation]
where τ is a “temperature” that controls the sharpness of partition.
The values of τ are specified in Appendix C.
The effect of the edge partitioner could be consideredas a soft assignment of each edge into different communities.
Setting temperature τ to be lowcould drive the soft assignment towards a hard assignment, and consequently, when aggregatingnode features with { A (1) , A (2) , · · · , A ( K ) } , the edge partitioner would concentrate the informationexchange between any two connected nodes at one community.
In other words, the mutual influencebetween two nodes, measured by the edge weight, is relatively high at the community that contributesthe most of the interactions between the pair, while being relatively weak at the other communities.
In our implementation, we further generalize Equation (3) to a metacommunity-based edge partition,specifically, by replacing γ k
Φ i,k Φ j,k with Φ i,k diag( γ k ) Φ ′ j,k .
In the new expression, Φ i,k denotesthe k th segment in node i ’s community-affiliation encoding, γ k is a vector of the activation levels forcommunities in the k th metacommunity.
This generalization allows the total number of communitiesto be greater than K , enhancing the model implementation flexibility at a minor interpretability cost.
The edge partitioner provides a soft separation of the neighborhood aggregation routes for eachcommunity, which is passed to the community-GNN bank :
Community-GNN bank.
The module takes X ∗ :
= X ∥ Φ as input, where the operator ∥ denotes“concatenation,” and learns community-specific node embeddings with K separate GNNs, i.e., the outputs are g (1) θ ( X ∗ ) , g (2) θ ( X ∗ ) , · · · , g ( K ) θ ( X ∗ ) ,
where g ( k ) θ ( · ) :
=
GNN θ (cid:0) · , A ( k ) (cid:1) , k ∈ { 1 , . .
. , K } .
The intention of the edge partitioner and community-GNN bank is to capture the node informationspecific to each community.
For instance, in a social network, such kind of information could be thedifferent social roles of people when considered affiliated with different social groups, e.g. , one maybe a former computer science major student in an alumni group, a research scientist in a work group,and an amateur chess player in a hobby club group.
As shown in Section 4.3, the node embeddingsproduced by the community-GNN bank are separable by communities.
We finally introduce the representation composer , which embeds a combination of the informationprovided by each community into a global representation via a GNN:
Representation composer.
Let H ( k ) :
= g ( k ) θ ( X ∗ ) denote the node representations learned from thek th community, where k = 1 , . .
, K , and f ( · ) denote the representation composer, whose functionality is to project a composite of community-specific node representations to one representation matrix,i.e., H V =
f (cid:0) H (1) , H (2) , · · · , H ( K ) (cid:1) :
=
GNN θ (cid:0) ∥
Kk =1 H ( k ) , A (cid:1) .
For graph-level tasks, we further pool node embeddings into a single vector representation h G , as in
Xu et al.
Taking softmax on the feature dimension of H V or h G gives the predicted probabilitiesof labels, from which we are able classify the unlabeled objects.
Cascading the edge partitioner andcommunity-GNN bank with the representation composer yields the probability p θ ( y | X , A , Φ ) .
3.3 Variational latent community inference
We use a community encoder as the module in the inference network to approximate the posteriordistribution p θ ( Φ | A , y ) , and provide the generative network with the critical latent variable Φ :
Community encoder.
The community encoder models the variational posterior q ω ( Φ | A , X ) bya Weibull distribution with shape K and scale Λ , whose parameters are learned by a GNN as
K ∥ Λ = GNN ω (cid:0) X , A (cid:1) .
A Weibull random sample from q ω ( Φ | A , X ) could be created throughthe inverse CDF transformation of a uniform random variable, given as follows:
[Equation]
where ⊙ and ⊘ denote element-wise multiplication and division.
3.4 The overall training algorithm
We train VEPM by optimizing the evidence lower bound (ELBO), decomposed into three terms, as
[Equation]
where L task =
E q ω ( Φ | A , X ) log p θ ( y o |
A , Φ , X ) , L egen =
E q ω ( Φ | A , X ) log p θ ( A | Φ ) , and L KL =− D KL (cid:0) q ω ( Φ | A , X ) ∥ p ( Φ ) (cid:1) .
These three terms correspond to the classification task, edge generation, and KL-regularization, respectively.
Note that our specifications of Φ ’s prior and variationalposterior yield an analytical expression of L KL , as described in detail in Appendix B.
Since modeling the likelihood of the labels involves stacking multiple modules embodied by deepneural networks, approximating the posterior p θ ( Φ | A , y o ) is difficult from an optimization perspective.
To effectively train VEPM, we leverage the edge generation process and break downthe overall training pipeline into two phases: an unsupervised pretrain , followed by a supervisedfinetune .
Specifically, we first pretrain the inference network by optimizing L egen + L KL to linklatent communities with edge generation.
We then join the pretrained inference network up with thegenerative network , and train the entire architecture with the objective function given in Equation (5).
In this stage, the latent communities are finetuned by L task to align them to the task.
The efficacy ofpretraining the model with an unsupervised loss is also expressed in other work [29] that involvesoptimizing graph structure for GNNs with a graph autoencoder.
We summarize the overall trainingpipeline in Appendix A, and empirically study the effect of bi-phase training algorithm in Section 4.2.
Empirical Evaluation
4.1 Node & graph classification
Datasets & experimental settings.
For node classification, we consider three citation networks:
Cora, Citeseer, and Pubmed, which provide bag-of-words document representations as node featuresand (undirected) citations as edges, and use the train/test/validation split standardized by Kipf and
Welling [5]; For graph classification, we consider four bioinformatics datasets (MUTAG, PTC, NCI1,
PROTEINS) and four social network datasets (IMDB-BINARY, IMDB-MULTI, REDDIT-BINARYand REDDIT-MULTI).
The input node features are crafted in the same way as Xu et al.
Most ofthe baselines are compared following the 10-fold cross validation based evaluation protocol proposedby Xu et al.
Since this protocol is criticized for being prone to overestimating model performance[30], we also evaluate our model following Zhang and Chen [31], who conduct a more rigoroustrain-validation-test protocol.
To counter the memory complexity expansion by using K GNNs inthe community-GNN bank , we limit the size of hidden dimension in each community-GNN to 1 / K ofwhat is commonly used among the baselines ( e.g. , 64 for both tasks).
We elaborate other details ofthe benchmarks, model architecture, and hyperparameter determination in Appendix C.
[Table]
Node classification.
We use classification accuracyas the evaluation metric for node classification.
Table 1 reports the average performance of VEPM( ± standard error) against related baselines that arecategorized into three different groups.
The firstgroup consists of the GCNs [5] and its variant GCN64 that expands the hidden dimension from 16 to64.
VEPM outperforms the first group by a significant margin.
The explanation is twofold: (i) VEPMaugments the node attributes with community information carried by the inferred node-communityaffiliations, (ii) VEPM learns community-specificnode embeddings.
The second group includes SIGVAE [21] and WGCAE [22], both of which learn node embeddings from graph generative modelsjointly optimized with a supervised loss.
The performance gain that VEPM obtains could be attributedto our unique label predictive process, which not only appends extra community information to nodeattributes, but also injects structural patterns of communities into neighborhood aggregation.
Thethird group [13, 32, 27] is focused on learning node embeddings leveraging heterogeneous hiddenrelations.
When fitting the hidden relations via attention mechanism, they only use information fromthe node features through label supervision, whereas our approach also takes the observed graphstructure into account via a graph generative model.
The additional information from the graphutilized by VEPM could explain the enhancement achieved by VEPM over the third group.
Graph classification.
For the first protocol, we compare VEPM with classical graph classificationbaselines [33–35], generic-GNN-based models [13, 32, 6], and GNNs with relation-based or task- driven graph factorization [25, 26, 28].
Results in Table 2 show that VEPM achieves the best graphclassification performance on 5 out of 8 benchmarks, and the second-best performance on the other 3benchmarks, including NCI1, where VEPM outperforms all the other GNN-based models.
[Table]
For the second protocol, aside from a traditional method
[Table]
WL [33]
[Table]
and a generic-GNN-based method
[Table]
DGCNN [36], we compare VEPM with two GNNs [37, 31] that adapt capsule neural networks [38]to graph-structured data, which aim to learn different aspects of graph properties via dynamic routing[39].
The results in Table 3 show that VEPM outperforms these GNN-based methods on most of thebenchmarks.
This indicates that the aspects of graph information, as defined with communities andlearned via aggregating node features with partitioned graphs, are more pertinent to the end tasks
[Table] [Table]
[Figure]
(this
[Figure] [Figure]
Classification with reduced labels.
Thegain of incorporating the graph generationprocess into VEPM is that the observedgraph structure also provides informationfor hidden community detection and edgepartition, which is beneficial if the amountof labeled data is not sufficient for both decomposing the graph into hidden factorsand semi-supervised task-learning.
To illustrate this point, based on the evaluation protocol by Kipf and Welling [5] for node classification and Xu et al.
[6] for graph classification, we re-evaluate the performanceof VEPM under reduced training labels on
Cora (for node classification) and MUTAG(for graph classification), along with generic-GNN-based methods [5, 13, 6] and decomposition-basedmethods [27, 28].
The results are shown in Figure 2: the performance of all models is negativelyimpacted by the reduction in training labels, those suffering from the strongest performance declineare DisenGCN [27] and FactorGCN [28] that decompose graphs solely based on label supervision,while VEPM also performs graph factorization with limited labels, it consistently outperforms theother methods under all settings, and the superiority becomes even more evident as the amount ofannotated data further decreases.
We consider the enhanced robustness under sparsely labeled data asa crucial feature to have for real-world graphs.
4.2 Ablation studies
Various input features.
We evaluate the model performance with four types of inputs: randomnoise, hand-crafted features by Xu et al.
[6], node-community affiliation scores ( i.e. , Φ ), and thecombination of the last two.
The quality of these inputs are evaluated by the graph classificationaccuracy as summarized in Table 4.
Two conclusions could be drawn: (i) Φ is comparably informativeto the end task as hand-crafted node features, both are significantly better than random noise (see rows 1 – 3 of Table 4); (ii) the information Φ and X hold for the task enhances the discriminativelevel of learned representations in a complementary manner (see rows 2, 4 of Table 4).
These resultsverify the quality of Φ inferred by VEPM as additional node features.
[Table]
With or without pretraining.
Finally
[Table]
we compare the performance of VEPM that undergoes bothpretraining and finetuning against that of a VEPM trained with Equation (5) from scratch.
Results in
Table 5 shows that comparing with training from scratch, the state of the community encoder afterpretraining is better for the classification tasks, suggesting the benefits brought by pretraining
[Table] [Table]
4.3 Qualitative analysis
Visualizing community structures.
To show that through partitioning the edges, VEPM identifieseach latent community from the original graph, in Figure 3, we plot the adjacency matrices of a200-node subgraph of Cora, before and after edge partition.
The subgraph is created via breadth-firstsearch [40] node selection to ensure connectivity.
We sort the nodes sample S in order to present aclearer view of the community structures.
Specifically, we prepare K buckets; for node u , we computethe total interactions it engages under metacommunity k by µ u,k := (cid:80) v ∈ S Φ u,k diag( γ k ) Φ ′ v,k , andassign it to bucket k ′ , where k ′ =
arg max k µ u,k .
We first sort the buckets by descending their countsof assigned nodes, then sort the nodes within bucket k, k ∈ { 1 , . . .
, K } , by descending value ofµ u,k .
The Φ used for sorting is sampled at the end of training.
The community structures that VEPM detects could be found in Figure 3a as the blocks on the maindiagonal.
The fact that most of the bright spots ( i.e. , edges) are located in these on-diagonal blocksreflects dense node connections internal to each community.
Sparsely distributed off-diagonal spotsprovide evidence of overlapping membership between the communities, which is also taken intoaccount by our graph generative model.
Figures 3b to 3i show that the edge partitioner has donea sound job in separating latent communities.
Due to the limited size of the subgraph, some smallmetacommunities may not contain any large-weighted edges between the nodes in this subgraph,which accounts for the appeared emptiness in Figures 3h and 3i.
Such kind of artifact disappears whenwe replace the subgraph by the full graph of Cora, as visualized in Figures 6 to 14 in Appendix I.
Visualizing latent representations.
The previous visualization experiment verifies the followingpropositions: (i) the identified hidden structures are latent communities, and (ii) the partitioned graphsare different from each other.
We now study how they benefit representation learning.
Cora and
MUTAG are selected as the representatives for node and graph classification benchmarks.
For the
MUTAG dataset, we remove graphs that contain node categories with less than 5 instances (less than10 from a totality of 188) and randomly sample 10 graphs for the visualization experiments.
We first visualize Φ obtained at the end of the unsupervised pretrain stage (Figures 4a and 4b), wherethe node-community affiliation vectors are projected to 2-D space via t-SNE [41].
Proposition (i)ensures that the node information carried by Φ is about communities.
We color code the scattersby node labels, both Figures 4a and 4b exhibit strong correlation between the spatial clusters andcolors, which indicates that even without label supervision, the node information provided with Φhas discriminative power on classifying nodes, which is coherent with the conclusions in Section 4.2.
We then visualize the community-specific node emebddings obtained at the end of the supervisedfinetune stage (Figures 4c and 4d).
Similarly, t-SNE is adopted to reduce the dimensionality ofobtained node embeddings.
This time we color code the scatters by the latent metacommunity theycorrespond to.
Both Figures 4c and 4d exhibit clear boundaries between the colored clusters, whichshows that the model is able to extract different information from multiple communities, which
[Figure] [Figure]
enhances the overall expressiveness of learned node or graph representations and potentially leads tobetter model performance on the downstream tasks.
4.4 Complexity analysis
Recall that N , M , K denote the numbers of nodes, edges, and metacommunities in the graph;
F denotes the feature dimension; and L 1 , L 2 , L 3 denote the number of layers in the communityencoder , in the community-GNN bank , and in the representation composer .
The time complexity oftraining VEPM is O
(( L 1 + L 2 + L 3 ) MF + ( L 1 + L 2 / K + L 3 ) NF 2 + N 2 F ) .
For a sparse graphwhere N 2 ≫ M , the computational overhead can be reduced to O ( M ) 1 if graph reconstruction isaccelerated via subsampling the nodes to O ( √ M ) , as in Salha et al.
Effects and implications ofadopting such type of acceleration algorithms are discussed in Appendix E.
The space complexity of
VEPM is O
(( L 1 + L 2 + L 3 ) NF + KM +
( L 1 + L 2 / K + L 3 ) F 2 ) , among which O (( L 1 + L 2 / K + L 3 ) F 2 )is contributed by model parameters.
It is noteworthy to point out that the memory cost of graphreconstruction is manageable by computing the dense matrix multiplication block-wise with a fixedmaximum block size.
From all aspects of complexity, VEPM (with acceleration) is comparable to
GATs [13, 32] and models involving graph factorization [27, 28].
Conclusion
Moving beyond treating the graph adjacency matrix as given, we develop variational edge partitionmodels (VEPMs) to extract overlapping node communities and perform community-specific nodefeature aggregations.
Specifically, we first utilize a GNN-based inference network to obtain nodecommunity affiliation strengths, with which we augment node attributes, and partition the edgesaccording to the intensities of node interactions with respect to each community.
We learn GNNbased node embeddings for each community by aggregating node features with the correspondingpartitioned graph, and aggregate all community-specific node embeddings for the downstream tasks.
Extensive qualitative and quantitative experiments on both node-level and graph-level classificationtasks are performed to illustrate the working mechanism and demonstrate the efficacy of VEPMs insupervised graph representation learning.
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Abstract
(b) Did you describe the limitations of your work?
[Yes] See Appendix
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemen-
tal material
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.1 and Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See supplemental
material
(b) Did you mention the license of the assets?
[Yes] See supplemental material
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See supplemental material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]
A The Pseudocode of Training Algorithm
Algorithm
The algorithm of VEPM
Data: node features X , observed edges A , and observed labels y o .
Modules: Community ENCoder ( CENC ), Edge PARTitioner ( EPART ), Community-GNN BANK ( CGNN BANK ) and REPresentation COMPoser ( REPCOMP ).
Parameters: ω in the inference network , and θ in the generative network .
Initialize ω and θ ; repeat Φ , K , Λ ← CENC( A , X ) ; compute L egen and L KL , given in Equation (5); ω ← ω − η unsup · ∇ ω ( L egen + L KL ) ; until CENC converges repeat Φ , K , Λ ← CENC( A , X ) ;
A (1) , A (2) , · · · , A ( K ) ← EPART( Φ , A ) ; for step ← 1 to M do ˆ p y
o ← softmax (cid:0) REPCOMP ◦ CGNN BANK( Φ , X , A (1) , A (2) , · · · , A ( K ) ) (cid:1) ; compute L task , given in Equation (5);
θ ← θ − η sup , θ · ∇ θ L task ; end for compute L ; ω ← ω − η sup , ω · ∇ ω L ; until the whole model converges
B Selection Rationale for the Probabilistic Distributions
The Bernoulli-Poisson distribution, which augments binary edges into counts of edges, not onlyadds interpretability in terms of edge partition, but also is well suited for sparse graphs.
The gammaprior not only ensures positivity and promotes sparsity on Φ , but also allows an analytic KL termby combining with the Weibull variational distribution, which is reparameterizable.
Here are someproperties of the Weibull distribution [43] that we would like to highlight:
• Similar density characteristics with gamma distribution
The Weibull distribution has similar characteristics with the gamma distribution, i.e. , the densityfunctions of the two distributions are similar, and both enjoy the flexibility in modeling sparse andnonnegative latent representations.
[Equation]
• Easily Reparameterization
The latent variable x ∼ Weibull ( k, λ ) can be easily reparameterized as
[Equation]
leading to an expedient and numerically stable gradient calculation.
• Analytic KL-
Moverover, the KL-divergence between the Weibull and gamma distributions has an analytic expression formulated as
[Equation] [Equation]
C Experimental Settings
We evaluate VEPM on 11 benchmarks.
For node classification, we consider three citation networks:
Cora, Citeseer, and Pubmed, which provide bag-of-words document representations as node featuresand (undirected) citations as edges.
For graph classification, we consider four bioinformatics datasets(MUTAG, PTC, NCI1, PROTEINS) and four social network datasets (IMDB-BINARY, IMDBMULTI, REDDIT-BINARY and REDDIT-MULTI).
The input node features are crafted in the sameway as Xu et al.
We summarize the statistics of these datasets in Table 6.
For semi-supervisednode classification, we use the train/test/validation split standardized by the GCNs [5] to make a faircomparison.
We employ two protocols to evaluate the performance of VEPM on graph classification.
Most of the baselines are compared following the 10-fold cross validation based evaluation protocolproposed by Xu et al.
Since this protocol is criticized for being prone to overestimating modelperformance [30], we also evaluate our model following Zhang and Chen [31], who conduct a morerigorous train-validation-test protocol.
In the node classification experiments, for the community encoder , we adopt a similar encoderstructure devised by Kipf and Welling [7], with the nonlinearity set to softplus .
We select K ,the number of metacommunities, through cross-validation, with K = 4 for Citeseer and K = 8 for
Cora and Pubmed.
We set the depth of the generative network to two layers, and evenly distribute thelayers to the community-GNN bank and representation composer .
Specifically, the community-GNNbank is implemented with K single-layer GCNs [5] and following Veli
ˇ ckovi ´ c et al.
[13], we set theiroutput dimensions as 64/ K .
The representation composer is implemented with a single-layer GCN[5].
We record all hyperparameters in Table 7 in Appendix C.3.
For graph classification, we develop VEPM based on both GCN [5] and GIN [6], and preset the modelhyperparameters when evaluating under the first protocol.
We provide structural detials, the rationalebehind hyperparameters determination in Appendix C.2 and sensitivity analysis in Appendix D.
Through our studies, we find the variation of VEPM’s graph classification performance with respectto different choices of model architecture or K is quite minor.
For the second protocol, we follow themethod of Zhang and Chen [31] to select hyperparameters, and report the test accuracy when themodel achieves its highest validation accuracy over 100 epochs of training.
All experiments are carried out on NVIDIA Quadro RTX 5000, with Pytorch 1.9.0 and Python 3.8.
C.1 Data Statics
[Table]
C.2 Hyperparameter determination for the graph classification tasks
The model we develop for the graph classification tasks adopt a 2-layer GCN [5] for the communityencoder .
The dimension of ϕ is set to 100, and the dimension of the first hidden layer is set to200 according to the 2:1 ratio in the number of neurons in the first two hidden layers of the GCNbased encoder structure which is proved useful in node classification.
We implement the generative network based on GIN [6], which has four consecutive GIN layers whose hidden dimensions areall 64.
Similar to our practice in VEPM for node classification, we evenly distribute the 4 layers tothe community-GNN bank and the representation composer ; in the community-GNN bank , the 64dimension hidden units are evenly partitioned to all community-specific GNN components.
Betweenthe two choices of K we learned from the node classification experiments, we pick the smaller one,i.e. , K = 4 for the graph classification task, as the sizes of the graphs are much smaller than those fornode classification.
We conduct sensitivity analyses on the choices of K and the architectures of thegenerative network , and present the results in Appendix D.
[Table]
C.3 All hyperparameters
[Table] [Table]
[Table] [Table]
[Table] [Table]
D Hyperparameter Sensitivity Analysis for Graph ClassificationD.1 Number of metacommunities, K
To investigate the sensitivity of VEPM at different choices of K , the number of metacommunities, wehold all other hyperparameters as constants and run 10-fold cross validations with K ∈ { 4 , 5 , 6 , 7 , 8 } .
From the results in Table
[Table]
we can find that (i) the variation in graph classification performancethat
VEPM achieves with different values of K is not significant, most of the results recorded in thesame column are within one standard deviation with each other; (ii) there is not a collective patternbetween model performance and K shared across all benchmarks, in other words, not a single settingof K could consistently outperform other settings on all datasets.
Given the above observations, weconclude that the performance of VEPM performance on graph classification is not significantlyinfluenced by the number of metacommunities within a reasonable range.
[Table]
D.2 Network Structure of the generative network
In this part, we focus on studying the potential influence of different network structures on the graphclassification tasks.
As we decide to make the generative network in VEPM to have a comparablenumber of parameters with GIN [6], the variation space left for the network structure is at how todistribute the 4 GIN layers to the community-GNN bank and representation composer .
We use thename convention
VEPM-
{ L 1 } - { L 2 } to denote the variant with L 1 GIN layers in the community-GNNbank and L 2 GIN layers in the representation composer , where L 1 , L 2 ∈ Z + and L 1 + L 2 = 4 .
Wehold all other hyperparameters as constants and evaluate VEPM under 3 structures: VEPM- { 1 } - { 3 } ,
VEPM- { 2 } - { 2 } and VEPM- { 3 } - { 1 } .
From the results shown in Table 9, the graph classificationperformance that VEPM achieves at different combinations of L 1 and L 2 are quite close to eachother, the differences among the results in each column are at low statistical significance level.
Wehence conclude that the predictive power of VEPM is relatively stable at the variation of the networkstructure of the generative network .
[Table]
E Subsampling-Based Acceleration Algorithm
Like all neural graph generative models that model the connection statuses of all (cid:0) N 2 (cid:1) node pairs, e.g. ,
VGAE [7] and its various extensions, the time complexity of (pre)training VEPM is also O ( N 2 ) .
Thus a direct implementation that considers all node pairs in each iteration will face the scalabilityissue as N increases.
Fortunately, such computation burden could be reduced to O ( N 2 S ) throughsubgraph-sampling based acceleration, where N S denotes the number of nodes in the subgraph aftersampling, as analogous to FastGAE [42].
Specifically, we develop an explainable and flexible strategy to sample nodes from the original graph with replacement , where node i is sampled at probability p
i :
[Equation]
With f ( · ) measuring the “importance” of node i , q i denotes the probability of sampling node i for itsrelative importance in graph G .
The sharpness of the probability distribution { q i } i =1 ,N is adjusted bythe exponent α , where greater sharpness can be obtained with a higher value of α , and { q i } i =1 ,Nwould be equivalent to an uniform distribution if α is set to 0.
Likewise, 1 − q i N − 1 can be interpretedas the probability that node i is sampled for its relative unimportance.
For a better coverage, both“important nodes” and “unimportant nodes” are to be included in the sampled subgraph, with theirpercentages balanced by k ∈ [0 , 1] , i.e. , the higher the value of k , the more “important nodes” areexpected to be included in the subgraph.
We use k = 0 .
9 , α = 1 , and node degrees as f ( · ) .
We record both node classification accuracy and the elapsed time per epoch for VEPM trained withand without subgraph sampling.
The modeled is evaluated on Cora, Citeseer and Pubmed where N Sis set to 200.
As shown in Table 10, the acceleration algorithm successfully reduces the time costby a huge margin, it also results in a slight decrease in model performance and a mild inflation inthe error bar.
However, the negative impacts are not significant enough to offset the achievement inscalability, especially considering the fact that with the implication in model performance, VEPMwith scalable training still outperforms other baselines on two out of three benchmarks.
[Table]
F More Ablation Studies
[Table]
Different edge partition.
In this part, we study how much a meaningful edge partition wouldbenefit the task.
Except for the partition we obtained from training the regular VEPM, we testedthe performance of VEPM under two edge partition schemes: even partition and random partition.
Even partition means all edges weights are 1 / K .
For random partition, we sample the unnormalizededge weights from U(0 , 100) , then normalize them along the community dimension with softmax .
The random edge weights are fixed for training to simulate an undesirable convergence state of edgepartition.
The results are recorded in Table 11.
Generally speaking, the performance of VEPMwith even edge partition is comparable to that of GIN, our base model; when we substitute randompartition for even partition, the model performance becomes worse than the base model.
Theseablation study results suggest that VEPM can benefit from enhancing the quality of edge partitions.
[Table]
Different representation composer .
Next, we use the partition obtained from a regular VEPM torepresent partitions that are meaningful for the task, and a random partition to represent “meaninglesspartitions.”
For each partition scheme, we contrast the performance of VEPM between two designsof the representation composer : (i) GNN-based design, represented by the current design; (ii) parsimonious design, represented by a fully-connected (FC) layer.
The results are in Table 12.
Whenthe edge partition is meaningful, i.e. , relevant to the task, the overall performance with a simplerrepresentation composer is generally slightly worse than the current design, but still better than most of the baselines.
However, if the partition is irrelevant to the task, the GNN-based representationcomposer would be the last thing standing between the model and a failed task, as the graph structureused for neighborhood aggregation is still informative to the task.
So the current design would slightlybenefit VEPM in a general case, and protects the performance of the model in the worst-case scenario.
[Table]
Different softmax temperatures.
We use the same way as the previous study to get representativesof meaningful partitions and meaningless partitions.
In this study, we focus on the effect of differentselections of τ , the temperature parameter of softmax .
The value of τ controls the sharpness ofthe partitioned edge weights, i.e. , a small τ drives the partitied weights for the same edge towardsa one-hot vector, whereas a large τ would eliminate the divergence among the edge weights andmake the edge partition no more different from an even partition.
The results of this experiment arerecorded in Table 13, the upper half of which is obtained from a meaningful edge partition, and thelower half is obtained from a random edge partition.
We can induce from these results that when thebasis to perform edge partition, namely, the inferred latent node interactions, are relevant to the task,a sharp edge partition that highlights the differences among the communities may be helpful for thetask.
Otherwise a smooth edge partition might be more favorable.
[Table]
G The Working Mechanism of VEPM
In this section we explore how the communities benefit the graph-analytic task in VEPM, and wedraw the conclusions from two exploratory experiments on the Cora dataset.
We first assess therelevance of the learned communities to the task.
In order to see the statistical dependency betweencommunities and the task, we first hard-assign each node to the community of the highest affiliationstrength, then we compute the normalized mutual information (NMI) between the hard-partitionedcommunities and node labels.
NMI is a score between 0 and 1, with higher values indicating strongerstatistical dependencies between two random variables.
In the above experiment, we obtain an NMIof 0 .
316 when the communities are learned solely by the unsupervised pretrain , such value increasesto 0 .
322 after the supervised finetune .
The results of this experiment show that the initial communitieswe obtained from the unsupervised pretrain are meaningful to the task, and the relevance between theinferred communities and the task would be enhanced by the supervised finetune .
Next, we explore how the information provided by each community facilitates the task.
To see theeffect of each community, we first train VEPM until convergence, then use the node representationslearned by each community GNN to train an SVM and visualize the averaged 10-fold cross-validationresults by a confusion matrix.
With the results shown in Figure 5, we can further conclude that mostof the communities can provide task-related information, as we can observe from Figure 5 that thediagonal cells in most of the confusion matrices have darker shades than off-diagonal cells.
Beyondthat, the information extracted from different communities is complementary to each other.
Forexample, in the second column of Figure 5, information from the upper community can separate classes 3 and 4 well, but would mix classes 4 and 5; information from the lower community, on theother hand, can separate classes 4 and 5 but works worse on separating classes 3 and 4.
In summary, the communities learned by VEPM have the following properties: (i) the communitiesare relevant to the task; (ii) the information provided be different communities are complementary, sothe overall amount of information for the task is accumulated over communities.
Both properties areintuitively helpful for the downstream task.
[Figure]
Please
H Limitations and broader impacts
While our model exhibits superiority over traditional convolution based graph neural networks onboth node-level and graph-level representation learning tasks, several caveats are noteworthy.
Fromthe perspective of model design, we only focus on intra-community node interaction in our graphgenerative model, while it is a common practice and usually leads to good results in modelinghomophilous graphs, it cannot well explain the edges generated as a result of cross-community nodeinteraction, which potentially limits its modeling performance on heterogeneous graphs.
VEPM offers a a new perspective to model the network with latent communities, it separates latentcommunities from the entire network and models them individually.
Used with goodwill, it canunderstand what optimally benefits each community and help policy-making accordingly.
However,along with its modeling power that could be harnessed to benefit communities, it can be dangerous ifbeen put into socially harmful applications.
After all, no model is self-immune to malicious purpose,the potential misusages of VEPM include but not limited to group-targeted broadcast of fake news orconspiracy theories.
I Partitioned Adjacency Matrix