E DGE
P
ARTITION M ODULATED G RAPH C ONVOLUTIONAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph convolutional networks (GCNs), which propagate the node features through the edges and learn how to transform the aggregated features under label supervision, have achieved great success in supervised feature extraction for both graph-level and node-level classiﬁcation tasks.
However, GCNs typically treat the graph adjacency matrix as given and ignore how the edges could be formed by different latent inter-node relations.
In this paper, we introduce a relational graph generative process to model how the observed edges are generated by aggregating the node interactions over multiple overlapping node communities, each of which represents a particular type of relation that contributes to the edges via a logical OR mechanism.
Based on this relational generative model, we partition each edge into the summation of multiple relation-speciﬁc weighted edges, and use the weighted edges in each community to deﬁne a relation-speciﬁc GCN.
We introduce a variational inference framework to jointly learn how to partition the edges into different communities and combine relation-speciﬁc GCNs for the end classiﬁcation tasks.
Extensive evaluations on real-world datasets have demonstrated how the proposed method helps enhance the discriminative representation power and its efﬁcacy in learning both node and graph-level representations.
I NTRODUCTION
The graph data structure is widely adopted to describe interconnected objects, such as users in a social network and atoms held up by chemical bonds to form molecules.
The edges in real-world graphs usually indicate strong and logically reasonable relations between nodes.
Graph convolutional networks (GCNs) (Defferrard et al., 2016; Kipf & Welling, 2017;
Veli ˇ ckovi ´ c et al., 2018;
Klicpera et al., 2019a;b) capture this property of graph data by introducing neighborhood aggregation into regular perceptron layers.
While this approach typically enhances the feature dependencies among strongly-related adjacent nodes, the absence of relational inference makes the aggregation mechanism indifferent to the multiple types of latent inter-node relations, ignoring which limits the ultimate potential of the model performance.
Extensive efforts have been made to seek solution for performance enhancement in dealing with graphs with heterogeneous relations (Schlichtkrull et al., 2018; Vashishth et al., 2020; Nickel et al., 2015), where the common ground these methods converges to is to learn multiple graphs, each one speciﬁed to one type of relation.
The separation of relations-speciﬁc graphs from one graph is easy when these relations are explicitly annotated by edge labels, but is non-trivial when the relations are non-observable.
To address this issue, some GCNs introduce techniques such as multi-head (Veliˇckovi´c et al., 2018; Gao & Ji, 2019) or multi-hop (Abu-El-Haija et al., 2019) aggregation that increase the model capacity for modeling multi-relations, however, the aggregation weights in these models are either short of interpretability or too dependent on human knowledge to adapt to the underlying relations.
Another line of research aims to disentangle relation-speciﬁc node representations by task-driven graph factorization (Ma et al., 2019; Yang et al., 2020), where the relational disentanglement is performed relying on label supervision, hence their performances would be negatively impacted by the scarcity of observed labels.
In contrast to the aforementioned methods, we introduce edge partition modulated graph convolutional networks (EPM-GCNs), a generative framework that models the generation of edges and labels from latent relations.
To begin with, we represent the latent relations with co-membership of K overlapping latent communities.
For instance, in a social network where people are linked by online friendship, one person may be simultaneously afﬁliated with multiple social groups (e.g., graduating from the same school A, or working for the same company B, etc.).
Then some similaritybased relations (e.g., schoolmates of A, colleagues in B) would naturally exist among people that are afﬁliated with the same social group.
Under this modeling assumption, the strength of each relation between a pair of nodes is measured by the multiplication of the following latent quantities: (i) the average interaction rate among co-members in the corresponding community and (ii) how strong the two ending nodes are afﬁliated with this community.
The edges are further modeled by the logical OR of binary latent edges independently generated through the Bernoulli-Poisson link function given the strengths of each relation.
This part of the generative model explains the shrinkage of heterogeneous relations into binary latent edges.
We next build up the generation process of labels.
Since one node simultaneously engages in multiple relations, we generate its representation as a composition of relation-speciﬁc sub-representations.
These sub-representations are learned by three steps: the ﬁrst step partitions the edges by normalizing the strengths of K community-based relations, and deﬁnes K independent GCNs with each set of weighted edges; the second step obtains the desirable sub-representations by running the K
GCNs in parallel.
The labels of nodes or graphs could be predicted with the node representations generated in this way.
This generative model allows the community-based relations to be inferred from both labels and edges, which not only systematically solves the relation non-observable issue, but also incorporates edges as an alternative source of information provided for the posterior inference of latent relations.
Since the observed edges dominate labels in quantity, our method would not heavily depend on label supervision, hence may suffer less from the scarcity of labels.
We summarize the major contributions of EPM-GCNs as follows:
• We develop a novel generative model that generates labels from relation-speciﬁc subrepresentations, which are at ﬁrst learned through independent neighborhood aggregations with weighted edges that are partitioned by the strengths of latent relations; • We integrate the label generation process with the generation of edges, which enables the observed edges to compensate for the scarcity of observed labels by providing extra information for latent communities/relations; • The proposed EPM-GCNs are end-to-end trainable via variational inference; • We analyze the working mechanism of EPM-GCNs and evaluate the models over various real-world network datasets.
Empirical results show that the EPM-GCNs achieve state-ofthe-art performance in most of the node and graph-level classiﬁcation tasks.
E DGE PARTITION MODULATED GCN S
In this section, we present EPM-GCNs that learn the representation of node or graph from relationspeciﬁc sub-representations.
As shown in Figure 1, the overall architecture of EPM-GCNs consists of a generative network (parameterized with θ ) and an inference network (parameterized with ω ).
Section 2.1 outlines the data generation process and corresponding supportive modules in the generative network , Section 2.2 elaborates the module in the inference network as well as the inferencelearning details.
2. G ENERATIVE TASK - LEARNING WITH LATENT COMMUNITIES
For the generative supervised graph analytic tasks under discussion, the information of a graph G with N nodes could be summarized by the triplet ( X , A , y ) , where X is the design matrix whose rows represent the features of individual nodes, A is the binary adjacency matrix of G , with each “1” entry indicating the presence of an edge between the corresponding node pair, and y ∈ { y o , y u } is the set of labels, with subscripts o and u denoting the observed and unobserved parts of labels, respectively.
The goal of our latent factor model is to describe the generation process of observed edges and labels.
Based on that, we are enabled to predict the holdout labels from their posterior.
The joint generation of edges A and labels y is modeled by stacking up the generation process of edges and the generation of labels given the edges.
[Figure]
2.1. T HE GENERATION OF EDGES
We adopt the generation process developed in Zhou (2015) to model the formation of edges A :
[Equation]
) which has an alternative representation under the Bernoulli-Poisson link as A i,j =
[Equation]
∼
Poisson (cid:0) (cid:80) Kk =1 γ k Φ i,k Φ j,k (cid:1) .
In this expression, Φ is a positive node-community afﬁliation matrix, whose entry at the i th row and k th column could be interpreted as the strength that node i is afﬁliated with community k , γ k ( k ∈
[ K ] ) is a positive community activation level indicator that measures the member interaction frequencies featured by community k , and γ k Φ i,k Φ j,k quantiﬁes the interaction rate between nodes i and j through community k .
In our practice, we treat γ =
[ γ 1 , γ 2 , · · · , γ k ] (cid:48) as a part of the trainable parameters θ .
In this paper, we focus on another equivalent representation that partitions each edge into the logical disjunctions of K latent binary edges, expressed as
[Equation]
where ∨ is the logical disjunction ( i.e. , logical OR) operator and p ijk denotes the probability that the interaction of nodes i and j in community k results in an edge.
The connection between nodes i and j would establish as long as their interaction in one community forms up an edge, in other words, nodes i and j would be disconnected only if their interactions in ALL communities fail to generate edges.
Link function 1 gives the conditional probability p θ ( A | Φ ) .
Note that the node interaction over each community uniquely corresponds to one type of inter-node relation, the generation of A reﬂects the aggregation of these heterogeneous relations, suggesting that one would need to partition the edges to separate these relations.
2.1. T HE GENERATION OF LABELS :
In consonance with the graph generative model that establishes the graph from latent communities, we develop the generation process of labels from all relation-speciﬁc representations.
To achieve that, we ﬁrst need to separate each relation from their totality, i.e. , factorize the edges with the edge partitioner module:
Edge PARTitioner ( EPART ).
The edge partitioner takes edges A and community-afﬁliation score matrix Φ as inputs and returns K positive-weighted edges: { A (1) , A (2) , · · · , A ( K ) } , s . t .
(cid:80)
Kk =1 A ( k ) =
A .
The partition function follows the form
[Equation]
Here τ denotes the “temperature” that controls the sharpness of weights, its value is speciﬁed in
Table 6 in the appendix.
The edge weights produced from edge partitioner represent the proportion that each community-based relation takes in contributing to edge formation.
When τ is set sufﬁciently small, A (1) , A (2) , · · · , A ( K ) would approximate binary factors graphs of G , we hence refer to them as “relational graph factors”.
Replacing γ k Φ i,k Φ j,k with Φ i,k diag ( γ k ) Φ (cid:48)
j,k generalizes Equation 2 to metacommunity-based edge partition, in this expression, the total number of communities is greater than K , and Φ i,k denotes the k th segment in node i ’s community-afﬁliation encoding.
The generalized edge partitioner enriches the choices of Φ ’s dimensionalities, which enhances the ﬂexibility in model implementation.
Subsequent to the edge partitioner follows the relational GCN bank :
Relational GCN BANK ( RGCN BANK ).
The relational GCN bank is made up of K independent
GCN components, coupling with the corresponding relational graph factors.
It takes X ∗ :
= X (cid:107) Φ as input 2 , and outputs g (1) θ ( X ∗ ) , g (2) θ ( X ∗ ) , · · · , g ( K ) θ ( X ∗ ) ; g
[Equation]
The function of this design is to learn relation-wise separable node representations.
We now qualitatively justify the fulﬁllment of this requirement.
Without loss of generality, let us treat the relational graph factors as approximately binary, this means the each observed edge is annotated with the relation that makes the greatest contribution in its generation.
Let us denote the relations as r 1 , r 2 , · · · , r k which are uniquely featured with communities c 1 , c 2 , · · · , c K .
Assuming node u connects to its neighbors v 1 and v 2 by relations r 1 and r 2 , respectively.
Without edge partition, features of v 1 would be propagated to v 2 in two runs of neighborhood aggregation even if they connects to u with different community identities.
This issue is moderated by the relational GCN bank , as the aggregation path ( v 1 , u ) would only be sent to the GCN component for relation r 1 whereas the path ( u, v 2 ) would only exist in the GCN component corresponds to relation r 2 .
As a result, in each
GCN component, nodes only share features through edges explained by the same relation, we hence consider the outputs of relational GCN bank as relation-speciﬁc.
Finally, we learn the representations of nodes from relation-speciﬁc sub-representations.
This step involves the representation composer module:
REPresentation COMPoser ( REPCOMP ).
Let H ( k ) :
= g ( k ) θ ( X ∗ ) denote the node representations learned from the k th relation, k ∈
[ K ] , and function f ( · ) denote the representation composer, whose functionality is to project a composite of relation-speciﬁc node representations to one representation matrix, i.e., H V = f (cid:0) H (1) , H (2) , · · · , H ( K ) (cid:1) :
= GCN θ (cid:0) (cid:107) Kk =1 H ( k ) , A (cid:1) .
We could further obtain h G , the vector representation of graph G , by sending the node level representations
H V to a graph pooling layer, i.e., h G :=
GRAPHPOOL( H V ) .
Taking softmax on the feature dimension of the representation composer ’s output gives the uncertainty distribution of labels, from which we are able to make predictions on the category of the object (node or graph).
Cascading the edge partitioner , the community-GCN bank with the representation composer yields the probability p θ ( y | A , Φ , X ) .
2. T RAINING EPM-GCN: INFERENCE AND LEARNING 2.2. T HE REPARAMETERIZABLE EVIDENCE LOWER BOUND OBJECTIVE
We set the prior distribution of Φ as Φ i,k iid ∼ Gam( α, β ) , ∀ ( i, k ) ∈ [ N ] × [ K ] , and use a community encoder 3 to approximate the posterior distribution p ( Φ | A , y ) :
Community ENCoder ( CENC ).
The community encoder models the variational posterior q ω ( Φ ) by a Weibull distribution with shape K and scale Λ , both parameters are learned by a GCN, i.e. ,
K (cid:107) Λ =
GCN ω ( X , A ) .
A random sample from q ω ( Φ ) could be created through the inverse CDF transformation of a uniformly distributed variable, given as follows:
[Equation]
The operators (cid:12) , (cid:11) and ◦ (in the front of superscript 4 ) denote element-wise multiplication, division and power, respectively.
To jointly infer the latent factors Φ and use them to learn the task, we optimize the evidence lower bound objective (ELBO) L , given by Equation 3, i.e. ,
[Equation]
where the notations P Φ and Q Φ represent the prior and the variational posterior distributions of Φ , respectively.
The three right hand side terms correspond to the classiﬁcation task, graph reconstruction, and KL-regularization, we thereby refer to them as L task , L rec and L KL in the sequel.
Note that the aforementioned speciﬁcations of Φ ’s prior and variational posterior, as in Zhang et al.
(2018a), yield an analytical expression of L KL .
2.2. T HE OVERALL TRAINING ALGORITHM
To effectively train EPM-GCNs, we separate the overall training algorithm into an unsupervised pretrain stage, followed by a supervised ﬁnetune stage.
The unsupervised pretrain: The ultimate goal of the community encoder is to ﬁnd a community assignment according to which edge partitions would facilitate task learning.
Since the overall likelihood of node labels involves stacking multiple modules embodied by deep neural networks, inferring the posterior p ( Φ | A , y o ) is difﬁcult from an optimization perspective.
To seek better model convergence, we break this optimization problem into two stages, namely, we ﬁrst train the community encoder to discover the latent communities that explains the graph architecture, then ﬁnetune the communities to align them to task-learning.
The ﬁrst stage is the unsupervised pretrain , at which we pretrain the community encoder until convergence by optimizing L rec + L KL , i.e. , updating model parameters ω towards ω ∗ , which optimizes E q ω ( Φ ) log p θ ( A | Φ ) − D KL ( Q Φ (cid:107) P Φ ) .
The variational probability q ω ( Φ ) is learned to approximate p ( Φ | A ) .
The heuristic that the unsupervised pretrain is generally supportive to the ultimate goal is that although communities and node categories describe node similarity from different aspects, these two aspects of similarity are semantically mutually informative, i.e. , the discovered communities from graph architecture is inherently correlated with node labels, which is suggested by previous work on graph representation learning with community information (Sun et al., 2019), in which the node classiﬁcation results clearly beneﬁt from the node representations that hold community information.
Such assumption is further qualitatively veriﬁed in Section 4.2.
We ﬁnd through practice that the unsupervised pretrain not only stabilizes model convergence, but also provides a reasonable initial state for subsequent supervised optimization thus leads to better performances.
The supervised ﬁnetune: This stage involves iteratively running inference and learning .
In the inference step, we ﬁx the generative network , and train the inference network by optimizing the ELBO
L , given in Equation 3.
Each inference step runs after M learning steps, where we treat the inference network as stationary and update the generative network by optimizing L task .
Both L and L task are computed over the training set { y o , A , X } .
The supervised ﬁnetune updates the approximation target of the inference network from p ( Φ | A ) to p ( Φ | A , y o ) , which ﬁnetunes community inference with label supervision, making the subsequent edge-partition more facilitative to the classiﬁcation task.
The entire training pipeline of EPM-GCNs is summarized in Algorithm 1 (in Appendix A).
After running the unsupervised pretrain and the supervised ﬁntune stages, the variational distribution q ω ( Φ ) is trained to approximate the posterior p ( Φ | y o , A ) .
We hence approximate the predictive distribution of unobserved labels with E q ω ( Φ )
[ p θ ( y u | A , Φ , X )] , and estimate it via Monte Carlo estimation.
In particular, we estimate p ( y u | A , X , y o ) with 1 S (cid:80) Ss =1 p θ ( y u | A , Φ ( s ) , X ) , where Φ ( s ) iid ∼ q ω ( Φ ) for s = 1 , . . .
, S .
For simplicity, we set S = 1 unless speciﬁed otherwise.
[Table]
R ELATED W ORK
Deep graph generative models: Advances in this line of work include adopting deep architectures in learning node embeddings (Kipf & Welling, 2016;
Li et al., 2018) and joint optimization with associated graph-analytic tasks such as semi-supervised node classiﬁcation (Hasanzadeh et al., 2019;
Wang et al., 2020), community discovery (Sun et al., 2019; Mehta et al., 2019), etc .
Similar to these models, we regularize the node embeddings Φ by its probabilistic dependency with the graph structure, but instead of simply passing the learned embeddings to a black-box prediction network, we use them to develop the generative process of the labels, leading to better interpretability.
GCNs modeling multi-relational data: The general idea is to individually process node representations aggregated from different types of relations.
This logistic is widely adopted by GCN variants handling graph data where relational annotations are either explicit (Schlichtkrull et al., 2018; Vashishth et al., 2020) or implicit.
In the latter setup, the missing relations are conventionally completed with human knowledge (Abu-El-Haija et al., 2019; Vashishth et al., 2020), or learned via label supervision (Veli ˇ ckovi ´ c et al., 2018; Gao & Ji, 2019; Ma et al., 2019;
Yang et al., 2020).
Contrary to these methods which treat edges as given, we jointly model the generation of labels as well as the edges, and systematically model the relations by posterior inference.
The inferred relations are then used to partition edges, learn relation-speciﬁc sub-representations, and ﬁnally optimize the downstream tasks.
E MPIRICAL EVALUATION
We compare the proposed EPM-GCNs against related baselines on two fundamental tasks: node classiﬁcation and graph classiﬁcation.
In the sequel, we use the sufﬁxes - n and - g to distinguish the variants of EPM-GCN developed for node and graph-level classiﬁcation tasks, respectively.
4. D ATASETS PREPARATION
We evaluate EPM-GCNs on 11 benchmarks.
For node classiﬁcation, we use 3 citation networks:
Cora, Citeseer and Pubmed.
All three datasets provide bag-of-words document representations as node features and (undirected) citations as edges.
For graph classiﬁcation, we use 4 bioinformatics datasets (MUTAG, PTC, NCI1, PROTEINS) and 4 social network datasets (IMDB-BINARY,
IMDB-MULTI, REDDIT-BINARY and REDDIT-MULTI).
The input node features are crafted in the same way with Xu et al.
We summarize the statistcis of selected datasets in Table 1.
4. V ISUALIZING LEARNED REPRESENTATIONS
To qualitatively assess the proposed model, we visualize the spatial distribution of Φ obtained at the end of the unsupervised pretrain stage (Figures 2(a), 2(b)), and the relation-speciﬁc subrepresentations obtained at the end of the supervised ﬁnetune stage (Figures 2(c), 2(d)).
Node representation are projected to 2-D space via t-SNE (Van der Maaten & Hinton, 2008).
We select Cora and
MUTAG as representatives of the node and graph classiﬁcation datasets.
For the MUTAG dataset, we remove less than 10 graphs that contain node categories with less than 5 instances and randomly sample 10 graphs for visualization.
In Figures 2(a) and 2(b), the spatial clustering of nodes from the same class indicates strong correlation between inferred latent communities and node labels, which not only demonstrates our model’s ability in capturing community structure (even in small graphs with only tens of nodes), but also positively supports our model designs, including (i) input feature augmentation: X ∗ =
X (cid:107) Φ and (ii) edge partition.
Figures 2(c) and 2(d) further justiﬁes that the sub-representations { g (1) θ ( X ∗ ) , g (2) θ ( X ∗ ) , · · · , g ( K ) θ ( X ∗ ) } are not repetitive, signaling that the proposed model successfully extracts relation-speciﬁc representations, which potentially enhance the discriminative power of node or graph representations and lead to better performances on the downstream tasks.
[Figure]
4. N ODE CLASSIFICATION
Experimental settings: To make a fair comparison, we use the train/test/validation split standardized by the vanilla GCN (Kipf & Welling, 2017) for Cora, Citeseer and Pubmed datasets.
For the community encoder , we adopt a similar encoder structure devised by Kipf & Welling (2016) with softplus activations, which outputs 16-dimensional community-afﬁliation scores for each node.
We set the number of metacommunities K = 4 for Cora and Citeseer and K = 8 for Pubmed dataset, the values of K are selected through cross-validation.
The relational graph factors are then obtained through the generalized edge partition.
Before being sent to the relational GCN bank , these graph factors are row-normalized by sparse softmax.
We implement the relational GCN bank with K single-layer GCNs with ReLU activation, following (Veli ˇ ckovi ´ c et al., 2018), we set their output dimensions as 64/ K .
The representation composer is also implemented with a single-layer GCN.
With these speciﬁcations, the amount of parameters in the generative network remains comparable with that of a 2-layer GCN with 64 hidden units (GCN-64) regardless of the choice of K .
We record the complete hyperparameter speciﬁcations in Table 6 (in Appendix C).
[Table]
Performance comparison:
We use classiﬁcation accuracy as the evaluation metric in the node classiﬁcation tasks.
Table 2 reports EPM-GCN- n ’s average performances ( ± standard error) against the paper records of the average performances of 3 groups of related baselines on the three citation network datasets, where ∗ marks the results reported by
Veliˇckovi´c et al.
The ﬁrst group (Defferrard et al., 2016; Kipf & Welling, 2017) are plain GCNs that aggregated node features with only one set of weighted edges.
These models are designed to handle at most two types of node relations: connected or not, so it is not surprised to ﬁnd that EPM-GCN- n outperforms these models by a wide margin.
Since removing (i) relational inference and (ii) relation-speciﬁc feature learning degenerates EPM-GCN- n to GCN-64, the improvement of our model against GCN-64 could be explained as the joint effects of the mentioned two practices.
The second group (Hasanzadeh et al., 2019; Wang et al., 2020) are graph generative models that jointly optimized with semi-supervised loss.
Both SIG-VAE (Hasanzadeh et al., 2019) and WGCAE (Wang et al., 2020) adopt similar graph generative model and loss structure to ours, the difference is that these models directly projects the latent representations to node labels by a deterministic prediction network, whereas we sample labels from a edge-partition based label generation process.
Therefore our improvement with respect to the second group could be attributed to the marginal beneﬁt from our label generative model.
The third group of models (Veliˇckovi´c et al., 2018; Gao & Ji, 2019; Ma et al., 2019) employ multi-head
[Table] [Table]
and adaptively weighted feature aggregation mechanisms, which greatly enhances their ability in handling multi-relational data.
However, these models lack systematic modeling of latent relations, which on the other hand is the strength of our model.
Our performance boost against the third group on most of the benchmarks demonstrates the marginal beneﬁt of our relational inference model.
4. G RAPH CLASSIFICATION
Experimental settings: For EPM-GCN- g , we implement the community encoder with a 2-layer
GCN and produces the latent community-afﬂiation encoding with 100 dimensions.
The number of metacommunities is set to K = 4 and corresponding relational graph factors are generated through the generalized edge partition.
We choose GIN layer as the building block to deﬁne the rest of the model, and set the total network depth as 5 (including the input layer) such that EPM-GCN- g is comparable with the vanilla GIN (Xu et al., 2019).
We deﬁne the K components in the relational
GCN bank by sharing the input layer and evenly partitioning the neuron units in the 2 top hidden layers.
The 2 ending layers are treated as the representation composer .
The remaining implementation details are included in Table 6 (in Appendix C).
Performance comparison: We follow GIN (Xu et al., 2019) to perform 10-fold cross validation for each dataset and report the average classiﬁcation accuracy ( ± standard error).
As shown in Table 3, we compare the developed EPM-GCN- g with several state-of-the-art baselines for graph classiﬁcation, including the WL subtree kernel (Shervashidze et al., 2011), Anonymous Walk Embeddings (AWL) (Ivanov & Burnaev, 2018) and other deep learning architectures like DCNN (Atwood &
Towsley, 2016), DGCNN (Zhang et al., 2018b), and PYTCHY-SAN (Niepert et al., 2016).
We also compare EPM-GCN- g with ﬁve supervised GNNs including GCN (Kipf & Welling, 2017),
GAT (Veli ˇ ckovi ´ c et al., 2018), GraphSAGE (Hamilton et al., 2017), and two variants of GIN (Xu et al., 2019): GIN-0 and GIN- (cid:15) .
Thanks to our uniquely designed multi-relational inference as well as relation-speciﬁc graph representation learning and composition, EPM-GCN- g achieves state-of-theart graph classiﬁcation performances on 5 out of 8 benchmarks, and the second-best performance on
NCI1 dataset, where it outperforms all the other deep architectures.
Notably, EPM-GCN- g achieves 75.6 % accuracy on PTC dataset with a signiﬁcant 7.9 % improvement over the second place.
4. A BLATION STUDY
To further study the marginal beneﬁt of each practice we take in implementing EPM-GCN, as well as model’s overall sensitivity to variations in hyperparameter setup, we evaluate EPM-GCN’s graph classiﬁcation performances over various choices of input features, number of metacommunities, and network structure.
Various input features: We ﬁrst evaluate model performances with four types of inputs: random noise, hand-crafted features by Xu et al.
(2019), community-afﬁliation scores, and hand-crafted features concatenated with community-afﬁliation scores.
Results are shown in Table 4.
The classiﬁcation accuracy measures the discriminative level of graph representations learned from these four types of inputs, comparing the ﬁrst and second rows with the third row tells us that communityafﬁliation scores are meaningful node representations and comparably informative to the task as
0 2 4 6 8 10 12 14 16 Number of Communities
A v e r age A cc u r a cy (a) IMDB-B 0 2 4 6 8 10 12 14 16 Number of Communities A v e r age A cc u r a cy (b) IMDB-M 0 2 4 6 8 10 12 14 16 Number of Communities
A v e r age A cc u r a cy (c) MUTAG
2 4 6 8 10 12 14 16 Number of Communities
A v e r age A cc u r a cy (d) PTC hand-crafted node features.
We could further conclude by comparing the second row with the fourth row that the information Φ and X hold for the task enhances the discriminative level of learned representations in a complementary manner.
Both conclusions justify the quality of Φ as additional node features.
Number of communities: We next study the variation of the model’s performances with growing number of metacommunities.
Figure 3 plots the average graph classiﬁcation accuracy over 10fold cross-validation against different selections of K ∈ { 1 , 2 , 4 , 8 , 16 } .
In the context of graph classiﬁcation, K denotes the number of all metacommunities with at least one presence among the graphs in the dataset.
We could see that the optimal results are typically achieved with K > 1 , which is in consistent with the model assumption that the graph contains multiple underlying relations, the result also reﬂects the efﬁcacy of the proposed model in modeling heterogeneous latent relations.
Network structures: Among our proposed results in Table 3, the network structure for the label generation is not particularly ﬁne-selected and hidden layers are evenly distributed to the relational
GCN bank and the representation composer , In this part, we focus on studying how the balance between these two modules inﬂuences the model performance on downstream tasks.
We use the name convention EPM-GCN- g - { L 1 } - { L 2 } to denote the variant with L 1 layers (including the input layer) in the relational GCN bank and L 2 layers in the representation composer , where L 1 > 0 and
L 1 + L 2 = 5 .
As Table 5 shows, although the optimal layer assignments for each dataset are not the same, i.e. , the IMDB-B and IMDB-M datasets generally require less relation-wise locality in node representations than MUTAG or PTC datasets, both modules are veriﬁed to be necessary for the graph-analytic tasks, because neither EPM-GCN- g - { 1 } - { 4 } nor EPM-GCN- g - { 5 } - { 0 } triumphs over other layer combinations in any of these experiments.
Additionally, the variation of model performances under different layer assignment demonstrates that datasets with larger amount of graphs are generally more insensitive to the change of model architecture, so the way to increase model robustness on datasets with less graphs would be a topic worth studying in the future.
[Table]
C ONCLUSION
Moving beyond treating the graph adjacency matrix as given, we develop novel EPM-GCNs to aggregate the node interactions over multiple overlapping node communities.
More speciﬁcally, we ﬁrst construct a community encoder to project each node to its community-afﬁliation scores
[Table]
and then partition the edges between nodes according to the edge-formation contribution of each communitybased relation with a edge partitioner .
Then we obtain the relation-speciﬁc sub-representations with the relation GCN bank and ﬁnally put them together to make predictions with a representation composer .
Extensive qualitative and quantitative experiments on both node and graph-level has been made to demonstrate the efﬁcacy of our model
[Table]
R EPRODUCIBILITY STATEMENT
The PyTorch code to reproduce the results in the paper has been included in the supplementary material.
Detailed hyperparameter settings have been provided in Table 6 in the Appendix.
A PPENDIX A T HE PSEUDOCODE OF TRAINING ALGORITHM
Data: node features X , observed edges A and observed labels y o
Modules : CENC , EPART , RGCN BANK and REPCOMP
Parameters: ω in the inference network , and θ in the generative network
Initialize ω and θ ; while the community encoder is not converged do Φ , K , Λ ← CENC( A , X ) ;
Compute L rec and L KL , given in Equation 3; ω ← ω − η unsup · ∇ ω ( L rec + L KL ) ; /*
Unsupervised pretrain */ endwhile the whole model is not converged do Φ , K , Λ ← CENC( A , X ) ;
A (1) , A (2) , · · · , A ( K ) ← EPART( Φ , A ) ; for step ← 1 to M do ˆ p y
o ← softmax (cid:0) REPCOMP ◦ RGCN BANK( Φ , X , A (1) , A (2) , · · · , A ( K ) ) (cid:1) ;
Compute L task , given in Equation 3; θ ← θ − η sup , θ · ∇ θ L task ; /
* The learning step */ end
Compute L ; ω ← ω − η sup , ω · ∇ ω L ; /*
The inference step */ end
B P ROPERTY OF W EIBULL DISTRIBUTION • Similar PDF with Gamma Distribution
The Weibull distribution owns similar probability density functions (PDF) with a gamma one, which makes it ﬂexible to model sparse and nonnegative latent representations:
Weibull
[Equation]
Gamma
[Equation]
• Easily Reparameterization
The latent variable x ∼ Weibull ( k, λ ) can be easily reparameterized as
[Equation]
leading to a similar gradient calculation with the Gaussian reparameterization.
• Analytic KL-Divergence
Moverover, the KL-divergence between the Weibull and gamma distributions has an analytic expression formulated as
[Equation]
C I MPLEMETATION DETAILS
In our experiments
[Table]
we ﬁnd the initial state of inference module provided by the unsupervised pretrain stage is essential to EPM-GCN’s performances.
The hyperparameters are either inherited from base models or selected on the basis of crossvalidation.
Note that for the graph-level classiﬁcation task, the parameters are shared across experiments on all datasets.
And the reported results are potentially improvable if dataset-speciﬁc cross-validations are applied.
[Table] [Table]