S TRUCT P OOL : S TRUCTURED G RAPH P OOLING VIA C ONDITIONAL R ANDOM F IELDS
A BSTRACT
Learning high-level representations for graphs is of great importance for graph analysis tasks.
In addition to graph convolution, graph pooling is an important but less explored research area.
In particular, most of existing graph pooling techniques do not consider the graph structural information explicitly.
We argue that such information is important and develop a novel graph pooling technique, know as the S TRUCT P OOL , in this work.
We consider the graph pooling as a node clustering problem, which requires the learning of a cluster assignment matrix.
We propose to formulate it as a structured prediction problem and employ conditional random ﬁelds to capture the relationships among the assignments of different nodes.
We also generalize our method to incorporate graph topological information in designing the Gibbs energy function.
Experimental results on multiple datasets demonstrate the effectiveness of our proposed S TRUCT P OOL
I NTRODUCTION
Graph neural networks have achieved the state-of-the-art results for multiple graph tasks, such as node classiﬁcation (Veli ˇ ckovi ´ c et al., 2018; Gao & Ji, 2019b; Gao et al., 2018) and link prediction (Zhang & Chen, 2018; Cai & Ji, 2020).
These results demonstrate the effectiveness of graph neural networks to learn node representations.
However, graph classiﬁcation tasks also require learning good graph-level representations.
Since pooling operations are shown to be effective in many image and NLP tasks, it is natural to investigate pooling techniques for graph data (Yu & Koltun, 2016; Springenberg et al., 2014).
Recent work extends the global sum/average pooling operations to graph models by simply summing or averaging all node features (Atwood & Towsley, 2016; Simonovsky & Komodakis, 2017).
However, these trivial global pooling operations may lose important features and ignore structural information.
Furthermore, global pooling are not hierarchical so that we cannot apply them where multiple pooling operations are required, such as Graph UNet (Gao & Ji, 2019a).
Several advanced graph pooling methods, such as SORTPOOL (Zhang et al., 2018), TOPKPOOL (Gao & Ji, 2019a), DIFFPOOL (Ying et al., 2018), and SAGPOOL (Lee et al., 2019) , are recently proposed and achieve promising performance on graph classiﬁcation tasks.
However, none of them explicitly models the relationships among different nodes and thus may ignore important structural information.
We argue that such information is important and should be explicitly captured in graph pooling.
In this work, we propose a novel graph pooling technique, known as the S TRUCT P OOL , that formulates graph pooling as a structured prediction problem.
Following DIFFPOOL (Ying et al., 2018), we consider graph pooling as a node clustering problem, and each cluster corresponds to a node in the new graph after pooling.
Intuitively, two nodes with similar features should have a higher probability of being assigned to the same cluster.
Hence, the assignment of a given node should depend on both the input node features and the assignments of other nodes.
We formulate this as a structured prediction problem and employ conditional random ﬁelds (CRFs) (Lafferty et al., 2001) to capture such high-order structural relationships among the assignments of different nodes.
In addition, we generalize our method by incorporating the graph topological information so that our method can control the clique set in our CRFs.
We employ the mean ﬁeld approximation to compute the assignments and describe how to incorporate it in graph networks.
Then the networks can be trained in an end-to-end fashion.
Experiments show that our proposed S TRUCT P OOL outperforms existing methods signiﬁcantly and consistently.
We also show that S TRUCT P OOL incurs acceptable computational cost given its superior performance.
B ACKGROUND AND R ELATED W ORK 2. G RAPH C ONVOLUTIONAL N ETWORKS
A graph can be represented by its adjacency matrix and node features.
Formally, for a graph
G consisting of n nodes, its topology information can be represented by an adjacency matrix
A ∈ { 0 , 1 } n × n , and the node features can be represented as X ∈ R n × c
assuming each node has a c -dimensional feature vector.
Deep graph neural networks (GNNs) learn feature representations for different nodes using these matrices (Gilmer et al., 2017).
Several approaches are proposed to investigate deep GNNs, and they generally follow a neighborhood information aggregation scheme (Gilmer et al., 2017; Xu et al., 2019; Hamilton et al., 2017; Kipf & Welling, 2017; Veliˇckovi´c et al., 2018).
In each step, the representation of a node is updated by aggregating the representations of its neighbors.
Graph Convolutional Networks (GCNs) are popular variants of GNNs and inspired by the ﬁrst order graph Laplacian methods (Kipf & Welling, 2017).
The graph convolution operation is formally deﬁned as:
[Equation]
where ˆ A =
A
+ I is used to add self-loops to the adjacency matrix, D denotes the diagonal node degree matrix to normalize ˆ A , X i ∈ R n × c i are the node features after i th graph convolution layer,
P i ∈
R
c
i × c
i +1 is a trainable matrix to perform feature transformation, and f ( · ) denotes a non-linear activation function.
Then X i ∈ R n × c
i is transformed to X i +1 ∈ R n × c
i +1 where the number of nodes remains the same.
A similar form of GCNs proposed in (Zhang et al., 2018) can be expressed as:
[Equation]
It differs from the GCNs in Equation (1) by performing different normalization and is a theoretically closer approximation to the Weisfeiler-Lehman algorithm (Weisfeiler & Lehman, 1968).
Hence, in our models, we use the latter version of GCNs in Equation (2).
2. G RAPH P OOLING
Several advanced pooling techniques are proposed recently for graph models, such as SORTPOOL,
TOPKPOOL, DIFFPOOL, and SAGPOOL, and achieve great performance on multiple benchmark datasets.
All of SORTPOOL (Zhang et al., 2018), TOPKPOOL (Gao & Ji, 2019a), and SAGPOOL (Lee et al., 2019) learn to select important nodes from the original graph and use these nodes to build a new graph.
They share the similar idea to learn a sorting vector based on node representations using GCNs, which indicates the importance of different nodes.
Then only the top k important nodes are selected to form a new graph while the other nodes are ignored.
However, the ignored nodes may contain important features and this information is lost during pooling.
DIFFPOOL (Ying et al., 2018) treats the graph pooling as a node clustering problem.
A cluster of nodes from the original graph are merged to form a new node in the new graph.
DIFFPOOL proposes to perform GCNs on node features to obtain node clustering assignment matrix.
Intuitively, the cluster assignment of a given node should depend on the cluster assignments of other nodes.
However, DIFFPOOL does not explicitly consider such high-order structural relationships, which we believe are important for graph pooling.
In this work, we propose a novel structured graph pooling technique, known as the S TRUCT P OOL , for effectively learning high-level graph representations.
Different from existing methods, our method explicitly captures high-order structural relationships between different nodes via conditional random ﬁelds.
In addition, our method is generalized by incorporating graph topological information A to control which node pairs are included in our CRFs.
2. I NTEGRATING CRF S WITH GNN S
Recent work (Gao et al., 2019; Qu et al., 2019; Ma et al., 2019) investigates how to combine CRFs with GNNs.
The CGNF (Ma et al., 2019) is a GNN architecture for graph node classiﬁcation which explicitly models a joint probability of the entire set of node labels via CRFs and performs inference via dynamic programming.
In addition, the GMNN (Qu et al., 2019) focuses on semi-supervised object classiﬁcation tasks and models the joint distribution of object labels conditioned on object attributes using CRFs.
It proposes a pseudolikelihood variational EM framework for model learning and inference.
Recent work (Gao et al., 2019) integrates CRFs with GNNs by proposing a CRF layer to encourage similar nodes to have similar hidden features so that similarity information can be preserved explicitly.
All these methods are proposed for node classiﬁcation tasks and the CRFs are incorporated in different ways.
Different from existing work, our S TRUCT P OOL is proposed for graph pooling operation and the energy is optimized via mean ﬁeld approximation.
All operations in our S TRUCT P OOL can be realized by GNN operations so that our S TRUCT P OOL can be easily used in any GNNs and trained in an end-to-end fashion.
S TRUCTURED G RAPH P OOLING 3. G RAPH P OOLING VIA N ODE C LUSTERING
Even though pooling techniques are shown to facilitate the training of deep models and improve their performance signiﬁcantly in many image and NLP tasks (Yu & Koltun, 2016; Springenberg et al., 2014), local pooling operations cannot be directly applied to graph tasks.
The reason is there is no spatial locality information among graph nodes.
Global max/average pooling operations can be employed for graph tasks but they may lead to information loss, due to largely reducing the size of representations trivially.
A graph G with n nodes can be represented by a feature matrix X ∈ R n × c and an adjacent matrix A ∈ { 0 , 1 } n × n .
Graph pooling operations aim at reducing the number of graph nodes and learning new representations.
Suppose that graph pooling generates a new graph ˜
G with k nodes.
The representation matrices of ˜ G are denoted as ˜ X ∈ R k × ˜ c and ˜ A ∈ { 0 , 1 } k × k .
The goal of graph pooling is to learn relationships between X , A and ˜ X , ˜ A .
In this work, we consider graph pooling via node clustering.
In particular, the nodes of the original graph G are assigned to k different clusters.
Then each cluster is transformed to a new node in the new graph ˜
G .
The clustering assignments can be represented as an assignment matrix M ∈ R n × k .
For hard assignments, m i,j ∈ { 0 , 1 } denotes if node i in graph G belongs to cluster j .
For soft assignments, m i,j ∈ [0 , 1] denotes the probability that node i in graph G belongs to cluster j and (cid:80)
j m i,j = 1 .
Then the new graph ˜ G can be computed as
[Equation]
where g ( · ) is a function that g (˜ a i,j ) =
if ˜ a i,j > 0 and g (˜ a i,j ) = 0 otherwise.
3. L EARNING C LUSTERING A SSIGNMENTS VIA C ONDITIONAL R ANDOM F IELDS
Intuitively, node features describe the properties of different nodes.
Then nodes with similar features should have a higher chance to be assigned to the same cluster.
That is, for any node in the original graph G , its cluster assignment should not only depend on node feature matrix X but also condition on the cluster assignments of the other nodes.
We believe such high-order structural information is useful for graph pooling and should be explicitly captured while learning clustering assignments.
To this end, we propose a novel structured graph pooling technique, known as S TRUCT P OOL , which generates the assignment matrix by considering the feature matrix X and the relationships between the assignments of different nodes.
We propose to formulate this as a conditional random ﬁeld (CRF) problem.
The CRFs model a set of random variables with a Markov Random Field (MRF), conditioned on a global observation (Lafferty et al., 2001).
We formally deﬁne Y =
{ Y 1 , · · · , Y n } as a random ﬁeld where Y i ∈ { 1 , · · · , k } is a random variable.
Each Y i indicates to which cluster the node i is assigned.
Here the feature representation X is treated as global observation.
We build a graphical model on Y , which is deﬁned as G (cid:48) .
Then the pair ( Y, X ) can be deﬁned as a CRF, characterized by the Gibbs distribution as
[Equation]
where c denotes a clique, C G (cid:48) is a set of cliques in G (cid:48) , Z ( X ) is the partition function, and ψ c ( · ) is a potential function induced by c (Kr¨ahenb¨uhl & Koltun, 2011; Lafferty et al., 2001).
Then the Gibbs
[Figure] [Figure]
The unary energy matrix can be obtained by multiple GCN layers using X and A .
The pairwise energy is measured by attention matrix using node feature X and topology information A .
Then by performing iterative updating, the mean ﬁeld approximation yields the most probable assignment matrix.
Finally, we obtain the new graph with 4 nodes, represented by ˜ X and ˜ A .
energy function for an assignment y
= { y 1 , · · · , y n } for all variables can be written as
[Equation]
Finding the optimal assignment is equivalent to maximizing P ( Y | X ) , which can also be interpreted as minimizing the Gibbs energy.
3. G IBBS E NERGY WITH T OPOLOGY I NFORMATION
Now we deﬁne the clique set C G (cid:48) in G (cid:48) .
Similar to the existing CRF model (Kr¨ahenb¨uhl & Koltun, 2011), we include all unary cliques in C G (cid:48) since we need to measure the energy for assigning each node.
For pairwise cliques, we generalize our method to control the pairwise clique set by incorporating the graph topological information A .
We consider (cid:96) -hop
connectivity based on A to deﬁne the pairwise cliques, which builds pairwise relationships between different nodes.
Let
A (cid:96) ∈ { 0 , 1 } n × n represent the (cid:96) -hop connectivity of graph G where a (cid:96)i,j = 1 indicates node i and node j are reachable in G within (cid:96) hops.
Then we include all pairwise cliques ( i, j ) in C G (cid:48)
if a (cid:96)i,j = 1 .
Altogether, the Gibbs energy for a cluster assignment y can be written as
[Equation]
where ψ u ( y i ) represents the unary energy for node i to be assigned to cluster y i .
In addition, ψ p ( y i , y j ) is the pairwise energy, which indicates the energy of assigning node i, j to cluster y i , y j respectively.
Note that we drop the condition information in Equation (6) for simplicity.
If (cid:96) is large enough, our CRF is equivalent to the dense CRFs.
If (cid:96) is equal to 1, we have A (cid:96) =
A so that only 1-hop information in the adjacent matrix is considered.
These two types of energy can be obtained directly by neural networks (Zheng et al., 2015).
Given the global observations X and the topology information A , we employ multiple graph convolution layers to obtain the unary energy Ψ u ∈ R n × k .
Existing work on image tasks (Kr¨ahenb¨uhl & Koltun, 2011) proposes to employ Gaussian kernels to measure the pairwise energy.
However, due to computational inefﬁciency, we cannot directly apply it to our CRF model.
The pairwise energy proposed in (Kr¨ahenb¨uhl & Koltun, 2011) can be written as
[Equation]
where k ( m ) ( · , · ) represents the m th Gaussian kernel, x i is the feature vector for node i in X , w ( m ) denotes learnable weights, and µ ( y i , y j ) is a compatibility function that models the compatibility between different assignment pairs.
However, it is computationally inefﬁcient to accurately compute the outputs of Gaussian kernels, especially for graph data when the feature vectors are highdimensional.
Hence, in this work, we propose to employ the attention matrix as the measurement of pairwise energy.
Intuitively, Gaussian kernels indicate how strongly different feature vectors are connected with each other.
Similarly, the attention matrix reﬂects similarities between different feature vectors but with a signiﬁcantly less computational cost.
Speciﬁcally, each feature vector x
i is attended to any other feature vector x j if the pair ( i, j ) is existing in clique set C G (cid:48) .
Hence, the pairwise energy can be obtained by
[Equation]
It can be efﬁciently computed by matrix multiplication and normalization.
Minimizing the Gibbs energy in Equation (6) results in the most probable cluster assignments for a given graph G .
However, such minimization is intractable, and hence a mean ﬁeld approximation is proposed (Kr ¨ ahenb ¨ uhl &
Koltun, 2011), which is an iterative updating algorithm.
We follow the mean-ﬁeld approximation to obtain the most probable cluster assignments.
Altogether, the steps of our proposed S TRUCTP OOL are shown in Algorithm 1.
All operations in our proposed S TRUCT P OOL can be implemented as GNN operations, and hence the S TRUCT P OOL can be employed in any deep graph model and trained in an end-to-end fashion.
The unary energy matrix can be obtained by stacking several
GCN layers, and the normalization operations (step 3&9 in Algorithm 1) are equivalent to softmax operations.
All other steps can be computed by matrix computations.
It is noteworthy that the compatibility function µ ( y i , y j ) can be implemented as a trainable matrix N ∈ R k × k , and automatically learned during training.
Hence, no prior domain knowledge is required for designing the compatibility function.
We illustrate our proposed S TRUCT P OOL in Figure 1 where we perform S TRUCT P OOL on a graph G with 6 nodes, and obtain a new graph ˜ G with 4 nodes.
3. C OMPUTATIONAL C OMPLEXITY A NALYSIS
We theoretically analyze the computational efﬁciency of our proposed S TRUCT P OOL .
Since computational efﬁciency is especially important for large-scale graph datasets, we assume that n > k, c, ˜ c .
The computational complexity of one GCN layer is O ( n 3 + n 2 c + nc ˜ c )
≈ O ( n 3 ) .
Assuming we employ i layers of GCNs to obtain the unary energy, its computational cost is
O ( in 3 ) .
Assuming there are m iterations in our updating algorithm, the computational complexity is O ( m ( n 2 c + n 2 k + nk 2 ))
≈ O ( mn 3 ) .
The ﬁnal step for computing ˜ A and ˜ X takes
O ( nkc + n 2 k
+ nk 2 )
≈ O ( n 3 ) computational complexity.
Altogether, the complexity S TRUCTP OOL is O (( m + i ) n 3 ) , which is close to the complexity of stacking m + i layers of GCNs. [Table] [Table]
D EEP G RAPH N ETWORKS FOR G RAPH C LASSIFICATION
In this section
[Table]
we investigate graph classiﬁcation tasks which require both good node-level and graph-level representations.
For most state-of-the-art deep graph classiﬁcation models, they share a similar pipeline that ﬁrst produces node representations using GNNs, then performs pooling operations to obtain high-level representations, and ﬁnally employs fully-connected layers to perform classiﬁcation.
Note that the high-level representations can be either a vector or a group of k vectors.
For a set of graphs with different node numbers, with a pre-deﬁned k , our proposed S TRUCT P OOL can produce k vectors for each graphs.
Hence, our method can be easily generalized and coupled to any deep graph classiﬁcation model.
Specially, our model for graph classiﬁcation is developed based on DGCNN (Zhang et al., 2018).
Given any input graph, our model ﬁrst employs several layers of GCNs (Equation (2)) to aggregate features from neighbors and learn representations for nodes.
Next, we perform one S TRUCT P OOL layer to obtain k vectors for each graph.
Finally, 1D convolutional layers and fully-connected layers are used to classify the graph.
E XPERIMENTAL S TUDIES 4. D ATASETS AND E XPERIMENTAL S ETTINGS
We evaluate our proposed S TRUCT P OOL on eight benchmark datasets, including ﬁve bioinformatics protein datasets: ENZYMES, PTC, MUTAG, PROTEINS (Borgwardt et al., 2005), D&D (Dobson & Doig, 2003), and three social network datasets: COLLAB (Yanardag & Vishwanathan, 2015b),
IMDB-B, IMDB-M (Yanardag & Vishwanathan, 2015a).
Most of them are relatively large-scale and hence suitable for evaluating deep graph models.
We report the statistics and properties of them in
Supplementary Table 6.
Please see the Supplementary Section A for experimental settings.
We compare our method with several state-of-the-art deep GNN methods.
PATCHYSAN (Niepert et al., 2016) learns node representations and a canonical node ordering to perform classiﬁcation.
DCNN (Atwood & Towsley, 2016) learns multi-scale substructure features by diffusion graph convolutions and performs global sum pooling.
DGK (Yanardag & Vishwanathan, 2015a) models latent representations for sub-structures in graphs, which is similar to learn word embeddings.
ECC (Simonovsky & Komodakis, 2017) performs GCNs conditioning on both node features and edge information and uses global sum pooling before the ﬁnal classiﬁer.
GRAPHSAGE (Hamilton et al., 2017) is an inductive framework which generates node embeddings by sampling and aggregating features from local neighbors, and it employs global mean pooling.
SET2SET (Vinyals et al., 2015) proposes an aggregation method to replace the global pooling operations in deep graph networks.
DGCNN (Zhang et al., 2018) proposes a pooling strategy named SORTPOOL which sorts all nodes [Table] by learning and selects the ﬁrst k nodes to form a new graph.
DIFFPOOL (Ying et al., 2018) is built based on GRAPHSAGE architecture but with their proposed differentiable pooling.
Note that for most of these methods, pooling operations are employed to obtain graph-level representations before the ﬁnal classiﬁer.
In addition, we compare our S TRUCT P OOL with three graph kernels:
Graphlet (Shervashidze et al., 2009), Shortest-path (Borgwardt & Kriegel, 2005), and WeisfeilerLehman subtree kernel (WL) (Weisfeiler & Lehman, 1968).
4. C LASSIFICATION R ESULTS
We evaluate our proposed method on six benchmark datasets and compare with several state-of-theart approaches.
The results are reported in Table 1 where the best results are shown in bold and the second best results are shown with underlines.
For our S TRUCT P OOL , we perform 10-fold cross validations and report the average accuracy for each dataset.
The 10-fold splitting is the same as DGCNN.
For all comparing methods, the results are taken from existing work (Ying et al., 2018; Zhang et al., 2018).
We can observe that our S TRUCT P OOL obtains the best performance on 5 out of 6 benchmark datasets.
For these 5 datasets, the classiﬁcation results of our method are signiﬁcantly better than all comparing methods, including advanced models DGCNN and DIFFPOOL.
Notably, our model outperforms the second-best performance by an average of 3.58% on these 5 datasets.
In addition, the graph kernel method WL obtains the best performance on COLLAB dataset and none of these deep models can achieve similar performance.
Our model can obtain competitive performance compared with the second best model.
This is because many graphs in COLLAB only have simple structures and deep models may be too complex to capture them.
4. C OMPARISONS OF D IFFERENT P OOLING M ETHODS
To demonstrate the effectiveness of our proposed pooling technique, we compare different pooling techniques under the same network framework.
Speciﬁcally, we compare our S TRUCT P OOL with the global sum pool, SORTPOOL, TOPKPOOL, DIFFPOOL, and SAGPOOL.
All pooling methods are employed in the network framework introduced in Section 3.5.
In addition, the same 10-fold cross validations from DGCNN are used for all pooling methods.
We report the results in Tableand the best results are shown in bold.
Obviously, our method achieves the best performance on ﬁve of six datasets, and signiﬁcantly outperforms all comparing pooling techniques.
For the dataset ENZYMES, our obtained result is competitive since SAGPOOL only slightly outperforms our proposed method by 0.34%.
Such observations demonstrate the structural information in graphs is useful for graph pooling and the relationships between different nodes should be explicitly modeled.
4. S TUDY OF C OMPUTATIONAL C OMPLEXITY
Dataset m =m =m =m =
E NZYMES 62.
D&D 82.
P
[Table]
As mentioned in Section 3.4, our proposed S TRUCT P OOL yields O (( m + i ) n 3 ) computational complexity.
The complexity of DIFFPOOL is O ( jn 3 ) if we assume it employs j layers of GCNs to obtain the assignment matrix.
In our experiments, i is usually set to 2 or 3 which is much smaller than n .
We conduct experiments to show how different iteration number m affects the prediction accuracy and the results are reported in Table 3.
Note that we employ the dense CRF form for all different m .
We can observe that the performance generally increases with m increasing, especially for large-scale dataset D&D.
We also observe m = 5 is a good trade-off between time complexity and prediction performance.
Notably, our method can even outperform other approaches when m = 1 .
Furthermore, we evaluate the running time of our S TRUCT P OOL and compare it with
DIFFPOOL.
For 500 graphs from large-scale dataset D&D, we set i =
j = 3 and show the averaging time cost to perform pooling for each graph.
The time cost for DIFFPOOL is 0.042 second, while our S TRUCT P OOL takes 0.049 second, 0.053 second and 0.058 second for m = 1 , m = 3 , m = 5 respectively.
Even though our S TRUCT P OOL has a relatively higher computational cost, it is still reasonable and acceptable given its superior performance.
4. E FFECTS OF T OPOLOGY I NFORMATION
[Table]
[Table]
Next, we conduct experiments to show how the topology information A (cid:96) affects the prediction performance.
We evaluate our S TRUCT P OOL with different (cid:96) values and report the results in Table 4.
Note that when (cid:96) is large enough, our S TRUCT P OOL considers all pairwise relationships between all nodes, and it is equivalent to the dense CRF.
For the datasets IMDB-M and PROTEINS, we can observe that the prediction accuracies are generally increasing with the increasing of (cid:96) .
With the increasing of (cid:96) , more pairwise relationships are considered by the model, and hence it is reasonable to obtain better performance.
In addition, for the dataset IMDB-B, the results remain similar with different (cid:96) , and even (cid:96) =
yields competitive performance with dense CRF.
It is possible that 1-hop pairwise relationships are enough to learn good embeddings for such graph types.
Overall, dense CRF consistently produces promising results and is a proper choice in practice.
4. G RAPH I SOMORPHISM N ETWORKS WITH S TRUCT P OOL
[Table]
Recently,
Graph
Isomorphism
Networks (GINs) are proposed and shown to be more powerful than traditional GNNs (Xu et al., 2019).
To demonstrate the effectiveness of our S TRUCT P OOL and show its generalizability, we build models based on GINs and evaluate their performance.
Speciﬁcally, we employ GINs to learn node representations and perform one layer of the dense form of our S TRUCT P OOL , followed by 1D convolutional layers and fully-connected layers as the classiﬁer.
The results are reported in the Table 5, where we employ the same 10-fold splitting as GINs (Xu et al., 2019) and the GIN results are taken from its released results.
These ﬁve datasets include both bioinformatic data and social media data, and both small-scale data and large-scale data.
Obviously, incorporating our proposed S TRUCT P OOL in
GINs consistently and signiﬁcantly improves the prediction performance.
It leads to an average of 4.52% prediction accuracy improvement, which is promising.
C ONCLUSIONS
Graph pooling is an appealing way to learn good graph-level representations, and several advaned pooling techiques are proposed.
However, none of existing graph pooling techniques explicitly considers the relationship between different nodes.
We propose a novel graph pooling technique, known as S TRUCT P OOL , which is developed based on the conditional random ﬁelds.
We consider the graph pooling as a node clustering problem and employ the CRF to build relationships between the assignments of different nodes.
In addition, we generalize our method by incorporating the graph topological information so that our method can control the pairwise clique set in our CRFs.
Finally, we evaluate our proposed S TRUCT P OOL on several benchmark datasets and our method can achieve new state-of-the-art results on ﬁve out of six datasets. A CKNOWLEDGEMENT
This work was supported in part by National Science Foundation grants DBI-1661289 and IIS1908198.
A A PPENDIX A. D ATASETS AND E XPERIMENTAL SETTINGS
[Table]
We report the statistics and properties of eight benchmark datasets in Supplementary Table 6.
For our S TRUCT P OOL , we implement our models using Pytorch (Paszke et al., 2017) and conduct experiments on one GeForce GTX 1080 Ti GPU.
The model is trained using Stochastic gradient descent (SGD) with the ADAM optimizer (Kingma & Ba, 2014).
For the models built on DGCNN (Zhang et al., 2018) in Section 4.2, 4.3, 4.4, 4.5, we employ GCNs to obtain the node features and the unary energy matrix.
All experiments in these sections perform 10-fold cross validations and we report the averaging results.
The 10-fold splitting is exactly the same as DGCNN (Zhang et al., 2018).
For the non-linear function, we employ tanh for GCNs and relu for 1D convolution layers.
For the models built on GINs in Section 4.6, we employ GINs to learn node features and unary energy.
Here the 10fold splitting is exactly the same as GINs.
We employ relu for all layers as the non-linear function.
For all models, 1D convolutional layers and fully-connected layers are used after our S TRUCT P OOL .
Hard clustering assignments are employed in all experiments.
A. E FFECTS OF P AIRWISE E NERGY
[Table]
We conduct experiments to show the importance of the pairwise energy.
If the pairwise energy is removed, the relations between different node assignments are not explicitly considered.
Then the method is similar to the DIFFPOOL.
We compare our method with such a baseline that removes the pairwise energy.
Experimental results are reported in Table 7.
The network framework is the same as introduced in Section 3.5 and the same 10-fold cross validations from DGCNN are used.
Obviously, our proposed method consistently and signiﬁcantly outperforms the baseline which excludes pairwise energy.
It indicates the importance and effectiveness of incorporating pairwise energy and considering high-order relationships between different node assignments.
A. S TUDY OF H IERARCHICAL N ETWORK S TRUCTURE
To demonstrate how the network depth and multiple pooling layers affects the prediction performance, we conduct experiments to evaluate different hierarchical network structures.
We ﬁrst deﬁne a network block contains two GCN layers and one S TRUCT P OOL layer.
Then we compare three [Table] different network settings: 1 block with the ﬁnal classiﬁer, 2 blocks with the ﬁnal classiﬁer, and 3 blocks with the ﬁnal classiﬁer.
The results are reported in Table 8.
For the dataset Proteins, we observe that the network with one block can obtain better performance than deeper networks.
We believe the main reason is dataset Proteins is a small-scale dataset with an average number of nodes equal to 39.06.
A relatively simpler network is powerful enough to learn its data distribution while stacking multiple GCN layers and pooling layers may lead to a serious overﬁtting problems.
For the dataset D&D, the network with 2 blocks performs better than the one with 1 block.
Since D&D is relatively large scale, stacking 2 blocks increases the power of network and hence increases the performance.
However, going very deep, e.g., stacking 3 blocks, will cause the overﬁtting problem.
A. S TUDY OF G RAPH P OOLING R ATE
[Table]
We follow the DGCNN (Zhang et al., 2018) to select the number of clusters k .
Speciﬁcally, we use a pooling rate r ∈ (0 , 1) to control k .
Then k is set to an integer so that r × 100% of graphs have nodes less than this integer in the current dataset.
As suggested in DGCNN, generally, r = 0 .is
a proper choice for bioinformatics datasets and r = 0 .
6 is good for social network datasets.
In addition, we conduct experiments to show the performance with the respect to different r values.
We set r = 0 .
9 to evaluate the performance on a large-scale social network dataset
D&D.
The average number of nodes in dataset D&D is 284.32 and the maximum number of nodes is 5748.
The results are reported in Table 9 where the ﬁrst row shows different pooling rates, the second row reports the corresponding k values and the ﬁnal row shows the results.
For simplicity, we employ the network structure with 1 block and a ﬁnal classiﬁer (as deﬁned in Section A.3).
We can observe that the performance drops when r, k is relatively large or small.
In addition, the model can obtain competitive performance when r is set to a proper range, for example, r ∈ [0 . 3 , 0 .
7] for dataset D&D.