H AAR P OOLING
: G RAPH P OOLING WITH C OMPRES
SIVE H AAR B ASIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep Graph Neural Networks (GNNs) are instrumental in graph classiﬁcation and graph-based regression tasks.
In these tasks, graph pooling is a critical ingredient by which GNNs adapt to input graphs of varying size and structure.
We propose a new graph pooling operation based on compressive Haar transforms, called HaarPooling .
HaarPooling is computed by following a chain of sequential clusterings of the input graph.
The input of each pooling layer is transformed by the compressive Haar basis of the corresponding clustering.
HaarPooling operates in the frequency domain by the synthesis of nodes in the same cluster and ﬁlters out ﬁne detail information by compressive Haar transforms.
Such transforms provide an effective characterization of the data and preserve the structure information of the input graph.
By the sparsity of the Haar basis, the computation of HaarPooling is of linear complexity.
The GNN with HaarPooling and existing graph convolution layers achieves state-of-the-art performance on diverse graph classiﬁcation problems.
I NTRODUCTION
Graph Neural Networks (GNNs) have demonstrated excellent performance in node classiﬁcation tasks and are very promising in graph classiﬁcation and regression (Bronstein et al., 2017; Battaglia et al., 2018;
Zhang et al., 2018b; Zhou et al., 2018; Wu et al., 2019).
In node classiﬁcation, the input is a single graph with missing node labels that are to be predicted from the known node labels.
In this problem, GNNs with appropriate graph convolutions can be trained based on the single graph that is provided, and achieve state-of-the-art performance (Defferrard et al., 2016; Kipf & Welling, 2017; Ma et al., 2019b).
Different from node classiﬁcation, graph classiﬁcation is a task where the label of any given graph-structured sample is to be predicted based on a training set of labeled graph-structured samples.
This is similar to the image classiﬁcation task tackled by traditional deep convolutional neural networks.
The major difference is that here each input sample may have an arbitrary adjacency structure, instead of the ﬁxed regular grids that are used in images.
An example of graph-structured data are the molecules of different sizes shown in Figure 1.
This raises two important challenges:
1) How can GNNs exploit the graph structure information of the input data? 2)
How can GNNs handle input graphs with varying number of nodes and connectivity structures?
These problems have motivated the design of proper graph convolution and graph pooling to allow
GNNs to capture the geometric information of each data sample (Zhang et al., 2018a;
Ying et al., 2018;
Cangea et al., 2018; Gao & Ji, 2019; Knyazev et al., 2019; Ma et al., 2019a; Lee et al., 2019).
Graph convolution plays an important role especially in question 1).
The following graph convolution, as proposed by Kipf & Welling (2017), is a widely accepted example:
[Equation]
Here (cid:98)
[Equation]
D − 1 / 2 ∈ R N × N is a normalized version of the adjacency matrix A of the input graph, where I is the identity matrix and (cid:101) D is the degree diagonal matrix for A
+ I .
Further,
X in ∈ R N × d is the array of d -dimensional features on the N nodes of the graph, and W ∈ R d
× m is the ﬁlter parameter matrix.
The graph convolution in equation 1 captures the structural information of the input in terms of A (or (cid:98) A ), and W transforms the feature dimension from d to m .
The ﬁlter size d × m does not depend on the graph size, which allows a ﬁxed network architecture to process input graphs of varying size.
However, the GCN convolution preserves the number of nodes and hence the output dimension of the network is not unique.
Graph pooling provides an effective way to overcome this obstacle.
Among several approaches that have been proposed, only
EigenPooling (Ma et al., 2019a) incorporates both features and graph structure.
However, this is based on eigenpairs of graph Laplacian and suffers from a high computational cost.
We provide an overview of this and other graph pooling methods in Section 2.
In this paper, we propose a new graph pooling strategy based on a sparse Haar representation of the data, which we call HaarPooling .
This is based on the Haar basis (Li et al., 2019), which incorporates graph structure and features, and is computationally efﬁcient.
Suppose we have an input X in ∈ R N × d .
HaarPooling is then deﬁned as
[Equation]
where Φ ∈ R N × N 1 , N 1 < N .
Each column of Φ is a compressive Haar basis vector.
By applying
HaarPooling in equation 2, the number of nodes is compressed from N to N 1 .
The Haar basis provides a sparse representation which distills graph structural information.
Cascading pooling layers we can obtain an output of a ﬁxed dimension, regardless of the size of the inputs.
The sparsity of the Haar basis matrix ensures that the computation of HaarPooling is efﬁcient.
Generating the
Haar basis and computing the Haar transform has a computational cost O ( N ) (up to a log term of N ) for input graphs with N nodes.
Experiments in Section 5 demonstrate that GNNs with HaarPooling achieve state-of-the-art performance on various graph classiﬁcation tasks.
This paper is organized as follows.
Section 2 gives an overview of existing work on graph pooling.
Section 3 details the components and computational ﬂow for HaarPooling.
Section 4 provides the mathematical details on HaarPooling, including the compressive Haar basis, compressive Haar transforms, and efﬁcient implementations.
Section 5 reports our experimental results on benchmark graph classiﬁcation tasks compared with existing graph pooling methods.
Section 6 concludes the paper.
Proofs and implementation details are deferred to the appendix.
R ELATED W ORK
Graph pooling is a necessary step when building a GNN model for graph classiﬁcation, as one needs a uniﬁed graph-level rather than node-level representation for graph inputs for which size and topology are changing.
The most direct pooling method takes the global mean and sum of node representations obtained by the graph convolutional layer (Duvenaud et al., 2015) as a simple graph-level representation.
However, this pooling operation treats all the nodes equally and ignores the global geometry of the graph.
ChebNet
(Defferrard et al., 2016) uses a graph coarsening procedure to build the pooling module, which requires graph clustering algorithms to obtain subgraphs.
One drawback of this topology-based strategy is that it does not incorporate the node features in the pooling.
Global pooling methods consider the information of node embeddings to obtain the entire graph representation.
As a general framework for graph classiﬁcation problems, MPNN (Gilmer et al., 2017) uses the Set2Set method (Vinyals et al., 2015) to obtain graph-level representations.
Zhang et al.
(2018a) proposed a SortPool method that sorts feature representation of nodes before feeding them into traditional 1-D convolutional and dense layers.
But these global pooling techniques cannot guarantee hierarchical graph representations that may contain useful information in the graph structure.
A prominent recent idea is to build a differentiable and data-dependent pooling layer with learnable operations/parameters, which has brought substantial improvements on graph classiﬁcation tasks.
Ying et al.
(2018) proposed a differentiable pooling layer (DiffPool) that learns a cluster assignment matrix over the nodes using the output of a GNN model.
One common problem with DiffPool is its huge storage complexity, which results from the computation of the soft clustering assignments.
Cangea et al.
; Gao & Ji (2019); Knyazev et al.
(2019) used a Top-K pooling method that samples a subset of important nodes by employing a trainable projection vector.
Lee et al.
(2019) introduced Self-Attention Graph Pooling (SAGPool) by replacing the way node scores are computed in Top-K pooling by a GCN module.
These hierarchical pooling methods technically still employ mean/max pooling procedures to aggregate the feature representation of super-nodes, which would lead to information loss.
Diehl et al.
(2019) proposed EdgePool which is a scheme which considers edge contraction and thus takes into account of graph structure in pooling.
There are also spectral-based pooling methods that take account of both the graph structure and its node features.
Noutahi et al.
(2019) proposed the Laplacian Pooling (LaPool) method that dynamically selects centroid nodes and their corresponding follower nodes by an attention mechanism that uses the graph Laplacian.
Ma et al. (2019a) introduced EigenPool which uses local graph Fourier transform to extract subgraph information utilizing both node features and structure of the subgraph.
Its potential drawback lies in the inherent bottleneck of computing Laplacian-based graph Fourier transform, given the huge cost in the eigendecomposition of the graph Laplacian.
This shortcoming partially motivates our present work.
H AAR P OOLING
In this section we give an overview of the proposed HaarPooling framework.
First we deﬁne the pooling architecture in terms of a chain, i.e., a sequence of graphs ( G 0 , G 1 , . . .
, G K ) , where the nodes of each G j +1 correspond to clusters of nodes of G j +1 , j = 0 , . . .
, K − 1 .
Each layer in the chain determines which sets of nodes are pooled together.
Then we construct the compressive Haar transform, which compresses the dimension of the features.
Chain of coarse-grained graphs for pooling Graph pooling amounts to deﬁning a sequence of coarse-grained graphs.
In our chain, each graph is an induced graph that arises from grouping (clustering) certain subsets of nodes from the previous graph.
We use clustering algorithms to generate the groupings of nodes.
There are many good candidates, such as spectral clustering (Shi & Malik, 2000), k -means clustering (Pakhira, 2014), DBSCAN (Ester et al., 1996), OPTICS (Ankerst et al., 1999) and METIS (Karypis & Kumar, 1998).
Any of these will work with HaarPooling.
Figure 3 shows an example of a chain with 3 levels, for an input graph G 0 .
Compressive Haar transforms on chain For each layer of the chain, we will have a feature representation.
We deﬁne these in terms of the Haar basis.
Haar basis represents graph-structured data by low and high frequency Haar coefﬁcients in frequency domain.
The low frequency coefﬁcients con- a b c d e f g h
Ga b c d e f g h Ga b c d e f g h
G tain the coarse information of the original data while the high frequency coefﬁcients contain the ﬁne details.
In the HaarPooing, the data is pooled (or compressed) by discarding ﬁne detail information.
The Haar basis can be compressed in each layer.
Consider a chain where at level j the two subsequent graphs have N j +
and N j nodes, N j +
< N j .
For each of these graphs, we can create a Haar basis with N j +1 and N j elements, respectively.
The elements of the smaller layer are obtained by compressing a subset of the elements from the other layer.
These new vectors form the matrix Φ j of size N j +1 × N j .
We call Φ j compressive Haar basis matrix for this particular j th layer.
This then deﬁnes the compressive Haar transform Φ Tj X in for feature X in with size N j × d .
Computational strategy of HaarPooling The HaarPooling is then deﬁned as follows.
Deﬁnition 1 (HaarPooling) .
The HaarPooling for a graph neural network with K pooling layers is deﬁned as
[Equation]
where N j > N j +
and N K = 1 , Φ j or Φ ( j ) N j × N j +1 is the N j × N j +1 compressive Haar basis matrix for the j th layer, X in j ∈ R N j
×
d j is the input feature array, and X out j ∈ R N j +
×
d j is the output feature array.
The corresponding layer is called HaarPooling layer.
HaarPooling has following key properties.
• The HaarPooling reduces layer by layer the ﬁrst dimension of input feature.
In the last pooling layer, the output feature is compressed as a vector with length d K − 1 and each original input sample would generate such a vector with the same length.
This then makes it possible to deal with input graph-structured data with different size and structure.
• The HaarPooling uses the sparse Haar representation on chain structure.
In each HaarPooling layer, the representation then combines the features of input X in j with the geometric information of the graphs of the j th and ( j + 1) th layers of the chain.
• By the property of Haar basis, the HaarPooling only drops the high frequency (or detailed) information of the input data.
The X out j has good approximation to X in j .
Thus, the major data information (i.e. the low frequency coefﬁcients) is preserved in the pooling, and the loss of the information is small.
• Since the Haar basis matrix is very sparse, HaarPooling can be computed very fast, with near linear computational complexity.
Example Figure 4 shows the computational details of the HaarPooling associated with the chain from Figure 3.
There are two HaarPooling layers.
In the ﬁrst layer, the input X in1 with size 8 × dis transformed by the compressive Haar basis matrix Φ (0)8 × 3 which consists of the ﬁrst three column vectors of the full Haar basis Φ (0)8 × 8 in (a), and output is a 3 × d 1 matrix X out1 .
In the second layer, the input X in2 with size 3 × d 2 (usually X out1 followed by convolution) is transformed by the compressive Haar matrix Φ (1)3 × 1 which is the ﬁrst column vector of the full Haar basis matrix Φ (1)3 ×in (b).
By the construction of the Haar basis in relation to the chain (details in Appendix B), each of the ﬁrst three column vectors φ (0)1 , φ (0)2 and φ (0)3 of Φ (0)8 × 3 has only up to three different values.
This bound is exactly the number of nodes of G 1 .
For each column φ (0) (cid:96) , all nodes with the same parent take the same value.
Similarly, the 3 × 1 vector φ (1) 1 is constant.
This means that the HaarPooling
[Figure]
synthesizes the node feature by adding the same weight to the nodes that are in the same cluster of the coarser layer, and in this way, pools the feature using the graph clustering information.
M ATHEMATICS AND C OMPUTATION FOR H AAR P OOLING
Chain of graphs by clustering For a graph G =
( V, E, w ) , where V, E, w are the vertices, edges, and weights on edges, a graph G cg :
= ( V cg , E cg , w cg ) is a coarse-grained graph of G if | V cg | ≤ | V | and each node of
G has only one parent node in G cg associated with it.
Each node of G cg is called a cluster of G .
For integers J > 0 , a coarse-grained chain for G is a sequence of graphs G 0 → J :=
G 0 , G 1 , . .
. , G J ) with G 0 =
G and such that G j +1 is a coarse-grained graph of G j
=
( V j , E j , w j ) for each j = 0 , 1 , . . .
, J − 1 , and G J has only one node.
Here, we call the graph G J the top level or the coarsest level and
G 0 the bottom level or the ﬁnest level .
The chain G 0 → J hierarchically coarsens graph
G .
We use the notation J + 1 for the number of layers of the chain to distinguish the number
K of layers for pooling.
The chain for graph G can be created by any clustering method.
For details about graphs and chains, we refer the reader to the examples by Chung & Graham (1997);
Hammond et al.
(2011); Chui et al.
; Wang & Zhuang (2018; 2019).
4. C OMPRESSIVE H AAR TRANSFORMS
Haar basis The construction of Haar basis is rooted in the theory of Haar wavelet basis which was ﬁrst introduced by Haar (1910).
It is a special example of the more general Daubechies wavelets (Daubechies, 1992).
Haar basis is later constructed on graph by Belkin et al.
(2006), and also Chui et al.
; Wang & Zhuang (2018; 2019).
The construction of the Haar basis is based on a chain of the graph.
If the topology of the graph is well reﬂected by the clustering of the chain, then the Haar basis contains the crucial geometric information of the graph.
For a chain G
→ J , on the j th-layer graph G j , j = 0 , . .
, J , there is a Haar orthogonal basis { φ ( j ) (cid:96) } N j (cid:96) =0 deﬁned on
G j , where N j is the size of G j and N j +1 < N j for j = 0 , . . .
, J − 1 .
Suppose two consecutive layers j, j + 1 .
The ﬁrst N j +1 members of φ ( j ) (cid:96) , (cid:96) = 1 , . . . , N j +1 , are deﬁned on the ﬁner layer j + 1 , and can be reduced into the φ ( j +1) (cid:96) , (cid:96) = 1 , . . .
, N j +1 , as follows.
For ﬁrst (cid:96) = 1 , . . .
, N j +1 , φ ( j ) (cid:96) ( v ) =
φ ( j +1) (cid:96) ( Pa G ( v )) /
(cid:112) | Pa G ( v ) | , i.e. the value of the φ ( j ) (cid:96) ( v ) is equal to the scaled φ ( j +1) (cid:96) at the parent Pa G ( v ) of v and the scaled factor is one on square root of the number of nodes in the cluster which v lies in.
It means that φ ( j ) (cid:96) ( v ) for v sharing the parent have the same value.
This property is critical to pooling as φ ( j ) (cid:96) ( v ) can then be treated as weights for the graph G j on which the input feature deﬁned, and the nodes gain the same weight if they are in the same cluster.
On the other hand, the remaining Haar basis vectors φ ( j ) (cid:96) for (cid:96) =
N j +1 +1 , . . .
, N j are constructed to reﬂect the high-frequency information in the Haar wavelet decomposition.
This property is exploited by the compressive Haar basis which then pools the input feature into a lower (ﬁrst) dimension output feature.
The construction and its pseudo-codes for algorithmic implementation of the full Haar basis is detailed in Li et al.
; Wang & Zhuang (2019), which we also attach in the appendix.
Let { φ ( j ) (cid:96) } N j (cid:96) =1 , j = 0 , . . .
, J , be the sequence of Haar bases associated with the layers of chain G
→ J of a graph G .
For j = 0 , . . .
, J , we let the matrix (cid:101)
Φ j = ( φ ( j ) 1 , . . . , φ ( j ) N j ) ∈
R N j × N j and call the matrix (cid:101)
Φ j Haar transform matrix for layer j .
Orthogonality For each level j = 0 , . . .
, J , the sequence { φ ( j ) (cid:96) } N j (cid:96)
=1 , with N j :=
| V j | , is an orthonormal basis for the space l 2 ( G j ) of square-summable sequences on the graph G j , so that ( φ ( j ) (cid:96) )
T φ ( j ) (cid:96) (cid:48) =
δ (cid:96),(cid:96) (cid:48) .
For each j , { φ ( j ) (cid:96) } N j (cid:96) =1 is the Haar basis system for the chain G j → J .
Locality Let G
→ J be a coarse-grained chain for G .
If each parent of level G j , j = 1 , . . .
, J , contains at least two children, the number of different scalar values of the components of a Haar basis vector φ ( j ) (cid:96) , (cid:96) = 1 , . . .
, N j , is bounded by a constant independent of j .
In Figure 4, the Haar basis is created based on the coarse-grained chain G 0 → 2 :=
( G 0 , G 1 , G 2 ) , where
G 0 , G 1 , G 2 are graphs with 8 , 3 , 1 nodes.
The two colorful matrices show two Haar bases for the layers 0 and 1 in the chain
G 0 → 2 .
There are in total 8 vectors of the Haar basis for G 0 each with length 8 , and 3 vectors of the Haar basis for
G 1 each with length 3 .
Haar basis matrix for each level of the chain has up to 3 different values in each column as indicated by colors in each matrix.
For j = 0 , 1 , each node of G j is a cluster of nodes in G j +1 .
Each column of the matrix is a member of the Haar basis on the individual layer of the chain.
The ﬁrst three column vectors of (cid:101) Φ 1 can be reduced as an orthonormal basis of G 1 and the ﬁrst column vector of G 1 can be compressed to the constant basis for G 2 .
This connection ensures that the compressive Haar transform for HaarPooling is feasible and would allow fast algorithms of HaarPooling (see Section 4.2 below).
Adjoint and forward Haar transforms We use adjoint Haar transforms for HaarPooling, which as the sparsity of the Haar basis matrix, the transform is fast implementable.
The adjoint Haar transform for the signal f on G j is deﬁned as
[Equation]
and the forward Haar transform for (coefﬁcients) vector c :=
[Equation]
We call the components of ( (cid:101) Φ j ) T f the Haar (wavelet) coefﬁcients for f .
The adjoint Haar transform represents the signal in Haar wavelet domain by computing the Haar coefﬁcients for graph signal.
Here the adjoint and forward Haar transforms can be extended to a feature data with size N j × d j by replacing the column vector f by the feature array.
Proposition 2.
The adjoint and forward Haar Transforms are invertible in that for j = 0 , . .
, J and vector f on graph G j ,
[Equation]
Proposition 2 shows that the forward Haar transform can recover the graph signal f from the adjoint
Haar transform ( (cid:101) Φ j ) T f .
This means that adjoint and forward Haar transforms have zero-loss in graph signal transmission.
Compressive Haar transforms Now for a graph neural network, suppose we want to use K pooling layers.
We associate the chain G 0 → K of an input graph with the pooling by linking the j th layer of pooling with the j th layer of the chain.
Then, we can use the Haar basis system on the chain to deﬁne the pooling operation.
By the property of Haar basis, in the Haar transforms for layer j , 0 ≤ j ≤ K − 1 , of the N j Haar coefﬁcients, the ﬁrst N j +1 coefﬁcients are low-frequency coefﬁcients, which reﬂect the approximation to the original data, and the other ( N j − N j +1 ) coefﬁcients are in high frequency, which contain ﬁne details of the Haar wavelet decomposition.
To deﬁne pooling, we remove the high-frequency coefﬁcients in Haar wavelet representation and obtain the compressive
Haar transforms for the feature X in j at layers j = 0 , . .
, K − 1 , which then gives the HaarPooling in Deﬁnition 1.
As shown in the following formula, the compressive Haar transform synthesizes the neighbourhood information of the signal f as compared to the full Haar transform.
Thus, HaarPooling takes the average information of the data f over nodes in the same cluster.
[Equation]
2 = (cid:88) p ∈G j + (cid:88) p =
Pa ( v ) (cid:12)(cid:12)(cid:12) X in j ( v ) (cid:12)(cid:12)(cid:12) 2 , (5) where (cid:101) Φ j is the full Haar basis matrix at the j th layer and | Pa G ( v ) | means the number of nodes in the cluster which v lies in.
Here, in the ﬁrst equation, 1 / (cid:112) | Pa G ( v ) | can be taken out of summation as Pa ( v ) is in fact a set of nodes.
We show the derivation of formula in equation 5 in Appendix D.
In HaarPooling, the compression or pooling occurs in the Haar wavelet domain.
HaarPooling transforms the features on the nodes to the Haar wavelet domain and discards the high-frequency coefﬁcients in the sparse Haar wavelet representation.
Figure 4 shows a two-layer HaarPooling strategy.
The ﬁrst layer pools the input X
in0 by the compressive Haar basis matrix Φ (0)8 × 3 to the output X out with lower ﬁrst dimension.
The second layer pools the input X in1 by the Φ (1)3 × 1 to the output X out which ﬁrst dimension drops to one.
4. F AST C OMPUTATION OF H AAR P OOLING
For the HaarPooling introduced in Deﬁnition 1, we can develop a fast computational strategy by virtue of fast adjoint Haar transforms.
Let G
→ K be a coarse-grained chain of the graph G 0 .
For convenience, we label the vertices of the level- j graph G j by V j :=
(cid:8) v ( j ) 1 , . .
, v ( j ) N j (cid:9) .
Fast algorithm for HaarPooling The HaarPooling in equation 3 can be computed fast by using the hierarchical structure of the chain, as we introduce as follows.
For j = 1 , . . .
, K , let c ( j ) k be the number of children of v ( j ) k , i.e. the number of vertices of G j − 1 which belongs to the cluster v ( j ) k , for k = 1 , . . . , N j .
For j = 0 , we let c (0)
k
≡ 1 for k = 1 , . . .
, N 0 .
Now, for j = 0 , . .
, K and k = 1 , . .
. , N j , deﬁne the weight for the node v ( j ) k of layer j by
[Equation]
Let W
→ K := { w ( j ) k | j = 0 , . .
, K, k = 1 , . . . , N j } .
Then, for j = 0 , . .
, K , the weighted chain ( G j → K , W j → K ) becomes a ﬁltration if each parent of the chain G j
→ K has at least two children.
See e.g. (Chui et al., 2015, Deﬁnition 2.3).
Let j = 0 , . .
, K .
For the j th HaarPooling layer, let { φ ( j ) (cid:96) } N j (cid:96) =
1 be the Haar basis for the j th layer, which we also call the Haar basis for the ﬁltration ( G j → K , W j → K ) of a graph G .
For k = 1 , . . .
, N j , we let X ( v ( j ) k ) =
X ( v ( j ) k , · ) ∈ R
d j the feature vector at node v ( j ) k .
We deﬁne the weighted sum for feature X ∈ R N j × d j for d j ≥ 1 by
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
j + v (
[Equation] [Equation]
For each vertex v ( i ) k of G i , the S ( i ) (cid:0) X, v ( i ) k (cid:1) is the weighted sum of the S ( i − 1) (cid:0) X, v ( i − 1)
k (cid:48) (cid:1) at the level i − 1 for those vertices v ( i − 1) k (cid:48) of G i − 1 whose parent is v ( i ) k .
Theorem 3.
For 0 ≤ j ≤ K − 1 , let { φ ( i ) (cid:96) } N i (cid:96)
=1 for i =
j + 1 , . .
, K be the Haar bases for the ﬁltration (
G j → K , W j → K ) at layer i .
Then, the compressive Haar transform for the j th HaarPooling layer can be computed by, for the feature X ∈ R N j × d j and (cid:96) = 1 , . . .
, N j ,
[Equation]
where i is the largest possible number in { j + 1 , .
, K } such that φ ( i ) (cid:96) is the (cid:96) th member of the orthonormal basis { φ ( i ) (cid:96) } N i (cid:96) =
1 for l 2 ( G i ) , v ( i ) k are the vertices of G i and the weights w ( i ) k are given by equation 12.
We give the algorithmic implementation of Theorem 3 in Algorithm 1, which provides a fast algorithm for HaarPooling at each layer.
Computational complexity With increasing graph size, the sparsity of the Haar basis matrix (cid:101) Φ j becomes more pronounced (Li et al., 2019).
This sparsity implies fast computation for HaarPooling.
The computational complexity of HaarPooling is determined by the adjoint Haar transforms.
In the ﬁrst step of Algorithm 1, the total number of summations for all elements of Step 1 is no more than (cid:80) j −i =
N i +1 ;
In the second step, by the locality of the Haar basis, the total number of multiplication and summation operations is at most 2 (cid:80) N j (cid:96) =1 C
= O ( N j ) .
Here C is the constant which bounds the number of different values of the Haar basis vector.
Thus, the computational cost of Algorithm 1 is
O ( N j ) .
We run an experiment to evaluate the CPU computational time of HaarPooling by Algorithmagainst the direct matrix product.
We use randomly generated graphs and features with size ranging from 2 to 5000 .
As shown in Figure 5, the fast HaarPooling has computational cost nearly proportional to the number of nodes N , while the ordinary matrix product incurs a cost close to order
O ( N 2 ) .
These results are consistent with the computational complexity analysis given above.
[Figure]
for
for input feature ar
ray with up to 5000 nodes.
The cost of HaarPooling has near linear computational complexity.
The cost of direct matrix product grows at O ( N 2 . 1 ) .
E XPERIMENTS
Data sets To verify whether the proposed framework can hierarchically learn good graph representations for classiﬁcation, we evaluate HaarPooling on ﬁve widely used benchmark data sets for graph classiﬁcation (Kersting et al., 2016), including one protein graph data set PROTEINS (Borgwardt et al., 2005; Dobson & Doig, 2003); two mutagen data sets MUTAG (Debnath et al., 1991;
Kriege & Mutzel, 2012) and MUTAGEN (Riesen & Bunke, 2008; Kazius et al., 2005) (full name
Mutagenicity); and two data sets that consist of chemical compounds screened for activity against non-small cell lung cancer and ovarian cancer cell lines, NCI1 and NCI109 (Wale et al., 2008).
We include data sets from different domains, sample and graph sizes to give a comprehensive understanding of how HaarPooling performs with data sets in various scenarios.
A summary information of the data sets is given in Table 1, which shows the data sets containing graphs with different sizes and structures: the number of data samples ranges from 188 to 4,337, the average number of nodes is from 17 .
93 to 39 .
06 and the average number of edges is from 19 .
79 to 72 .
[Table]
Baselines and running environment
We compare HaarPool with SortPool (Zhang et al., 2018a),
DiffPool (Ying et al., 2018), gPool (Gao & Ji, 2019), SAGPool (Lee et al., 2019), EigenPool (Ma et al., 2019a), CSM (Kriege & Mutzel, 2012) and GIN
(Xu et al., 2019) on the above data sets.
The experiments use PyTorch Geometric 1 (Fey & Lenssen, 2019) and were run in Google Cloud using
Nvidia Telsa T4 with 2560 CUDA cores, compute 7.5, 16GB GDDR6 VRAM.
Training procedures In experiments, we use a GNN with at most 3 GCN (Kipf & Welling, 2017) convolutional layers plus one HaarPooling layer, followed by three fully connected layers.
The hyperparameters of the network are adjusted case by case.
We use spectral clustering, which exploits the eigenvalues of the graph Laplacian, to generate a chain with the number of layers given.
Spectral clustering has shown good performance in coarsening a variety of data patterns and can handle isolated nodes.
We use random shufﬂing of the data set, which we split into training, validation, and test sets with proportions 80% , 10% and 10% respectively.
We use the Adam optimizer (Kingma & Ba, 2015), early stopping criterion, patience.
The speciﬁc values are provided in the appendix.
The early stopping criterion was that the validation loss does not improve for 50 epochs, with a maximum of 150 epochs, as suggested by Shchur et al.
Results The classiﬁcation test accuracy is reported in Table 6.
GNNs with HaarPooling have excellent performance on all data sets.
In 4 out of 5 datasets, it achieved top accuracy.
This shows that HaarPooling with appropriate graph convolution, can achieve top performance on a variety of graph classiﬁcation tasks, and in some cases improve state of the art by a few percent points
[Table] [Table]
C ONCLUSION
We introduced a new graph pooling method called HaarPooling.
HaarPooling has a mathematical formalism derived from compressive Haar transforms.
Unlike existing graph pooling methods,
HaarPooling takes into account both the graph structure and also the features over the nodes of the graph-structured input data, to compute a coarsened representation.
As an individual unit, HaarPooling can be applied in conjunction with any type of graph convolution in GNNs.
We show in experiments that HaarPooling reaches state of the art in several benchmark graph classiﬁcation tasks.
Moreover, having only linear computational complexity in the size of the inputs, HaarPooling is a very fast pooling method
A G RAPH C LASSIFICATION
Graph classiﬁcation This task is to categorize graph-structured data into several classes.
The training set consists of M pairs of samples (cid:0) ( x i , G i ) , y i (cid:1) , i = 1 , . . .
, M .
For the i th sample, G i =
( V i , E i , W i ) is a graph with vertex set V i of size | V i | =
N i (also called nodes) and edge set E i with weights W i .
The feature x i ∈ R N i × d is an array of d features per vertex, i.e., an R d -valued function over V i .
The label y
i is an integer from a ﬁnite set indicating which class the input sample ( x i , G i ) lies in.
The number of nodes N i and the graph structure E i ,
W
i usually vary over the different input samples.
Graph neural networks Deep graph neural networks (GNNs) are designed to work with graphstructured inputs of the form ( x i , G i ) described above.
A GNN is typically composed of multiple graph convolution layers, graph pooling layers, and fully connected layers.
A (graph) convolutional layer extracts an array of features from the previous array.
It changes the dimension d of the feature array but does not change the number of nodes N i .
Since the number of nodes of different inputs is variable, the number of nodes of the corresponding outputs is also variable.
This raises new challenges in comparison with traditional image classiﬁcation tasks, where the local structure connecting pixels is always ﬁxed (even if the number of pixels might be variable).
Graph pooling In GNNs, one uses graph pooling to reduce the ﬁrst dimension N of the feature arrays, and more importantly, to obtain outputs of uniform dimension (commonly followed by fully connected layers).
A general architecture uses a cascade of convolutional and pooling layers.
Figure 2 illustrates such an architecture with three blocks of graph convolutional and pooling layers, followed by a multi-layer perceptron (MLP) with three fully connected layers.
In practice, each block can include several convolutional layers but use only one pooling layer at most.
The exact architecture of GNNs with combined convolutional and pooling layers is mainly dependent upon the particular problem and the data set and is designed case by case.
B C ONSTRUCTION OF H AAR BASIS
Construction of Haar basis.
With a chain of the graph, one can generate a Haar basis for l 2 ( G ) following Chui et al.
(2015), see also Gavish et al.
We show the construction of Haar basis on G , as follows.
Step 1.
Let G cg
=
( V cg , E cg , w cg ) be a coarse-grained graph of G =
( V, E, w ) with N cg :=
| V cg | .
Each vertex v cg ∈
V cg is a cluster v cg
= { v ∈ V | v has parent v cg } of G .
Order V cg , e.g., by degrees of vertices or weights of vertices, as V cg
= { v cg1 , . .
. , v cg N cg } .
We deﬁne N cg vectors φ cg (cid:96) on
G cg by
[Equation]
and for (cid:96) = 2 , . . . , N cg ,
[Equation]
where χ cg j is the indicator function for the j th vertex v cg j ∈ V cg on G given by
[Equation]
Then, one can show that { φ cg (cid:96) } N cg (cid:96) =
1 forms an orthonormal basis for l 2 ( G cg ) .
Note that each v ∈ V belongs to exactly one cluster v cg ∈ V cg .
In view of this, for each (cid:96) = 1 , . . .
, N cg , we can extend the vector φ cg (cid:96) on G cg to a vector φ (cid:96), 1 on G by
[Equation]
here | v cg | :
= k (cid:96) is the size of the cluster v cg , i.e., the number of vertices in G whose common parent is v cg .
We order the cluster v cg (cid:96) , e.g., by degrees of vertices, as
[Equation]
For k = 2 , . . .
, k (cid:96) , similar to equation 11, deﬁne
[Equation]
where for j = 1 , . .
k (cid:96) , χ (cid:96),j is given by
[Equation]
One can show that the resulting { φ (cid:96),k : (cid:96) = 1 , . . .
, N cg , k = 1 , . .
k (cid:96) } is an orthonormal basis for l 2 ( G ) .
Step 2.
Let G
→ J be a coarse-grained chain for the graph G .
An orthonormal basis { φ ( J ) (cid:96) } N J (cid:96) =1 for l 2 ( G J ) is generated using equation 10 and equation 11.
We then repeatedly use Step 1: for j = 0 , . .
, J − 1 , we generate an orthonormal basis { φ ( j ) (cid:96) } N j (cid:96) =
1 for l 2 ( G j ) from the orthonormal basis { φ ( j +1) (cid:96) }
N j +1 (cid:96) =
1 for the coarse-grained graph G j +1 that was derived in the previous steps.
We call the sequence { φ (cid:96) :
= φ (0) (cid:96) } N 0 (cid:96) =1 of vectors at the ﬁnest level, the Haar global orthonormal basis or simply the Haar basis for G associated with the chain G 0 → J .
The orthonormal basis { φ ( j ) (cid:96) } N j (cid:96) = for l 2 ( G j ) , j = 1 , . . .
, J is called the associated (orthonormal) basis for the Haar basis { φ (cid:96) } N(cid:96)
=1 .
Besides the orthogonality, the Haar basis has the locality which is critical to the fast computation of
HaarPooling.
Compressive Haar basis Suppose we have constructed the (full) Haar basis { φ ( j ) (cid:96) } N j (cid:96) =0 for each layer G j of the chain G 0 → K .
The compressive Haar basis for layer j is { φ ( j ) (cid:96) } N j +1 (cid:96) =0 .
C F AST C OMPUTATION FOR H AAR P OOLING
Let G
→ K be a coarse-grained chain of the graph G 0 .
For convenience, we label the vertices of the level- j graph G j by V j :=
(cid:8) v ( j ) 1 , . .
, v ( j ) N j (cid:9) .
Fast algorithm for HaarPooling The HaarPooling in equation 3 can be computed fast by using the hierarchical structure of the chain, as we introduce as follows.
For j = 1 , . . .
, K , let c ( j ) k be the number of children of v ( j ) k , i.e. the number of vertices of G j − 1 which belongs to the cluster v ( j ) k , for k = 1 , . . . , N j .
For j = 0 , we let c (0)
k
≡ 1 for k = 1 , . . .
, N 0 .
Now, for j = 0 , . .
, K and k = 1 , . .
. , N j , deﬁne the weight for the node v ( j ) k of layer j by
[Equation] [Equation]
Let W
→ K := { w ( j ) k | j = 0 , . .
, K, k = 1 , . . . , N j } .
Then, for j = 0 , . .
, K , the weighted chain ( G j → K , W j → K ) becomes a ﬁltration if each parent of the chain G j
→ K has at least two children.
See e.g. (Chui et al., 2015, Deﬁnition 2.3).
D P ROOFS
Proof for equation 5.
We only need to prove the ﬁrst formula.
The second is obtained by deﬁnition.
To simplify notation, we let f = X in j .
By construction of Haar basis, for some layer j , the ﬁrst
N j +1 basis vectors
[Equation]
Then, the Fourier coefﬁcient of f for the (cid:96) th basis vector is the inner product
[Equation]
p
[Equation]
j
+
[Equation]
=
[Equation]
(cid:96) (cid:69) where we have let (cid:101) f
( p
[Equation]
=
[Equation]
(cid:112)
[Equation]
(cid:88) p =
[Equation]
( v ) f
( v ) .
This then gives
[Equation]
(cid:69)(cid:12)(cid:12)(cid:12)
[Equation]
Since { φ (cid:96) } N j +1 (cid:96) =1 forms an orthonormal basis on (cid:96) 2 ( G j +1 ) , (cid:13)(cid:13)
Φ Tj f (cid:13)(cid:13) 2 =
N j +1 (cid:88) (cid:96) =
[Equation]
[Equation]
This proves the left formula in equation 5.
Proof of Theorem 3.
By the relation between φ ( i ) (cid:96) and φ ( j ) (cid:96) , for i =
j + 1 , . .
, K and (cid:96)
= 1 , . .
. , N j +1 ,
[Equation]
v ( j +2) k (cid:48)(cid:48) )
w ( j +2) k
(cid:48)(cid:48)
φ ( j +2) (cid:96) ( v ( j +2) k
(cid:48)(cid:48) ) · · · · · · =
N i (cid:88) k
=1 S ( i ) ( X, v ( i ) k ) w ( i ) k
φ ( i ) (cid:96) ( v ( i ) k ) , where v ( j +1) k (cid:48) is the parent of v ( j ) k and v ( j +2) k
(cid:48)(cid:48) is the parent of v ( j +1) k , and we recursively compute the summation to obtain the last equality, thus completing the proof.
E E XPERIMENTAL S ETTING
The architecture of GNN is identiﬁed by the layer type and the number of hidden nodes at each layer.
For example, we denote 3GC256-HP-2FC256-FC128 to represent a GNN architecture with 3 GCNConv layers each with 256 hidden nodes plus one HaarPooling layer followed by 2 fully connected layers each with 256 hidden nodes and 1 fully connected layer with 128 hidden nodes.
The architecture for each data set is shown by Table 3.
The hyperparameters include batch size; learning rate, weight decay rate (these two for optimization); maximum number of epochs; patience for early stopping.
The choice of hyperparameters in each data set is shown in Table 4.
[Table]
-FC
F GNN WITH H AAR P OOLING ON T RIANGLES
We test GNN with HaarPooling on graph data set Triangles (Knyazev et al., 2019).
Triangles is a 10 classiﬁcation problem with 45000 graphs.
The average numbers of nodes and edges of graphs are
[Table]
20 . 85 and 32 .
respectively.
In the experiments, the network uses GIN convolution (Xu et al., 2019) as graph convolution and with HaarPooling or SAGPooling (Lee et al., 2019).
With SAGPooling, the network architecture uses two combined layers of GIN convolution and SAGPooling followed by combined layers of GIN convolution and global max pooling , denoted by GIN-SP-GIN-SP-GIN-MP, where SP means the SAGPooling and MP means global max pooling.
With HaarPooling, we test with two architectures: GIN-HP-GIN-HP-GIN-MP and GIN-HP-GIN-GIN-MP, where HP means
HaarPooling.
The data for training, validation and test are 35000, 5000 and 10000 respectively.
The hidden nodes in convoluational layers is 64, batch size is 60 and learning rate is 0.001.
Table 5 shows the test accuracy of the three networks.
It illustrates that both networks with HaarPooling outperform that with the SAGPooling.
[Table]
G P ROPERTY COMPARISON OF POOLING METHODS
Here we provide a comparison of the properties of HaarPooling with existing pooling methods.
The properties in comparison includes time complexity and space complexity, and whether involving the clustering, hierarchical pooling (which is then not a global pooling), spectral-based, node feature or graph structure and sparse representation.
We compare HaarPooling (denoted by HaarPool in the table) to other methods (SortPool, DiffPool, gPool, SAGPool and EigenPool).
The SortPool (i.e. SortPooling) is a global pooling which uses node signature (i.e. Weisfeiler-Lehman color of vertex) sorts all vertices by the values of the channels of the input data.
Thus, the time complexity (worst case) of SortPool is O ( | V | 2 ) and space complexity is O
( | V | ) .
Other pooling methods are all hierarchical pooling.
DiffPool and gPool both use the node feature and have time complexity
O ( | V | 2 )
The Diff- Pool learns the assignment matrices in end-to-end manner and has space complexity
O ( k | V | 2 ) for pooling ratio k .
The gPool projects all nodes to a learnable vector to generate scores for nodes, and then sorts the nodes by the projection scores; the space complexity is O ( | V | + | E | ) .
SAGPool uses the graph convolution to calculate the attention scores of nodes and then selects top ranked nodes for pooling.
The time complexity of SAGPool is O
( | E | ) and the space complexity is O
( | V | + | E | ) due to the sparsity of the pooling matrix.
EigenPool, which considers both the node feature and graph structure, uses the eigedecomposition of subgraphs (from clustering) of the input graph, and pools the input data by Fourier transforms of the assembled basis matrix.
Due to eigendecomposition, the time complexity of EigenPool is O
( | V | 2 ) and space complexity is O
( | V | 2 ) .
HaarPool which uses the sparse representation of data by compressive Haar basis has linear time complexity
O ( | V | ) (up to a log | V | term), and the space complexity is O
( | V | 2 (cid:15) ) , where (cid:15) is the sparsity of the compressive Haar transform matrix and is usually very small.
From the table, we can observe the HaarPool is the
[Table]
only pooling method which has time complexity proportional to the number of nodes, and thus has faster implementation.