D EEP G RAPH T REE N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose Graph Tree Networks (GTree), a self-interpretive deep graph neural network architecture which originates from the tree representation of the graphs.
In the tree representation, each node forms its own tree where the node itself is the root node and all its neighbors up to hop-k are the subnodes.
Under the tree representation, the message propagates upward from the leaf nodes to the root node naturally and straightforwardly to update the root node’s hidden features.
This message passing (or neighborhood aggregation) scheme is essentially different from that in the vanilla GCN, GAT and many of their derivatives, and is demonstrated experimentally a superior message passing scheme.
Models adopting this scheme has the capability of going deep.
Two scalable graph learning models are proposed within this GTree network architecture - Graph Tree Convolution Network (GTCN) and Graph Tree Attention Network (GTAN), with demonstrated state-of-the-art performances on several benchmark datasets.
The deep capability is also demonstrated for both models.
I NTRODUCTION
Graph Neural Networks (GNNs), a class of neural networks for learning on graph structured data, have been successfully applied in many areas to solve real world problems, such as link predictions in social networks (Fan et al., 2019), pattern recognition
(Ju et al., 2020; Shi & Rajkumar, 2020), product recommendation and personalized search in E-commerce (Zhu et al., 2019), fraud detection (Wang et al., 2019a;
Dou et al., 2020), protein interface predictions (Fout et al., 2017), various tasks in the semiconductor industry (Zhang et al., 2020; Lu et al., 2020), traffic forecasting (Yu et al., 2017), and natural language processing (Yao et al., 2019; Vashishth et al., 2020; Wu et al., 2021).
Among many different graph learning approaches, the class of spatial graph convolution based models, which adopts a message passing (or neighborhood aggregation) scheme to update node features, has gained particular attention due to its simplicity yet good performance.
The most representative work among this class is the Graph Convolutional Network (GCN) (Kipf & Welling, 2017) which limits the ChebNet (Defferrard et al., 2016) up to the 1st order polynomial, and together with some further approximation leading to a direct neighbor aggregation.
The success of the GCN model (Ying et al., 2018; Hu et al., 2020a) has led to the rapid development in spatial convolution based models, such as the Graph Attention Network (GAT) (Velickovic et al., 2018), GraphSage (Hamilton et al., 2017), APPNP (Klicpera et al., 2018), DAGNN (Liu et al., 2020), and etc.
Today the vanilla GCN and GAT are the two most popular baseline models.
One GCN or GAT layer aggregates only the direct neighbor nodes.
Although they work well in many test cases, the performance degrades significantly as the models go deep when larger receptive fields are desirable.
Stacking multiple GCN or GAT layers causes the over-smoothing effect (Xu et al., 2018;
Li et al., 2018; Liu et al., 2020) that nodes from different classes become indistinguishable.
The APPNP (Klicpera et al., 2018) and DAGNN (Liu et al., 2020) were proposed to address this problem and have achieved superior performances on several benchmark datasets.
The DAGNN (Liu et al., 2020) aggregates neighbors from different hops in one layer to utilize information from a larger receptive field as compared to the GCN.
Although it achieves the state-of-the-art performance, such multihop scheme mostly only adapts to GCNs from the model complexity perspective, as the sparse matrix multiplication trick can be applied to complete the computation for the hop-k aggregation ˜
A k ( XW ) =
˜ A ( ˜ A k − 1 XW ) , resulting a time complexity of O ( k | E | ) which is comparable to that of the GCN. Here ˜ A is the normalized adjacency matrix, X is the input feature map, W is the learnable weights, and | E | is the number of edges.
However, adapting such multi-hop scheme to attention-based GNN models (like the GAT, graph transformer (Dwivedi & Bresson, 2020; Hu et al., 2020b)) would be challenging in computation as these models require computations of the attention weight for the pair of each node and its hop-k neighbor node, which could be a too large number.
For the hop-k aggregation, the number of attention weights is the total number of non-zero entries in ˜ A k , which could be non sparse especially for undirected graphs.
For example, in the Cora dataset ˜
A 5 has ∼ 2.2 million non-zero entries as compared to ∼ 10k in ˜ A .
Furthermore, using ˜ A k ( XW ) to represent the aggregation of hop-k neighbors neglects the path information which is usually important for heterogeneous graphs with rich metapath information (Fu et al., 2020; Wang et al., 2019c).
We propose Graph Tree Networks (GTree), a self-interpretive deep graph neural network architecture which originates from the tree representation of the graphs.
In the tree representation, each node forms its own tree where the node itself is the root node and all its neighbors up to hop-k are the subnodes.
Under the tree representation, the message propagates upward from the leaf nodes to the root node naturally and straightforwardly to update the root node’s hidden features.
This message passing (or neighborhood aggregation) scheme is essentially different from that in the vanilla GCN, GAT and many of their derivatives, and is demonstrated experimentally a superior message passing scheme.
Models adopting this scheme has the capability of going deep.
Two scalable graph learning models are proposed within this GTree network architecture - Graph Tree Convolution Network (GTCN) and Graph Tree Attention Network (GTAN), with demonstrated state-of-the-art performances on several benchmark datasets.
The deep capability is also demonstrated for both models.
More interestingly, the current state-of-the-art APPNP model is found essentially a derivative model within our GTree network architecture with the drawback of requiring a manually tuned teleport probability α .
Our GTCN model outperforms the APPNP model in most test cases.
This work is structured as follows.
Section 2 illustrates the architecture of our Graph Tree Networks (GTree) and two derivative models - Graph Tree Convolution Network (GTCN) and Graph Tree
Attention Network (GTAN), along with the analysis of the model complexity.
Section 3 shows the experimental results including three parts: Section 3.1 is the performance test of our GTCN and GTAN models as compared to several popular and state-of-the-art GNN models on four benchmark datasets.
Section 3.2 is the performance test for our GTCN and GTAN models with deep layers.
We compare our model performances with the vanilla GCN and GAT at different depths.
Section 3.3 is the test of the effect of adding nonlinear layers in our GTCN model.
Section 4 is the discussion and conclusion of our work.
O UR A PPROACH : G RAPH T REE N ETWORKS 2. N ETWORK S TRUCTURE
[Table]
(cid:143) (cid:129) (cid:143) (cid:141) (cid:127) (cid:144)(cid:144) (cid:127) (cid:141)(cid:144) (cid:127) (cid:157)(cid:144) (cid:127) (cid:129) (cid:144) (cid:127) (cid:141)(cid:144) (cid:143) (cid:127) !
[Figure]
Generally graph learning based on direct neighbors only does not yield the best performance.
Incorporating multi-hop neighbors in learning usually brings in richer information.
To have a better learning of one node’s features, the first step is to find all the connections of its neighborhood. The tree representation is a straightforward method to show this topology (edge features are not considered for simplicity at this moment).
Figure 1 right illustrates the tree representation for node 1 in the graph shown in Figure 1 left, with considering up to hop-3 neighbors.
Here we use a directed graph just for illustration, while this tree representation also works for undirected graphs.
In the graph tree, x u ∈ R D is the vector of a node u’s input features.
In practice, usually a one layer MLP is first applied to the input feature map X to get the hidden feature map Z as the input for the following hidden layers.
h ku ∈ R F is the vector of a node u’s output hidden features at the k th hop from the root node.
These output hidden features at the same layer will be aggregated to update the hidden features of their parent nodes.
The final output of the root node u is denoted as h 0 u .
Some nodes may occur multiple times at the same level in the graph tree (like node 5 occurring twice at hopas shown in Figure 1), each of them is from a different hop-k path that may pass different messages to the root node.
The merit of this tree representation is that it displays all the neighborhood of the root node at once and how they are connected with each other.
This topological structure results in a natural formulation of information flow in our Graph Tree Network (GTree), expressed in Equation 1:
[Equation]
where x u is a vector of the input features of a node u, h ku is a vector of the hidden features of a node u at hop k, N u is the set of direct neighbors of a node u, f k ( · ) is a hop-wise nonlinear function at hop k.
The final output of a node u is h 0 u .
We can set a limit for the maximum tree depth (maximum hops in the neighborhood) in Equation 1 to adapt to different needs for the size of the receptive fields.
If the tree depth is L , then we have h Lu
= x u (assuming zero input from the neighbors at hop L + 1).
We use X and H to represent the feature map of the entire nodes in the graph, x v and h v to represent the vector of the features for a particular node v. Note that the order of layer numbers in the graph tree is reverse to the order of hidden layers in the neural network.
The k th layer in the graph tree (i.e. k th hop neighborhood) corresponds to the ( L − k ) th hidden layer in the neural network.
The message passing scheme in our GTree network is different from the one adopted by the GCN, GAT and many of their derivative models.
In GTree networks, we aggregate the hidden features of all direct neighbors of a node u and the input features of itself to update the hidden features h ku of the node u in each propagation layer; while in the GCN or GAT, the output hidden features of the node u from the last hidden layer is used in the aggregation function instead of its input features, as expressed in Equation 2.
We denote the hidden layer number as n in Equation 2 instead of k to avoid ambiguities.
With the tree representation of a graph, the propagation rule in Equation 1 is more intuitive and natural since each node in the tree only has the information of its input features x u before receiving information from its neighbors to update its output features.
In Section 3.2, we will experimentally demonstrate that the propagation scheme in our GTree network outperforms that adopted by the GCN and GAT in terms of the model deep capability.
[Equation]
2. G RAPH T REE C ONVOLUTION AND G RAPH T REE A TTENTION N ETWORKS
Note that Equation 1 expresses a general rule for the node feature propagation (or message passing) in GTree networks.
Different choices of the aggregation function and the nonlinear function f k will derive a variety of different graph learning models, which is inspiring.
In this work, we propose two models within our GTree network architecture: Graph Tree Convolution Network (GTCN) and Graph Tree Attention Network (GTAN).
The propagation rule of our GTCN model is expressed in Equation 3 in the matrix form.
[Equation]
where Z ∈ R N × F is the initial hidden feature map obtained by applying one layer MLP to the input feature X , H k ∈ R N × F is the hidden feature map at the k th hop, A 1 =
˜ A − A 2 , A 2 = ˜
A ⊙ I , ˜
A =
D − 12 ( A + I ) D − 12 .
D and A are the degree and adjacency matrix (no self loop), respectively.
⊙ is the Hadamard entrywise product.
A 1 and A 2 are the non-diagonal and diagonal components of the normalized adjacency matrix, respectively.
Note that we use an identity function for f k in Equation 3 as adding nonlinear layers f k at each tree level will degrade the model performance which will be demonstrated in Section 3.3.
The complete GTCN model is formulated in Equation 4:
[Equation]
where L is the predefined model depth, σ is an activation function like ReLU in our experiments,
Y ∈ R N × C is the output of the model, W 1 and W 2 are the learnable weights, f is a function depending on the learning task which could be a softmax in classifications, and an identity function in regressions.
Similarly, the GTAN model is described as in Equation 5 for one attention head, where w is the global learnable attention weights.
σ 1 and σ 2 are nonlinear activation functions.
We use ReLU and ELU for σ 1 and σ 2 , respectively in our experiments as the same settings in the vanilla GAT model (Velickovic et al., 2018).
Equation 6 shows the formulation for multihead attentions.
M is the number of attention heads.
[Equation]
2. M ODEL C OMPLEXITY
It is straightforward to see from Equation 4 and 5 that the time complexity of our GTCN and GTAN models is O ( L | E | F ) which is in the same order as the vanilla GCN and GAT. Here | E | is the number of edges, F is the dimension of hidden features, and L is the model depth.
2. R ELATION TO APPNP
The propagation rule of the APPNP model (Klicpera et al., 2018) is shown in Equation 7:
[Equation]
As compared to the general propagation rule in Equation 1, it is obvious that the APPNP model is essentially a derivative model within our GTree network architecture using a weighted sum aggregation function and an identity function for f k .
It employs a teleport probability α as the weight [Table] for the center node borrowed from the concept of PageRank.
This hyperparameter α requires fine tuning to achieve the best performance.
Our GTCN model uses the normalized adjacency matrix as the weight which does not require fine tuning and is more intuitive than that of the APPNP model.
In Section 3.1, we demonstrate that our GTCN model outperforms the APPNP model consistently on most benchmark datasets.
2. R ELATION TO C HILD -S UM T REE -LSTM
We include the Child-Sum Tree-LSTM (Tai et al., 2015) as one of our baseline models as it is also a derivative model within our GTree network architecture using a sum aggregation function on the direct neighbors and an LSTM cell as the nonlinear function f k .
We demonstrate that this TreeLSTM model outperforms the vanilla GCN on the Cora and MS Coauthor-CS datasets in Section 3.1, and its performance does not suffer much when going deep, as demonstrated in Section 3.2.
Yet this model has not drawn much attention in the graph learning community, possibly because it was originally targeted at natural language processing tasks.
E XPERIMENTS
In this section, we conduct our experiments in three parts.
The first one in Section 3.1 is to demonstrate the performance of our GTCN and GTAN models.
We choose a depth of 10 for both models to aggregate information from up to hop-10 neighbors in the graph, and compare their performances against two current state-of-the-art deep graph models: APPNP and DAGNN with the same depth of 10.
We also include the most popular GCN and GAT models, as well as the Child-Sum Tree-LSTM as our baseline models.
The second one in Section 3.2 is to demonstrate the deep capability of our models by comparing our model performance with that of the vanilla GCN and GAT at different model depths.
The Child-Sum Tree-LSTM is also included as a comparison.
The third one in Section 3.3 is to discuss the effect of adding nonlinear layers f k in our GTCN model.
The codes to reproduce our experiments are publicly available.
Datasets .
We work on four popular benchmark datasets (Cora, Citeseer, PubMed and MS CoauthorCS) to demonstrate the performance of our proposed models.
These four datasets are from the DGL library (Wang et al., 2019b) with fixed data splits for the Cora, Citeseer and PubMed datasets.
For the Coauthor-CS dataset, we split the data in the following way.
We randomly selected 20 training samples per class and 30 validation samples per class, and all remaining samples are used as the test samples.
We repeat the above process three times yielding three splits for the Coauthor-CS dataset to avoid any bias.
The statistics of the four benchmark datasets are summarized in Table 1, where for the MS CoauthorCS data we have separate tests on three different data splits.
All four datasets are structured as undirected graphs.
Note that the adjacency matrix for the Citeseer and PubMed datasets provided in DGL (Wang et al., 2019b) includes several self-loops.
We remove these self-loops to obtain the adjacency matrix in our experiments.
3. P ERFORMANCE OF G RAPH T REE C ONVOLUTION AND A TTENTION N ETWORKS
Baseline models .
We compare our GTCN and GTAN models to five GNN models: the vanilla GCN (Kipf & Welling, 2017), GAT (Velickovic et al., 2018), Child-Sum Tree-LSTM (Tai et al., 2015) and the current state-of-the-art APPNP (Klicpera et al., 2018) and DAGNN (Liu et al., 2020) models (we are also interested in comparing with the AdaGCN model (Sun et al., 2021) but the source code for the model has not been publicly available yet).
Model hyperparameters .
For fair comparisons, all models are set up in almost the same settings as in their original papers.
All implementations use Adam optimizer with two hyperparameters: learning rate and weight decay.
For the vanilla GCN, we use two layers with 64 hidden units and the dropout is 0.5 for all datasets.
The learning rate and weight decay are 0.01 and 5e-4 respectively for all datasets.
For the vanilla GAT, we also use two layers with 64 hidden units, and other hyperparameters are fine-tuned to obtain the best performance.
One attention head is used for both layers 1 .
The layer dropout and attention dropout are 0.8 and 0.8 respectively for the Cora and Citeseer datasets, 0.
and 0.2 respectively for the PubMed and Coauthor-CS datasets.
The learning rate and weight decay are 0.01 and 5e-4 respectively for all datasets.
For the 2-layer Child-Sum Tree-LSTM model, the input dropout and the dropout after the LSTM cell are 0.8 and 0.6 respectively.
The learning rate and weight decay are 0.01 and 5e-4 respectively for all datasets.
For the APPNP, DAGNN, and our GTCN and GTAN models, we use a maximum hop of 10, the same as that used by the original paper of the APPNP model (Klicpera et al., 2018).
64 hidden units are used for these four deep models.
For the APPNP, the dropout is 0.5 for the first MLP layer and the edge dropout is 0.5 for the propagation layer, the teleport probability α is 0.1 for the Cora, Citeseer and Pubmed datasets, and 0.2 for the Coauthor-CS dataset.
For the DAGNN, the dropout is 0.8 and learning rate is 0.01 for all datasets.
The weight decay is 2e-2 for the Citeseer dataset, and 5e-3 for the Cora, PubMed and Coauthor-CS datasets.
For our GTCN, we have two dropouts: one is for the initial one-layer MLP, and the other is for the propagation layer.
Corresponding dropout values are set to (0.6, 0.6) for the Cora dataset, (0.8, 0.6) for the Citeseer dataset, (0.8, 0.5) for the
Pubmed dataset and (0.6, 0.2) for the Coauthor-CS dataset.
The learning rate is 0.01 for the Cora,
Citeseer and Coauthor-CS datasets, and 0.02 for the PubMed dataset.
The weight decay is 5e-4 for the Cora, Citeseer and PubMed datasets, and 5e-3 for the Coauthor-CS dataset.
For our GTAN, we have the same two dropouts as for the GTCN, which are set to (0.6, 0) for the Cora, Citeseer and
PubMed datasets, and (0.2, 0.2) for the Coauthor-CS dataset.
The learning rate is set to 0.01 for all datasets.
The weight decay is set to 5e-4 for the Cora, Citeseer and PubMed datasets, and 5e-3 for the Coauthor-CS dataset.
The maximum training epochs is set to 1000 for all experiments. The early-stopping patience number is set to 100 for the DAGNN model (the same number is used in the original paper (Liu et al., 2020)), 300 for our GTAN model, and 200 for all other models.
Results .
We take 30 runs of each model on each dataset.
The test results in classification accuracy are represented by mean ± std after removing the lowest 10% and highest 10% points, and are summarized in Table 2.
The Macro-F1 scores of each model on each dataset are summarized in
Table 4 (See Section A.1).
It is demonstrated that our GTCN model outperforms the current stateof-the-art baseline models APPNP and DAGNN on the Cora, Citeseer, and Coauthor-CS datasets.
Our GTAN model performs not as good as our GTCN though, it also achieves the state-of-the-art performance on the Coauthor-CS dataset, and almost matches the current state-of-the-art baseline models on the Cora and PubMed datasets.
[Table]
T EST OF MODEL DEPTH CAPABILITY
In this section [Table] we demonstrate our models’ capability of going deep.
We compare our GTCN model with the vanilla GCN, and our GTAN model with the vanilla GAT.
We also include the ChildSum Tree-LSTM model in the comparison as it is a derivative model within our GTree network architecture.
All models are tested with a depth of 2, 5 and 10, respectively.
The classification accuracy is obtained with the same process as described in Section 3.1.
Model hyperparameters .
For all models under test, we use 64 hidden units for every intermediate hidden layer.
For the vanilla GCN, the dropout is 0.5, learning rate is 0.01 and weight decay is 5e-4 for all datasets and all model depths.
For the vanilla GAT, we use one attention head for all experiments.
At depth 2, the hyperparameters are the same as used in Section 3.1.
At depth 5, the hyperparameters are mostly the same as used at depth 2 except that we set 0 for the attention dropout for all datasets.
At depth 10, the hyperparameters are fine-tuned with the layer dropout and the attention dropout of (0.2, 0) for all datasets.
The learning rate and weight decay are 0.01 and 5e-3 respectively for all datasets.
For the Child-Sum Tree-LSTM, the settings are consistent for all three depths and four datasets, with the input dropout and the dropout after the LSTM cell being (0.8, 0.6), learning rate being 0.01 and weight decay being 5e-4.
For our GTCN model, the settings are the same as used in Section 3.1.
For our GTAN model, the settings for depth 2 and 5 are mostly the same as described in Section 3.1, except that the attention dropout is set to 0.6 for the Cora and Citeseer datasets.
Results .
The test results of all five models at different depths are illustrated in Figure 2.
More details are summarized in Table 5, 6, 7, and 8 respectively in Section A.2.
The results show that the performance of Graph Tree Network based models (GTCN, GTAN and Child-Sum Tree-LSTM) does not compromise in general when going deep, while the performance of the vanilla GCN and
GAT models degrades significantly when going deep which is known as the ”over-smoothing” issue.
This comparison demonstrates the superiority of the message passing scheme in our GTree network architecture.
3. E FFECT OF ADDING NONLINEAR LAYERS
We find that adding a nonlinear layer f k in Equation 1 will degrade the model performance.
To demonstrate this experimentally, we construct a variant of our GTCN model by adding a fully connected nonlinear layer with the ReLU activation function after each propagation layer, and call it a GTCN2 model.
The propagation rule in GTCN2 model is now H k =
ReLU (cid:0)(cid:0)
A 1 H k +1 + A 2 Z (cid:1) W (cid:1) .
We compare the performance of the GTCN and GTCN2 models on the Cora dataset at different depths (2, 5, and 10), and summarize the results in Table 3.
It is seen that adding a nonlinear layer after each propagation layer causes degradation in the model
[Figure]
performance at all depths.
This may explain the not-so-good performance of the Child-Sum TreeLSTM model, as it uses an LSTM cell as the nonlinear function for f k .
[Table]
C ONCLUSION
In this paper, we propose an innovative self-interpretive graph learning architecture: deep Graph
Tree Networks, which is naturally derived from the tree representation of graphs.
The message passing scheme in Graph Tree Networks is more natural and intuitive than that in the vanilla GCN and GAT models.
Two models within the Graph Tree Network architecture are proposed - Graph Tree Convolution and Graph Tree Attention Networks, with demonstrated state-of-the-art performance or matching current state-of-the-art performance.
We believe our Graph Tree Network is a promising and inspiring graph learning architecture in three aspects: 1.
The message passing scheme in our GTree networks is intuitive and interpretive from the tree representation of graphs.
This innovative graph network architecture should inspire following work on new and interesting graph learning models as different aggregation functions and transformation functions f k in Euqationwill lead to different models within our GTree network architecture.
One big advantage of our GTree Networks is that models within this architecture can go deep by stacking multiple propagation layers where each layer aggregate information only from the direct neighborhood.
This makes our GTree Network a more general architecture which can also work for heterogeneous graphs.
If we consider one heterogeneous graph with different types of nodes and edges as studied by other works (Wang et al., 2019c; Cen et al., 2019; Fu et al., 2020), an attention-based graph tree heterogeneous model - Heterogeneous Graph Tree Attention Network (HGTAN) is proposed and formulated in Equation 8:
[Equation]
where V is the set of all nodes, and | V | is the total number of nodes.
x cu ∈ R D c , c ∈ C , C is the set of node types, p ∈ P , P is the set of edge types.
W c ∈ R D c × F , w p ∈ R 2 F , Θ ∈ R D p × F , b ∈ R D p , and q ∈ R D p are learnable weights.
Dealing with metapaths in heterogeneous graphs is one of the biggest hurdles confronted by most of the current heterogeneous models like MAGNN (Fu et al., 2020) and HAN (Wang et al., 2019c), as the number of metapaths increases dramatically when aggregating multi-hop neighbors.
The big advantage of our proposed HGTAN model is that it avoids dealing with metapaths explicitly by stacking multiple propagation layers where each layer only aggregates the direct neighbors.
We are currently exploring the heterogeneous graph tree models and results will be presented in the future
A. T EST RESULTS ON MODEL PERFORMANCES IN M ACRO -F1 SCORE
[Table] [Table]
. T EST
[Table]
[Table] [Table] [Table]