D EEP G RAPH T REE N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose Graph Tree Networks (GTreeNets), a self-interpretive deep graph neural network architecture which originates from the tree representation of graphs.
In the tree representation, each node forms its own tree where the node itself is the root node and all its neighbors up to k-hop are the subnodes.
Under the tree representation, messages propagate upward from the leaf nodes to the root node naturally to update the root node’s hidden features.
This message passing scheme, which has better interpretability, is essentially different from that in the vanilla Graph Convolution Network (GCN), Graph Attention Network (GAT) and many of their derivatives.
Two scalable graph learning models are proposed within this GTreeNet architecture - Graph Tree Convolution Network (GTCN) and Graph Tree Attention Network (GTAN), with experimentally demonstrated state-of-the-art performance on several benchmark datasets and the capability of going deep.
I NTRODUCTION
Graph Neural Networks (GNNs), a class of neural networks for learning on graph structured data, have been successfully applied in many areas to solve real world problems, such as link predictions in social networks (Fan et al., 2019),
pattern recognition
(Ju et al., 2020; Shi & Rajkumar, 2020), product recommendation and personalized search in E-commerce (Zhu et al., 2019), fraud detection (Wang et al., 2019a;
Dou et al., 2020), protein interface predictions (Fout et al., 2017), power estimation and tier design in the semiconductor industry (Zhang et al., 2020; Lu et al., 2020), traffic forecasting (Yu et al., 2017), and natural language processing (Yao et al., 2019; Vashishth et al., 2020; Wu et al., 2021).
Among many different graph learning approaches, the class of spatial graph convolution based models, which adopts a message passing scheme to update node features, has gained particular attention due to its simplicity yet good performance.
The most representative work among this class is the Graph Convolutional Network (GCN) (Kipf & Welling, 2017) which limits the ChebNet (Defferrard et al., 2016) up to the 1st order polynomial, and together with some further approximation leading to a direct neighbor aggregation.
The success of the GCN model (Ying et al., 2018; Hu et al., 2020b) has led to the rapid development in spatial convolution based models, such as the Graph Attention Network (GAT) (Velickovic et al., 2018),
GraphSage
(Hamilton et al., 2017), APPNP (Klicpera et al., 2018), DAGNN (Liu et al., 2020), and etc.
Today the vanilla GCN and GAT are the two most popular baseline models.
One GCN or GAT layer aggregates only the direct neighbor nodes.
Although they work well in many test cases, their performance degrades as stacking multiple propagation layers to achieve larger receptive fields.
Xu et al. (2018) and Li et al. (2018) attribute such degradation in performance to over-smoothing effect that nodes from different classes become indistinguishable, while Liu et al.
(2020) attributes that to the intertwined propagation and transformation. A small neighborhood may not provide enough information especially when nodes are sparsely labeled (Klicpera et al., 2018; Liu et al., 2020). Many recent work has been devoted to extending the size of neighborhood utilized in graph learning.
APPNP (Klicpera et al., 2018) and DAGNN (Liu et al., 2020) are two most recent deep graph models with state-of-the-art performance on several popular benchmark datasets.
APPNP (Klicpera et al., 2018) is developed based on personalized PageRank to preserve local information which requires fine tuning of a teleport probability.
DAGNN (Liu et al., 2020) aggregates neighbors from different hops in parallel (in one layer) to utilize information from a larger receptive field as compared to
GCN.
Although DAGNN has achieved state-of-the-art performance, it may only be applicable to adopt simple aggregation function as the one used in GCN.
It could be computationally challenging to combine such multi-hop aggregation scheme with attention-based GNN models (such as GAT, and graph transformer (Dwivedi & Bresson, 2020; Hu et al., 2020c)); because this aggregation scheme requires computation of the attention weight for each pair of a node and its k-hop neighbor, which could be a too large number.
In the aggregation of k-hop neighborhood, the number of attention weights is the total number of non-zero entries in the k th power of the graph adjacency matrix A . Although the adjacency matrix is usually sparse, its k th power could be non-sparse especially for undirected graphs.
For example, in the Cora dataset (Wang et al., 2019b)
A 5 has ∼ 2.2 million non-zero entries while A has only ∼ 10k non-zero entries.
Furthermore, aggregating multi-hop neighborhood in one step neglects path information which is usually important for heterogeneous graphs with rich metapath information (Fu et al., 2020; Wang et al., 2019c).
We propose Graph Tree Networks (GTreeNets), a self-interpretive deep graph neural network architecture which originates from the tree representation of graphs.
In the tree representation, each node forms its own tree where the node itself is the root node and all its neighbors up to k-hop are the subnodes.
Under the tree representation, message propagates upward from the leaf nodes to the root node naturally to update the root node’s hidden features.
This message passing scheme, which has better interpretability, is essentially different from that in the vanilla Graph Convolution Network (GCN), Graph Attention Network (GAT) and many of their derivatives.
Two scalable graph learning models are proposed within this GTreeNet architecture - Graph Tree Convolution Network (GTCN) and Graph Tree Attention Network (GTAN), with experimentally demonstrated state-ofthe-art performance on several benchmark datasets and the capability of going deep by stacking multiple propagation layers.
Our models do not require finely-tuned hyperparameter in the propagation scheme.
This work is structured as follows.
Section 2 illustrates the architecture of our Graph Tree Networks (GTreeNets) and two derivative graph learning models - Graph Tree Convolution Network (GTCN) and Graph Tree Attention Network (GTAN), along with the analysis of model complexity.
Section 3 presents the experimental results including three parts: Section 3.1 shows the performance test of our GTCN and GTAN models as compared to several popular and state-of-the-art GNN models on five benchmark datasets.
Section 3.2 shows the performance test for our GTCN and GTAN models at different depths as compared to the vanilla GCN and GAT models.
Section 3.3 discusses the effect of adding transformation in propagation layers in our GTCN and GTAN models.
We conclude our work in Section 4 and discuss insights for future work.
O UR A PPROACH : G RAPH T REE N ETWORKS
We first introduce notations used throughout this paper. We follow the general convention to use bold uppercase and lowercase to represent matrices and vectors, respectively. The topology of a graph G =
( V, E ) with nodes V and edges E can be fully described by its adjacency matrix A and degree matrix D . | V | = N and | E | = M are the number of nodes and edges, respectively.
N u denotes the set of direct neighbors (1-hop neighbors) of node u . X ∈ R N × D represents the feature map for all nodes, where each row x u ∈ R D represents the feature vector of node u with dimension of D .
Y ∈ R N × C represents the class matrix for all nodes, where each row y u represents the class vector of node u with C classes.
Tree is a straightforward representation for the graph topology, where each node and its neighborhood form one tree with the node itself being the root node and its neighbors being the subnodes.
Figure 1 illustrates the tree representation for node 1 with up to 3-hop neighbors in the sample graph.
We use a directed graph just for illustration, the tree representation also works for undirected graphs.
In Figure 1, h ku ∈ R F is the vector of node u ’s output hidden features at k-hop away from the root node 1.
The final output of the root node 1 is denoted as h 01 .
Messages propagate upward from subnodes to the root node hop by hop in the tree.
Nodes may occur multiple times at the same level in the tree (like node 5 occurring twice at 2-hop as shown in Figure 1), each of them is from a different k-hop path that may pass different messages to the root node.
We assume a node u at any k-hop preserves its initial information before receiving information from its child nodes.
With this 2-hop neighbors: 4, 2, 5, 2,       4(cid:127) (cid:129)(cid:141)   …     (cid:127) (cid:129) (cid:127) (cid:141) (cid:127) (cid:143)  (cid:127) (cid:129)(cid:129) (cid:127) (cid:141)(cid:129) (cid:143) (cid:144) (cid:127) (cid:144)(cid:129) (cid:143) (cid:141) (cid:127) (cid:141)(cid:129) (cid:143) (cid:157) (cid:127) (cid:157) (cid:127) (cid:129) (cid:143) (cid:129) (cid:127) (cid:144) (cid:143) (cid:144) (cid:143) (cid:157) (cid:143) (cid:141) (cid:143) (cid:144) (cid:143) (cid:129) (cid:143) (cid:141) (cid:127) (cid:144)(cid:144) (cid:127) (cid:141)(cid:144) (cid:127) (cid:157)(cid:144) (cid:127) (cid:129)(cid:144) (cid:127) (cid:141)(cid:144) (cid:143) (cid:127) ! 1-hop neighbors: 2, 3,3-hop neighbors: 3, 5, 4, 2,target node:output: assumption, a node’s features are updated by aggregating its child nodes and its own initial features in the tree representation.
This tree representation of graphs leads to a new graph learning architecture - Graph Tree Networks (GTreeNets) as shown in Appendix A.1, with the general message passing rule formulated as Equation 1:
[Equation]
where k = L − 1 , . . L is the tree depth (number of propagation layer).
x u is the vector of input features of node u .
z u = ϕ k ( k u ) is the transformation of input features of node u , i.e. the initial hidden features for the propagation layer.
h ku is the vector of hidden features of node u at k-hop, and h Lu = z u .
f k ( · ) is a hop-wise (layer-wise) transformation function such as MLP.
Combination of different aggregation function and transformation function in Equation 1 derives various graph learning models under the GTreeNet architecture. We propose two graph tree models in Section 2.2 and 2.3: Graph Tree Convolution Network (GTCN) and Graph Tree Attention
Network (GTAN). 2.1. R ELATION TO P ROPAGATION S CHEME IN GCN- LIKE MODELS The general propagation rule in GCN-like models such as GCN, GAT, GraphSage, GIN referenceis [Equation]
Comparing to Equation 1, it is notable that the message passing scheme in out proposed GTreeNet is essentially different from the one adopted by the GCN-like models such as xxxxx.
In GTreeNet, we update a node’s hidden features by aggregating its neighbor’s hidden features and its own initial features. While in GCN-like models, a node’s hidden features are updated by aggregating its neighbor’s hidden features and its own hidden features from the last propagation layer. Our propagation scheme preserves the local information when stacking multiple propagation layers while the propagation scheme in GCN-like models does not.
Note that we use n to denote the propagation layer number in Equation 2 to avoid ambiguities.
2. G RAPH T REE C ONVOLUTION N ETWORK
The propagation rule of our proposed Graph Tree Convolution Network is
[Equation]
where ˆ A = ˜ D − 12 ˜ A ˜ D − 12 is the symmetric normalized adjacency matrix 1 .
˜
A =
( A + I ) is the adjacency matrix with added self-loops.
˜ D is the degree matrix with self-loops where ˜ D uu = (cid:80) v ˜ A uv . z u = MLP ( x u ) .
As compared to the general propagation rule in GTreeNet shown in Equation 1, our GTCN model uses an MLP as the transformation function ϕ k ( · ) , an identity function as f k ( · ) , and a weighted sum function as the aggregation function with weights being the elements of the normalized adjacency matrix.
The mathematical description of our GTCN model is then formulated as [Equation] where Z ∈ R N × F is the initial hidden feature map obtained by applying an MLP to the input feature map X . H k ∈ R N × F is the hidden feature map at the k th hop.
A 1 and A 2 are the non-diagonal and diagonal component of the normalized adjacency matrix, respectively.
W ∈ R F × C is the trainable weight matrix. Note that our model separates the transformation from propagation with no trainable parameters required in the propagation layers. By aggregating a node’s neighbor information with its own initial information, we always preserve the local information of a node.
These two characteristics make our model capable of going deep without comprising performance as demonstrated in Section 3.2.
2.2. R ELATION TO GCN AND APPNP
GCN . The propagation rule in GCN is [Equation] [Equation]
which uses weighted sum to aggregate the hidden features of node u and its neighbors from the last propagation layer, and an MLP with ReLU activation function as f n − 1 ( · ) to transform the feature representation.
Except for the different information used in aggregation as discussed in Section 2.1.1, another major difference between our GTCN model and the GCN model is that a transformation function is applied after each propagation layer in GCN which causes intertwined propagation and transformation, and introduces more trainable parameters.
These differences make the GCN model subject to compromised performance when stacking multiple propagation layers.
APPNP .
The propagation rule in APPNP is
[Equation]
which uses weighted sum to aggregate the hidden features of node u and its neighbors from the last propagation layer, as well as its initial hidden feature. Note that the APPNP also adopts an identity function as f n − 1 ( · ) , i.e. no transformation is applied after propagation. APPNP uses the hidden features of node u from the last propagation layer ( h n − 1 u ) in aggregation to update its hidden feature while our GTCN model does not. In addition, APPNP introduces a teleport probability α to adjust the influence from the neighborhood which requires fine tuning for different graphs.
Note that the APPNP model can be derived from our GTreeNet by adding self-loops in the graph. Therefore, node u is also a child node of itself when building the tree representation. In this case, the general propagation rule in the resulting tree representation becomes:
[Equation] 2.2. R ELATION TO C HILD -S UM T REE -LSTM
The Child-Sum Tree-LSTM (Tai et al., 2015) can also be derived from our GTreeNet, which uses a weighted sum to aggregate a node’s initial features and its neighbors, and an LSTM cell as the transformation function f k ( · ) . The aggregation weights are different from our GTCN model. 2. G RAPH T REE A TTENTION N ETWORK
The propagation rule of our proposed Graph Tree Attention Network is [Equation]
where α ku,v and α ku,u are trainable attention weights calculated as
[Equation]
As compared to the general propagation rule in GTreeNet shown in Equation 1, our GTAN model uses an MLP as the transformation function ϕ k ( · ) , an ELU function as f k ( · ) , and an attention based weighted sum function as the aggregation function. The mathematical description of our GTAN model is then formulated as
[Equation] where α ku,v and α ku,u are calculated as in Equation 9.
2. M ODEL C OMPLEXITY
It is straightforward to see from Equation 4 and 10 that the time complexity of our GTCN and GTAN models is O
( L | E | F ) which is in the same order as the vanilla GCN and GAT.
Here | E | is the number of edges, F is the dimension of hidden features, and L is the model depth.
E XPERIMENTS
In this section, we conduct our experiments in three parts.
The first one in Section 3.1 is to demonstrate the performance of our GTCN and GTAN models.
We choose a depth of 10 for both models to aggregate information from up to 10-hop neighbors in the graph, and compare their performances against two current state-of-the-art deep graph models: APPNP and DAGNN with the same depth.
We also include the most popular GCN and GAT models, as well as the Child-Sum Tree-LSTM as our baseline models.
The second one in Section 3.2 is to demonstrate the deep capability of our models by comparing our model performance with that of the vanilla GCN and GAT at different model depths.
The Child-Sum Tree-LSTM is also included as a comparison.
The third one in Section 3. is to discuss the effect of adding an MLP transformation function f k ( · ) with trainable parameters in our GTCN and GTAN models.
The codes to reproduce our experiments are publicly available.
Datasets .
We work on four popular benchmark datasets (Cora, Citeseer, PubMed and MS CoauthorCS) from DGL library (Wang et al., 2019b) and one OGB datasets ogbn-arxiv (Hu et al., 2020a) to demonstrate the performance of our proposed models.
The four datasets from the DGL library have fixed data splits for the Cora, Citeseer and PubMed datasets.
For the Coauthor-CS dataset, we split the data in the following way.
We randomly selected 20 training samples per class and 30 validation samples per class, and all remaining samples are used as the test samples.
We repeat the above process three times yielding three splits for the Coauthor-CS dataset to avoid any bias.
The ogbn-arxiv dataset is a homogeneous, unweighted, directed graph representing the citation between all Computer Science arXiv papers indexed by MAG.
Each paper has 128-dimensional features obtained by averaging the embeddings of words.
The task is to predict the subject areas (40 total classes) of the paper.
The obgl-ddi dataset is a homogeneous, unweighted, undirected graph representing the interaction between drugs. Each node represents one drug with no extra features, and an embedding layer is used as node feature. The task is to predict drug-drug interactions based on the already [Table]
known interactions.
The evaluation metric is to rank true drug interactions among a set of randomlysampled negative interactions, and count the ratio of positive edges that are ranked at 20-place or above (Hits@20).
The statistics of the four benchmark datasets are summarized in Table 1, where for the MS CoauthorCS data we have separate tests on three different data splits.
All four datasets are structured as undirected graphs.
Note that the adjacency matrix for the Citeseer and PubMed datasets provided in DGL (Wang et al., 2019b) includes several self-loops.
We remove these self-loops to obtain the adjacency matrix in our experiments.
3. P ERFORMANCE OF G RAPH T REE C ONVOLUTION AND A TTENTION N ETWORKS
Baseline models .
We compare our GTCN and GTAN models to five GNN models: the vanilla GCN (Kipf & Welling, 2017), GAT (Velickovic et al., 2018), Child-Sum Tree-LSTM (Tai et al., 2015) and the current state-of-the-art APPNP (Klicpera et al., 2018) and DAGNN (Liu et al., 2020) models (we are also interested in comparing with the AdaGCN model (Sun et al., 2021) but the source code for the model has not been publicly available yet).
Model hyperparameters .
For fair comparisons, all models are set up in almost the same settings as in their original papers.
All implementations use Adam optimizer with two hyperparameters: learning rate and weight decay.
For the vanilla GCN, we use two layers with 64 hidden units and the dropout is 0.5 for all datasets.
The learning rate and weight decay are 0.01 and 5e-4 respectively for all datasets.
For the vanilla GAT, we also use two layers with 64 hidden units, and other hyperparameters are fine-tuned to obtain the best performance.
One attention head is used for both layers 2 .
The layer dropout and attention dropout are 0.8 and 0.8 respectively for the Cora and Citeseer datasets, 0.
and 0.2 respectively for the PubMed and Coauthor-CS datasets.
The learning rate and weight decay are 0.01 and 5e-4 respectively for all datasets.
For the 2-layer Child-Sum Tree-LSTM model, the input dropout and the dropout after the LSTM cell are 0.8 and 0.6 respectively.
The learning rate and weight decay are 0.01 and 5e-4 respectively for all datasets.
For the APPNP, DAGNN, and our GTCN and GTAN models, we use a maximum hop of 10, the same as that used by the original paper of the APPNP model (Klicpera et al., 2018).
64 hidden units are used for these four deep models.
For the APPNP, the dropout is 0.5 for the first MLP layer and the edge dropout is 0.5 for the propagation layer, the teleport probability α is 0.1 for the Cora,
Citeseer and Pubmed datasets, and 0.2 for the Coauthor-CS dataset.
For the DAGNN, the dropout is 0.8 and learning rate is 0.01 for all datasets.
The weight decay is 2e-2 for the Citeseer dataset, and 5e-3 for the Cora, PubMed and Coauthor-CS datasets.
For our GTCN, we have two dropouts: one is for the initial one-layer MLP, and the other is for the propagation layer.
Corresponding dropout
[Table]
values are set to (0.6, 0.6) for the Cora dataset, (0.
The maximum training epochs is set to 1000 for all experiments.
The early-stopping patience number is set to 100 for the DAGNN model (the same number is used in the original paper (Liu et al., 2020)), 300 for our GTAN model, and 200 for all other models.
Results .
We take 30 runs of each model on each dataset.
The test results in classification accuracy are represented by mean ± std after removing the lowest 10% and highest 10% points, and are summarized in Table 2.
The Macro-F1 scores of each model on each dataset are summarized in
Table 4 (See Section A.2).
It is demonstrated that our GTCN model outperforms the current stateof-the-art baseline models APPNP and DAGNN on the Cora, Citeseer, and Coauthor-CS datasets.
Our GTAN model performs not as good as our GTCN though, it also achieves the state-of-the-art performance on the Coauthor-CS dataset, and almost matches the current state-of-the-art baseline models on the Cora and PubMed datasets.
3. T EST OF MODEL DEPTH CAPABILITY
In this section, we demonstrate our models’ capability of going deep.
We compare our GTCN model with the vanilla GCN, and our GTAN model with the vanilla GAT.
We also include the ChildSum Tree-LSTM model in the comparison as it is a derivative model within our GTree network architecture.
All models are tested with a depth of 2, 5 and 10, respectively.
The classification accuracy is obtained with the same process as described in Section 3.1.
Model hyperparameters .
For all models under test, we use 64 hidden units for every intermediate hidden layer.
For the vanilla GCN, the dropout is 0.5, learning rate is 0.01 and weight decay is 5e-4 for all datasets and all model depths.
For the vanilla GAT, we use one attention head for all experiments.
At depth 2, the hyperparameters are the same as used in Section 3.1.
At depth 5, the hyperparameters are mostly the same as used at depth 2 except that we set 0 for the attention dropout for all datasets.
At depth 10, the hyperparameters are fine-tuned with the layer dropout and the attention dropout of (0.2, 0) for all datasets.
The learning rate and weight decay are 0.01 and 5e-3 respectively for all datasets.
For the Child-Sum Tree-LSTM, the settings are consistent for all three depths and four datasets, with the input dropout and the dropout after the LSTM cell being [Figure] (0.8, 0.6), learning rate being 0.01 and weight decay being 5e-4.
For our GTCN model, the settings are the same as used in Section 3.1.
For our GTAN model, the settings for depth 2 and 5 are mostly the same as described in Section 3.1, except that the attention dropout is set to 0.6 for the Cora and
Citeseer datasets.
Results .
The test results of all five models at different depths are illustrated in Figure 2.
More details are summarized in Table 5, 6, 7, and 8 respectively in Section A.3.
The results show that the performance of Graph Tree Network based models (GTCN, GTAN and Child-Sum Tree-LSTM) does not compromise in general when going deep, while the performance of the vanilla GCN and
GAT models degrades significantly when going deep which is known as the ”over-smoothing” issue.
This comparison demonstrates the superiority of the message passing scheme in our GTree network architecture.
3. E FFECT OF ADDING NONLINEAR LAYERS
We find that adding a nonlinear layer f k in Equation 1 will degrade the model performance.
To demonstrate this experimentally, we construct a variant of our GTCN model by adding a fully connected nonlinear layer with the ReLU activation function after each propagation layer, and call it a GTCN2 model.
The propagation rule in GTCN2 model is now H k =
ReLU (cid:0)(cid:0)
A 1 H k +1 + A 2 Z (cid:1) W (cid:1) .
We compare the performance of the GTCN and GTCN2 models on the Cora dataset at different depths (2, 5, and 10), and summarize the results in Table 3.
It is seen that adding a nonlinear layer after each propagation layer causes degradation in the model performance at all depths.
This may explain the not-so-good performance of the Child-Sum TreeLSTM model, as it uses an LSTM cell as the nonlinear function for f k .
[Table]
C ONCLUSION
In this paper, we propose an innovative self-interpretive graph learning architecture: deep Graph
Tree Network (GTreeNet), which is naturally derived from the tree representation of graphs.
The message passing scheme in the GTreeNet has better interpretability than that in the vanilla GCN and GAT models.
Two models within the GTreeNet architecture are proposed - Graph Tree Convolution
Network (GTCN) and Graph Tree Attention Network (GTAN), with demonstrated state-of-the-art performances on several benchmark datasets and the capability of going deep by stacking multiple propagation layers.
The major advantage of the message passing scheme in the GTreeNet is that models adopting this message passing scheme can go deep by stacking multiple propagation layers instead of calculating multi-hop information directly, where only information from the direct neighbors is used in each layer.
This advantage may allow us to extend our GTCN and GTAN models to heterogeneous graphs.
Dealing with metapaths in heterogeneous graphs is one of the biggest hurdles confronted by most of the current heterogeneous models like MAGNN (Fu et al., 2020) and HAN (Wang et al., 2019c), as the number of metapaths increases dramatically when aggregating multi-hop neighbors.
The message passing scheme in our models helps avoid dealing with metapaths explicity as information on metapaths is self-contained by stacking multiple propagation layers. This opens future research opportunities.
The general propagation rule we formulate in the GTreeNet architecture may also open new research opportunities by exploring different aggregation schemes. A. GT REE N ET I LLUSTRATION Figure 3 is an illustration of our Graph Tree Network (GTreeNet).
A. T EST RESULTS ON MODEL PERFORMANCES IN M ACRO -F1 SCORE
[Table]
90.5 ± 0. 90.2 ± 0.
A.
[Table] [Table]
[Table] [Table]