Decoupled Graph Neural Networks based on Label Agreement Message Propagation
Abstract
Decoupling has become a new paradigm in Graph Neural Networks (GNNs) for its effectiveness and scalability.
However, this paradigm still faces two several restrictions: unsatisfying propagation, caused by noisy or confused edges, could greatly degrade model performance; fixed aggregation schema with the same propagation steps and the same combination weights for each node limit achieving optimal performance.
To address these problems, we propose a novel decoupled graph model named LA-DGNN based on label agreement message propagation and combine the intermediate feature after each propagation step as input.
In our method, we decouple the graph model which trains a base predictor based on multi-layer perceptrons with a pre-step to propagate features and a post-step to propagate labels.
We utilize an auxiliary label agreement model to generate proper edge weights to promote reliable propagation.
When training the base predictor, we concatenate all intermediate features after each propagation step to make the model dynamically learn information of neighbors at different distances.
Extensive experiments on five real-world datasets demonstrate that our method achieves superior performance over all baseline methods in terms of node classification accuracy.
Introduction
Graph Neural Networks (GNNs) have achieved great success in a wide range of graph-based applications, such as node classification, graph classification, link prediction and community detection (Li et al., 2018; Knyazev et al., 2019;
He et al., 2020; Bakshi et al., 2018).
Most GNNs follow the paradigm that features are transformed and aggregated via graph convolution layers to generate node representations.
Through K graph convolution layers, nodes obtain information from their K -hop neighborhoods which are called Receptive Field.
However, these models face the challenge of receptive field restriction.
The number of GNN layers grows with the size of receptive field, resulting in high computation and memory cost.
What’s worse, GNNs occur notorious over-smoothing issue while directly applying multiple layers (Li et al., 2019).
Many recent advancing works try to decouple the feature transformation and neighborhood aggregation in each convolution layer to access to more scalable and eﬀicient models.
For example, SGC (Wu et al., 2019) successively removes nonlinearities and collapsing weight matrices between consecutive layers to reduce excess complexity.
APPNP (Klicpera et al., 2019) separates the feature propagation and neural network training to achieve a much larger receptive field size.
They derive an improved propagation schema based on personalized PageRank to permit the use of far more propagation steps without leading to over-smoothing.
On the contrary, C&S (Huang et al., 2021) propagates label messages.
They train a base predictor with node features that ignores graph structure and apply two
[Figure]
post operation based on label propagation (Zhou et al., 2003; Wang & Zhang, 2008) which propagate residual errors and labels separately.
Both feature propagation and label propagation are based on the homophily hypothesis that features and labels vary smoothly over the edges of the graph.
However, this hypothesis is not fully matched in practice for graph edges are from noisy sources.
Therefore, some edges’ corresponding labels are not the same.
Besides, these models treat each propagation equally that use a hyper-parameter to mix current information and neighborhood information.
However, neighbors at different distances have different importance for nodes.
To address these two problems, based on previous advancements, we propose a novel decoupled graph model named LA-DGNN.
In our model, we first train a label agreement model based on the labels in training set to generate proper edge weight for each edge.
Then we propagate features according to the weighted adjacent matrix and concatenates intermediate features for each node after every propagation step.
The aggregated features serve as input for a base predictor based on multi-layer perceptrons.
After training, a post label propagation is applied to smooth predictions and leverage known labels.
Methodology
We start with notations used through this paper.
Let G =
( V , E , X ) denotes an undirected graph with nodes V and edges E .
The nodes in G are described by the feature matrix
X ∈ R n × h , where n is the number of nodes and h is the dimension of features.
Only a subset of nodes V L have observed labels and the goal is to infer the labels of the remaining nodes V U .
Our model follows the idea of decoupled GNNs, which treats message propagation as preand post- operations.
The illustration of LA-DGNN is demonstrated in Figure 1. We train a classifier based on the augmented features after a feature propagation process.
Then, we apply a label propagation process to smooth the prediction.
In order to ensure the correct propagation paths, we train a label agreement prediction model based on the known labels.
Aiming at getting the best receptive field for each node, we combine the features after every propagation step into a vector to feed into the base classifier.
2.1 Label agreement model
Most existing graph learning methods rely on the homophily hypothesis to incorporate graph structure into model design.
Therefore, the noisy edges which connect two nodes with different labels will damage the performance.
Take LP (Zhou et al., 2003) as an example, its performance on graphs without noisy edges achieves 17.1%, 8.9%, 18.7% improvement compared with graphs with noisy edges on Cora, Citeseer and Pubmed datasets under standard division.
A series of methods try to reduce the impact of the noisy edges.
For instance, GAT (Velickovic et al., 2018) utilizes the features of two connected nodes to learn the edge weights for convolution operation.
We borrow the idea from GAM (Stretcu et al., 2019) and utilize the edge weight learning process.
The features of two neighboring nodes are used as input to predict the label consistency of the two nodes.
First, we embed the node features into a hidden space based on multi-layer perceptrons.
e i =
mlp ( x i ) , where x i represents the feature vector of node i .
Then, we apply subtraction and square operations between the two embeddings to eliminate the influence of two nodes’ order, e ij =
( e i − e j ) 2 .
The calculation of the aggregated embeddings e ij can be considered to measure the distance between node i and node j in the hidden space, which is an intuitive representation of the agreement probability (Stretcu et al., 2019).
Finally, the aggregated embeddings are fed into a linear classifier to predict the confidence that the two nodes have the same label,
s = cls
( e i,j ) .
We use binary cross entropy loss with logits ℓ bce as the loss function of the agreement model, which is defined as:
[Equation]
2.2 Confidence-guided feature propagation
The output confidence scores of the label agreement model reflects the similarity of two connected nodes, which are ideal edge weights on graphs.
Appropriate edge weights will promote closer combination of similar nodes and weaken the effect of noisy edges.
We use the output confidence to build the confidence-guided adjacency matrix ˜ A in which each non-zero elements A ij is the sigmoid score predicted by the label agreement model using the feature of node i and node j ,
˜ A
=
Sigmoid ( LA − Model ( X , E )) .
As the previous convention, we apply the symmetric normalization adjacency matrix ˆ A from ˜ A in message passing process.
ˆ A = ˜ D − 1 / 2 ˜
A ˜ D − 1 / 2 , where ˜ D − 1 / 2 denotes the diagonal degree matrix of ˜ A .
The K -step feature propagation is defined as,
[Equation]
where X (0) =
X .
After K -step propagation, nodes obtain a vector [ X ( [Equation] ) , X (1) , · · · , X ( K ) ] that denotes the original feature and features after each propagation step.
As we discuss previously, different nodes have different sensitivities to receptive field.
In our model, we utilize a simple but effective approach that concatenates the original feature and propagated features as the input.
The following classifier is available to features at different steps, which is able to learn the importance of information at different distances automatically and adjust its concentration to the most important part.
The augmented node features are produced as:
[Equation]
2.3 Model training
After feature propagation, each node’s features are augmented by its neighbors.
We use the augmented feature matrix ˆ X to train a shallow classifier based on multi-layer perceptrons.
The cross entropy measurement between the predicted label distributions h i =
mlp (ˆ x i ) and the ground truth one-hot label vector is adopted as the loss function:
[Equation]
where L represents the size of training set, C denotes the number of classes and ˆ x i is the i -th row of ˆ X .
In the model training process, the label agreement model and the base predictor can not only be trained sequentially but also co-trained on small graphs.
In the co-trained schema, the total loss is L =
L CL + β L LA , where β is the balancing parameter.
2.4 Confidence-guided label propagation
Label smoothing has been found to be an effective way to incorporate label information at inference and improve model performance (Huang et al., 2021).
In our method, we also apply a confidence-guided label propagation to further boost the performance.
The label score matrix Z can be obtained by applying a softmax function to the output of the base predictor.
In order to directly incorporate labels at inference, the score of observed nodes are replaced with their ground truth as Z L = Y L .
We perform a weighted Q -step confidenceguided label propagation based on personalized PageRank to obtain the final predicted label distributions.
[Equation]
In this equation, Z (0) =
Z and α ∈ (0 , 1] is the teleport (or restart) probability which allow the nodes to preserve their own information with a certain probability.
The classification for node i ∈
V U is l j = arg max z ( Q ) j , where z ( Q ) j represents the j -th row of Z ( Q ) j .
Experiments
We conduct extensive experiments to demonstrate the effectiveness of our methods.
The quantitative results and analysis of different evaluations are presented here.
3.1 Datasets and baselines
We adopt five public datasets for evaluation: Cora (Sen et al., 2008),
Citeseer (Sen et al., 2008) and Pubmed (Namata et al., 2012) are three classic citation network graphs; CoauthorCS (Shchur et al., 2018) and Coauthor-Physics (Shchur et al., 2018) are two co-authorship graphs based on the Microsoft Academic Graph.
The detailed statistics are summarized in APPENDIX B.
As for the train/valid/test splits, we use 60%/20%/20% random splits following (Wang & Leskovec, 2020).
In the experiments, we compare our model with a range of representative methods, including LP (Zhou et al., 2003), GCN (Kipf & Welling, 2017), GAT (Velickovic et al., 2018),
JKNet (Xu et al., 2018), APPNP (Klicpera et al., 2019), C&S (Huang et al., 2021), GCNLPA (Wang & Leskovec, 2020).
LP is a parameter-free method.
GCN, GAT and APPNP are classical graph models.
APPNP propagates features in advance while C&S propagate labels afterwards.
GCN-LPA utilizes label propagation output as a regularization.
Detailed descriptions can be found in APPENDIX C.
3.2 Results
We record the accuracy of test set under best performance on valid set.
Each experiment is conducted five times and the mean and standard deviation are demonstrated in Table 1.
From the result table, we have several observations.
Our proposed model, LA-DGNN outperforms all the baseline methods on each dataset under the semi-supervised setting.
For example, our model improves the best-performing baseline model, GCN-LPA, by an absolute 0.84% accuracy on Cora.
For the two large coauthor graphs, the accuracy is also improved though slighter compared to the small graphs.
Even though LP is a parameter-free method, it still demonstrates competitive performance on most datasets.
GCN and GAT obtain worse results for the limited and fixed receptive field.
JKNet and APPNP outperform these shallow GNNs for their access to each propagated feature or large receptive fields.
C&S and
GCN-LPA also achieve higher performance on most datasets especially C&S shows comparable performance to our method, which indicates the key to improving performance is to incorporate label information.
[Table] [Table]
[Figure]
[Table]
[Figure]
3.3 Robustness Study
Further experiments are conducted to explore the robustness of our model to noisy edges.
The homophily of a graph characterizes how likely nodes with the same label are near each other in a graph.
Based on Pubmed with 80% homophily, we generate a graph with 50% homophily by adding noisy edges and a graph with 100% homophily by removing all noisy edges.
We perform our method and other baselines on the three graphs.
Each experiment is conducted three times and the results are shown in Figure 2.
From Figure 2, we observe that the performance of all methods is significantly improved when all noisy edges are removed and they achieve comparable accuracy.
When only half of the edges satisfy the label agreement, all baselines experience a large drop while our method still maintains a competitive performance.
These experiments demonstrate structure noise has a dramatic effect on model performance and our model is robust to structure noise.
Conclusion
In this paper, we propose a decoupled graph neural networks, LA-DGNN, with preprocessing feature propagation and post-processing label propagation.
In order to address the structure noise, we train a label agreement model additionally which takes the features of two connected nodes as input to predict the probability they have same labels.
The confidence scores of every edge generated by the label agreement model can guide the information propagation across the edges.
Aiming to solve the fixed aggregation schema problem, we concatenates intermediate features after each propagation step as a whole to enable our model dynamically adopts importance to neighbors at different distance.
Extensive experiments demonstrate LA-DGNN outperforms other classic and SOTA methods on five benchmark datasets and achieves both effectiveness and robustness.
A Structure Noise
Influence
[Figure]
We conduct experiments on Cora, Citeseer and Pubmed under the standard split to verify the influence of noisy edges which do not meet label agreement.
We apply Label Propagation separately to the original graph and the clear graph with all the noisy edges removed.
The classification accuracy is demonstrated in Figure 3.
The red areas represent the improvement that the model can obtain on a clean graph compared to a graph containing noisy edges.
We observe LP achieves 17.1%, 8.9%, and 18.7% accuracy improvement respectively.
The experimental results shows that a significant portion of the potential of the graph learning models is limited by the structure noise.
B Dataset Details
[Table]
We select five representative public graph datasets as our benchmark datasets.
Their detailed statistics are shown in Table 2.
Cora, Citeseer and Pubmed are three most widely used citation networks, where each node represents a paper and an edge indicates a citation relationship.
Coauthor-CS and Coauthor-Physics are two co-authorship network with larger size, where each node represents an author and an edge indicates co-author relationship.
We randomly split these data sets in the ratio of 60%:20%:20% to form the training set, validation set and test set, which is different from
lower label rate settings
in
order to
ameliorate sensitivity to hyper-parameters
C Compared methods
In this section, we give a detailed description of baseline methods including both classic and state-of-the-art methods.
• LP
(Zhou et al., 2003)
Label propagation is a classic semi-supervised learning algorithm that propagates the known labels along the graph to other unlabeled nodes.
• GCN (Kipf & Welling, 2017)
Graph Convolutional Network is a widely used approach for semi-supervised learning on graph-structured data that is based on an eﬀicient variant of convolutional neural networks.
• GAT (Velickovic et al., 2018)
Graph Attention Networks is a novel convolution-style graph neural networks based on masked self-attentional layers.
It specifies finegrained weights for neighborhood aggregation while does not depend on knowing the entire graph structure upfront.
JKNet (Xu et al., 2018)
Jumping Knowledge Networks is a novel technique to let model flexibly leverages different neighborhood ranges for each node to enable better structure-aware representation.
It leverages several approaches to selectively exploit information from neighborhoods of differing locality to break the limit of fixed numbers of neighborhood aggregations.
• APPNP (Klicpera et al., 2019)
Approximation Personalized Propagation of Neural Predictions is derived by considering the relationship between GCN and personalized PageRank.
It overcomes the limited range problem of many message passing models by decoupling prediction and propagation.
• C&S (Huang et al., 2021)
Correct And Smooth only utilizes shallow models that ignore the graph structure with two simple post-processing steps based on label propagation techniques.
In C&S, the graph structure is not used to learn parameters but instead as a post-processing mechanism, which reduce magnitude parameters.
• GCN-LPA (Wang & Leskovec, 2020) GCN-LPA is an end-to-end model that unifies GCN and LPA.
It serve LPA as regularization to assist the GCN to learn proper edge weights to improve classification performance.
D Implement Details
Here we provide some more details on the models that we use.
For all models, we apply the
Adam optimizer and tune the learning rate.
We use ELU as our activation function and add batch normalization layer after each linear layer.
On Cora, Citeseer, the dropout rate is 0.7 while it equals 0.2 on Pubmed, Coauthor-CS, Coauthor-Physics.
For Label Propagation, we propagate 10 steps with a teleport probability of 0.2.
For GCN and GAT, we build these models with three convolutional layers and 128 hidden channels.
For JKNet, we apply a 3-layer GCN as our base model and use concatenation aggregator.
As for APPNP, we use a 2-layer MLP with 10 steps of propagation.
On Cora, Citeseer and Pubmed, the label agreement model and base prediction model are trained jointly with a balancing parameter of 2.
On Coauthor-CS and Coauther-Physics, we train label agreement model at first and then train the base predictor based on the predicting edge weights.