Knowledge Distillation Improves Graph Structure Augmentation for Graph Neural Networks
Abstract
Graph (structure) augmentation aims to perturb the graph structure through heuristic or probabilistic rules, enabling the nodes to capture richer contextual information and thus improving generalization performance.
While there have been a few graph structure augmentation methods proposed recently, none of them are aware of a potential negative augmentation problem, which may be caused by overly severe distribution shifts between the original and augmented graphs.
In this paper, we take an important graph property, namely graph homophily, to analyze the distribution shifts between the two graphs and thus measure the severity of an augmentation algorithm suffering from negative augmentation.
To tackle this problem, we propose a novel Knowledge Distillation for Graph Augmentation (KDGA) framework, which helps to reduce the potential negative effects of distribution shifts, i.e., negative augmentation problem.
Specifically, KDGA extracts the knowledge of any GNN teacher model trained on the augmented graphs and injects it into a partially parameter-shared student model that is tested on the original graph.
As a simple but efficient framework, KDGA is applicable to a variety of existing graph augmentation methods and can significantly improve the performance of various GNN architectures.
For three popular graph augmentation methods, namely
GAUG, MH-Aug, and GraphAug, the experimental results show that the learned student models outperform their vanilla implementations by an average accuracy of 4.6% (GAUG), 4.2% (MH-Aug), and 4.6% (GraphAug) on eight graph datasets.
Codes are available at: https://github.com/LirongWu/KDGA .
Introduction
In many real-world applications, including social networks, chemical molecules, and citation networks, data can be naturally modeled as graphs.
Recently, the emerging Graph Neural Networks (GNNs) [4, 13, 48, 47, 22, 24, 28, 29] have demonstrated their powerful capability due to their superior performance in various graph-related tasks, including link prediction [59], node classification [21, 50], and graph classification [7].
Despite their great success, GNNs usually suffer from weak generalization due to its heavy reliance on the quantity of annotated labels and the quality of the graph structure.
To boost generalization capabilities, a natural solution is to increase the amount of training data by creating plausible variations of existing data, which have been widely adopted in fields such as computer vision [33, 30, 9, 39, 25, 31, 26, 27] and natural language processing [46, 1, 36, 5, 34].
The data augmentation on graphs can be mainly divided into two branches: node feature augmentation and graph structure augmentation.
While the former has been well studied by directly extending existing approaches for image and text data to graph data [56, 19, 15], comparatively little work has been done to study graph structure augmentation [35, 2, 62, 32].
Following the nomenclature of existing works [62, 32], we directly abbreviate graph (structure) augmentation to graph augmentation for the sake of brevity in this paper.
The purpose of graph augmentation is to reasonably perturb the graph structure through heuristic or probabilistic rules, enabling the nodes to capture richer contextual information and thus improving generalization performance.
For example, DropEdge [35] randomly removes a fraction of edges before each training epoch, in an approach reminiscent of dropout [40].
Besides, AdaEdge [2] iteratively adds (removes) edges between nodes predicted to have the same (different) labels with high confidence.
In contrast to these heuristic methods, GAUG [62] proposes to optimize the graph augmentation and GNN parameters in an end-to-end manner.
Similarly, MH-Aug [32] proposes a novel framework that draws a sequence of augmented graphs from an explicit target distribution, which enables flexible control of the strength and diversity of augmentation.
In this paper, we identify a potential negative augmentation problem for existing graph augmentation methods, i.e., the augmentation may cause overly severe distribution shift between the augmented graphs used for training and the original graph used for testing, which leads to suboptimal generalization.
Moreover, we conduct extensive experiments to demonstrate the existence and hazard of distribution shifts and find that the direction of distribution shifts may be opposed on homophily and heterophily graphs.
We propose a solution to the identified problem by adopting a Knowledge
Distillation for Graph Augmentation (KDGA) framework, which helps to reduce the potential negative effects of distribution shifts.
Specifically, it extracts the knowledge of any GNN teacher model trained on the augmented graphs and injects it into a partially parameter-shared student model that is tested on the original graph.
As a general framework, KDGA can significantly improve the vanilla implementations of various popular graph augmentation methods and GNN architectures.
Our contributions are summarized as follows: (1) We are the first to identify a potential negative augmentation problem for graph augmentation, and more importantly, we have described in detail what it represents, how it arises, what impact it has, and how to deal with it.
(2) We proposes a novel
Knowledge Distillation for Graph Augmentation (KDGA) framework for the identified problem by directly distilling contextual information from augmented graphs.
(3) We provide comprehensive experimental results showing that KDGA is applicable to a variety of graph augmentation methods and GNN models; it substantially outperforms the vanilla implementations across various datasets.
Background and Related Work
Structure Augmentation for Graphs.
Data augmentation is an effective technique to improve generalization.
Despite the great progress on node feature augmentation [56, 19, 15], comparatively little work study graph (structure) augmentation [35, 2, 62, 32, 6] due to the non-Euclidean property of structures.
For graph data, the mainstream algorithms for structure augmentation are divided into two categories: heuristic and learning-based.
As a typical heuristic algorithm, DropEdge [35] randomly remove edges according to the hand-crafted probability.
In a similar way, AdaEdge [2] iteratively adds (removes) edges between nodes predicted to have the same (different) labels.
Different from the above heuristic methods, GAUG [62] propose to optimize the graph augmentation and learnable GNN parameters in an end-to-end manner.
Instead, MH-Aug [32] proposes a sampling-based augmentation, where a sequence of augmented graphs are directly drawn from an explicit target distribution.
Graph Structure Learning and Graph Contrastive Learning.
Two closely related topics to graph augmentation are Graph Structure Learning [18, 23, 58, 20, 3] and Graph Contrastive Learning [19, 15, 57, 51, 63, 52], but they are quite different in terms of learning objectives and evaluation protocols .
The learning goal of structure learning is to estimate a new structure with high quality [10, 8].
Instead, graph augmentation aims to reasonably perturb the graph structure during training to produce a set of augmented graphs, enabling nodes to receive richer contextual information; such augmentations allow the model to generalize better across those variations.
As for the evaluation protocol, the augmented graphs are only used during training and are not available during testing.
In contrast, for graph structure learning, the learned structure is used during both training and testing.
There are also some recent works [41, 65, 55] exploring how to perform data augmentation for graph contrastive learning, but they focus on automatically selecting the most appropriate transformations from a given pool to improve contrastive learning, rather than learning customized augmentation strategies for GNNs.
More importantly, graph contrastive learning aims to learn transferable knowledge from abundant unlabeled data in an unsupervised setting and then generalize the learned knowledge to downstream tasks.
Instead, graph augmentation usually works in a semi-supervised setting, i.e., the label information is available during training.
The graph structure learning and contrastive learning are not newly born topics, and we refer readers to the recent surveys [49, 64].
Preliminaries
Notions.
Given a graph G =
( V , E ) , where V is the set of N = |V| nodes with features X
= [ x 1 , x 2 , · · · , x N ] ∈ R N × d and E denotes the edge set.
Each node v i ∈ V is associated with a d -dimensional features vector x i , and each edge e i,j ∈ E denotes a connection between node v i and v j .
The graph structure can also be denoted by an adjacency matrix A ∈ [0 , 1] N × N with A i,j =if e i,j ∈ E and A
i,j = 0 if e i,j / ∈ E . Consider a semi-supervised node classification task where only a subset of node V L with corresponding labels Y L are known, we denote the labeled set as
D L =
( V L , Y L ) and unlabeled set as D U
=
( V U , Y U ) , where V U = V\V L .
The node classification task aims to learn a mapping Φ : V → Y on labeled data D L , so that it can be used to infer labels Y U .
Background on Graph Homophily Ratio .
The homophily ratio is an important graph property that reflects the extent to which the graph structure adheres to the "label smoothness" criterion.
The graph homophily ratio r can be defined as the fraction of intra-class edges in the graph, as follows
[Equation]
where y i and y j are the ground-truth labels of node v i and v j .
In practice, the distribution space size of a discrete graph structure A ∈ [0 , 1] N × N is 2 N 2 , making it tractable to directly estimate the distribution differences between two discrete graph structures.
In this paper, we take the graph homophily as a desirable option to analyze the distribution shifts between the original and augmented graphs, thus measuring the severity of an algorithm suffering from the negative augmentation problem.
Methodology
In this section, we first make problem statements for graph augmentation in Sec.
4.1, highlight our motivations by analyzing the distribution shift between the original and augmented graphs in Sec.
4.2, then present a novel teacher-student Knowledge Distillation for Graph Augmentation (KDGA) framework in Sec. 4.3, and finally provide one of its specific instantiations in Sec. 4.4.
4.1 Problem Statement
Graph Representation Learning.
From the perspective of statistical learning, the key of node classification is to learn a mapping p ( Y | X , A ) based on node features X and graph structure A .
The learned mapping can be used to infer labels Y U on the graph structure A as shown in Fig. 1(a).
Graph Structure Learning.
The goal of graph structure learning is to estimate a more accurate structure
(cid:98) A by another mapping p ( (cid:98) A | X , A ) and then feed it into the mapping p ( Y | X , (cid:98) A ) along with node features X .
Finally, the learned mapping p
( Y | X , (cid:98) A ) can be used to infer labels Y U on the estimated (high-quality) structure (cid:98)
A instead of the original strucute A as shown in Fig. 1(b).
Graph Augmentation.
Instead of directly working with the original graph, we would like to leverage graph augmentation to reasonably perturb the graph structure and learn more generalizable representations.
In other words, we are interested in the following variant, as follows
[Equation]
where (cid:98) A ∈ [0 , 1] N × N is the augmented graph (structure).
In practice, the distribution space size of (cid:98) A is 2 N 2 , and it is intractable to enumerate all possible (cid:98)
A as well as estimate the exact values of the mappings p
( Y | X , (cid:98) A ) and p ( (cid:98)
A | X , A ) .
Therefore, we approximate them by tractable functions as
[Equation]
where q θ ( · ) and q ϕ ( · ) are approximation functions for p ( Y | X , (cid:98) A ) and p ( (cid:98) A
| X , A ) parameterized by θ and ϕ , respectively.
In practice, the function q θ ( Y | X , (cid:98) A ) can be generally implemented by
[Figure]
GNNs, and the q ϕ (
A | X , A ) can be implemented by graph augmentation methods to model the distributions of augmented graph structures.
Once the model training is finished, the mapping q θ ( Y | X , A ) can be used to infer labels Y U on the original structure A as shown in Fig.
1(c).
In summary, unlike graph representation and graph structure learning that leverage the same structure ( A or (cid:98) A ) for both training and testing, the graph structures for training and testing in graph augmentation are completely different, which may lead to a potential negative augmentation problem.
4.2 Motivation: Potential Negative Augmentation Problem
One may create a model by specifying specific implementations for functions q θ ( Y | X , (cid:98) A ) and q ϕ ( (cid:98)
A | X , A ) and then optimize it by maximizing the posterior p ( Y | X , A ) defined in Eq.
As we will explain here, however, this model may suffer from a potential negative augmentation problem caused by overly severe distribution shifts between the original and augmented graphs
[Figure] [Figure]
[Figure]
H o m o p h il y r
a t
i
o
o n A u g m e n t e d
G r a p h
[Figure]
CoraCiteseerCornellWisconsin
TexasActorChameleonSquirrel
[Figure]
The distribution shift itself is not necessarily harmful; it is actually a neutral phenomenon.
A proper distribution shift helps the model “see" more different graphs, enabling the nodes to receive more contextual information, thus improving generalization; however, an overly severe distribution shift can lead to a potential negative augmentation problem.
To illustrate it, we consider a node v i (id 129) of class C 1 from the real-world Wisconsin dataset in Fig. 2(a), it is initially connected to a node with the same class C 1 and three nodes from another class C 2 in the original structure A .
During the training process, the original structure A and node features X are fed together into q ϕ ( (cid:98) A | X , A ) to generate an augmented structure (cid:98) A .
Under the downstream supervision, it disconnects from three nodes from class C 2 and reconnects with three nodes from the same class C 1 , resulting in an augmented structure (cid:98) A with a much higher homophily ratio, that is, an overly severe distribution shift between the original and augmented graphs from the perspective of graph homophily property.
As a result, a model trained on the augmented structure (cid:98) A can successfully predict node i as class C 1 , but make a wrong prediction C 2 for node i when tested on the original structure A , which is termed as "negative augmentation" .
Furthermore, we plot the homophily ratios of the original structure A and augmented structure (cid:98)
A on eight datasets in Fig.
2(b), from which we can observe significant distribution shifts between the two graphs.
Moreover, while the above analysis is developed on a heterophily (Wisconsin) graph, we find that the identified distributional shift also exists in homophily graphs, only in a different direction.
Please see Sec.
5.3 for detailed experimental settings and results.
4.3 Knowledge Distillation for Graph Augmentation (KDGA)
The distribution shift is essentially a trade-off between better generalizability and higher risks of negative augmentation.
However, the optimal distribution shift may vary from dataset to dataset, or even from node to node, making it challenging to directly control the levels of distribution shifts.
In this paper, we have not attempted to control or prevent distribution shifts.
Instead, we allow for the existence of any level of distribution shifts, but we reduce their negative impact, i.e., the potential negative augmentation problem, by the proposed KDGA framework, which gradually distills the contextual information from the augmented graphs into a student model tested on the original graph.
The idea of KDGA is straightforward, yet as we will see, extremely effective.
In our case, we first generate soft distributions z Ti and z Si for node v i with the teacher and student models, respectively.
The knowledge distillation is first introduced in [14], where knowledge was transferred from a cumbersome teacher to a simpler student by optimizing the following objective function, as follows
[Equation]
In this paper, not to get a simpler student model, we adopt the knowledge distillation framework to address the identified negative augmentation problem caused by overly severe distribution shifts between the two graphs.
In short, we extract the knowledge of any teacher model trained on the augmented graphs and inject it into a student model tested on the original graph as in Fig. 1(d).
Teacher Model.
The teacher model can be implemented by any GNN, which takes node features X and augmented structure (cid:98) A as input and learn latent node representations via neighborhood feature aggregation.
Considering a L -layer GNN f θ ( X , (cid:98) A ) , the formulation of the l -th layer is as follows
[Equation]
where 0 ≤ l ≤ L − 1 , h (0) i,T = x i is the input feature, and N (cid:98)
A i is the neighborhood of node v i in the augmented structure (cid:98) A .
After L message-passing layers, the final node embedding h ( L ) i,T is passed to a linear prediction head g T ( · ) to obtain logits z T i
= g T ( h ( L ) i,T ) , and the model is trained by a crossentropy loss H ( · ) with ground-truth labels Y L , given by L TSUP =
(cid:80) i ∈V L H (cid:0)
y i ; softmax (cid:0)
z Ti (cid:1)(cid:1) .
Student Model.
The student model f θ ( X , A ) shares the parameters θ with the teacher model, but differs in that the it takes the original structure A as input, as shown in Fig. 3(c).
Besides, an additional linear prediction head g S ( · ) is used to map the node embedding h ( L ) i,S to logits z Si = g S ( h ( L ) i,S ) .
As already explained earlier, the augmented graphs enable the teacher model to receive richer contextual information, which helps to improve model generalization.
To allow the student model tested on the original structure to also benefit from it, we consider the contextual neighborhood information from both original and augmented structures and distill them into the student model, defined as
[Equation]
6) where τ 1 is the distillation temperature, and τ 21 is used to keep the gradient stability of this loss [14].
Discussions.
While a large number of methods on graph knowledge distillation [61, 53, 60] have been proposed, most of them adopt the standard teacher-student knowledge distillation framework as shown in Fig. 3(a), where the inputs to both teacher and student models are the same (structure).
Despite many progresses, their contributions have mostly focused on the special design of the teacher or student models.
For example, CPF [54] proposes to distill knowledge from a teacher GNN to a student MLP, but it specifically incorporates label propagation [16] into the student model to improve performance.
In contrast, GDK [11] utilizes label propagation in the teacher model to fully exploit both feature and topological information.
In our proposed KDGA framework, the graph structures fed to the teacher and student models are completely different.
Moreover, unlike the scheme in Fig. 3(b) where two parameter-independent teacher and student models are used, we adopt the architecture shown in Fig. 3(c) where the GNN parameters are shared but with two independent prediction heads to increase discriminability.
The behind motivation is that a parameter-independent student model has the risk of quickly fitting with the original structure under the optimization of downstream supervision, while failing to take full advantage of rich contextual information from the augmented graphs.
A detailed comparison of parameter-independent and parameter-shared schemes is reported in Table.
[Figure]
4.4 Instantiating KDGA with GraphAug
In practice, any existing graph augmentation method can be used to instantiate the proposed KDGA framework and achieve consistent improvements over the vanilla implementations, as shown in
Table.
In this subsection, we adopt a probabilistic generative-based graph augmentation method to model the function q ϕ ( (cid:98)
A | X , A ) , termed GraphAug, and use it to instantiate our KDGA framework.
Specifically, we introduce a set of discrete variables Λ = { λ i,j } Ni,j =1 to model the distribution of the augmented graph, where λ i,j ∈ { 0 , 1 } denotes the augmentation probability between node v i and v j .
Moreover, we avoid estimating the probability p ( λ i,j | µ i,j ) using independent local parameter µ i,j and instead fits a shared neural network to estimate it.
Specifically, we first transform the input to a low-dimensional hidden space, done by multiplying the node features with a parameter matrix
W ∈ R F × d , that is, e i =
W x i .
Then, we directly parameterize the probability λ i,j as
[Equation]
where σ ( · ) is an element-wise sigmoid function.
Next, to sample discrete augmented graphs from the learned augmentation distribution and make the sampling process differentiable, we adopt GumbelSoftmax sampling [17].
Specifically, the sampling process can be formulated as
[Equation]
, where M i,j
[Equation]
αp
[Equation]
i
[Equation]
j
[Equation]
X
[Equation]
A
[Equation]
1 − α
[Equation]
A i
[Equation]
j
[Equation]
8) where α ∈ [0 , 1] is the fusion factor to control the intensity of the graph augmentation, τ 2 is the augmentation temperature, and G ∼ Gumbel (0 , 1) is a gumbel random variate.
To warm-up the proposed GraphAug module, we first pre-train it with loss L Aug = 1 N 2 H ( A i,j , (cid:98)
A i,j ) , where H ( · ) denotes the cross-entropy loss.
Besides, we use classification loss L Cla =
|V L | (cid:80) i ∈V L (cid:0) H (cid:0)
y i , softmax ( z Ti ) (cid:1)
+ H (cid:0)
y i , softmax ( z Si ) (cid:1) (cid:1) to pre-train the teacher and student models until it converges.
Finally, the total loss to train the whole framework is defined as follows
[Equation]
where λ and κ are the weights to balance the influence of the two losses L Aug and L GKD .
[Table]
Experiments
Datasets.
The effectiveness of the proposed KDGA framework is evaluated on eight datasets.
We use two commonly used homophily graph datasets, namely Cora [38] and Citeseer [12] as well as six heterophily graph datasets: Cornell, Texas, Wisconsin , Aactor [42], Chameleon and Squirrel [37].
A statistical overview of these datasets is available in Appendix A .
We defer the implementation details and the best hyperparameter settings for each dataset to Appendix B and supplementary material.
Baselines.
As a general framework, KDGA can be combined with any GNN architecture and existing graph augmentation methods.
In this paper, we consider three GNN architectures, GCN [21], GraphSAGE [13], and GAT [43].
Besides, to demonstrate the applicability of KDGA to various graph augmentation methods in addition to the proposed GraphAug, we also consider two stateof-the-art learning-based baselines, GAUG [62] and MH-Aug [32].
In particular, two heuristics methods, DropEdge and AdaEdge, are also included in the comparison as baselines.
Moreover, we also compare KDGA with two semi-supervised methods: (1) GraphMix [44], a regularization method that performs linear interpolation between two data on graphs, and (2) SSL [63], that proposes two self-supervised tasks to fully exploit available information embedded in the graph structure.
Each set of experiments is run five times with different random seeds, and the average performance is reported.
5.1 Comparative Results
To evaluate the powerful capabilities of the proposed KDGA framework, we instantiate it with three learning-based graph augmentation methods, GAUG, MH-Aug, and GraphAug.
The experiments are conducted on eight datasets with three different GNN architectures.
From the experimental results shown in Table.
1, we can make the following observations: (1) Two heuristic graph augmentation methods, DropEdge and AdaEdge, can improve the performance of the vanilla GNNs overall.
However, such improvements are usually very limited and do not work for all datasets and GNN architectures.
For example, on the Citeseer dataset, the performance of AdaEdge drops over the vanilla GNNs by 1.9% (GCN), 1.7% (GraphSAGE), and 2.3% (GAT), respectively.
(2) There are huge gaps in the effectiveness of three learning-based augmentation methods on homophily and heterophily graphs.
While these methods can significantly improve performance on homophily graphs, their performance gains on heterophily graphs are greatly reduced and even detrimental.
For example, with GCN as the GNN architecture, the performance of GAUG improves by 2.1% on Cora, but drops by 1.5% and 1.2% on Texas and Cornell.
Such negative augmentation is mainly caused by the overly severe distribution shift between the original and augmented graphs as analyzed in
Sec. 4.2.
The proposed KDGA framework can consistently improve the performance of vanilla graph augmentation methods across three GNN architectures on all eight datasets, especially for those heterophily graphs.
For example, with GCN as the GNN architecture, the performance of GraphAug can be improved by 9.7% and 8.6% on the Squirrel and Texas datasets.
(4) Two semi-supervised approaches, SSL and GraphMix, can achieve comparable or even better performance than learningbased graph augmentation, especially on heterophily graphs.
However, by combining with KDGA,
GAUG, MH-Aug, and GraphAug outperform both SSL and GraphMix by a large margin overall.
[Figure]
5.2 Analysis on the Distribution Shift and Negative Augmentation
Next, we qualitatively and quantitatively analyze the distribution shift between the original and augmented graphs and explain how it can cause a potential negative augmentation.
Without loss of generality, we consider GCN as the GNN architecture and GraphAug as the augmentation method.
Visualizations of Neighborhood Variations.
First, we pick a node with the most drastic neighborhood variations and visualize its neighborhood of the original and augmented graphs in Fig.
4, where each node is colored according to its ground-truth label.
The visualizations show that there would be a huge gap between the neighborhoods of the original and augmented graphs, which causes a model that is well trained on the augmented graph to predict poorly on the original graph during testing.
Taking the Wisconsin dataset as an example, the selected node is connected to four nodes from the same class in the augmented graph, so it can be well trained to make correct predictions.
However, its neighborhood context is completely changed in the original graph, where the node is connected to three nodes from different classes, so it will be predicted with high confidence to an incorrect class.
Training Curves of Homophily Ratios.
We plot in Fig.
5 the training curves (w/o GKD Loss) of the homophily ratios of the original and augmented graphs during training.
It can be seen that their gaps are enlarged as training proceeds, which indicates that the distribution of the augmented graphs is gradually shifting from the original graph.
This shift may even reach 0.5 for some datasets (e.g.,
Texas), in which case the graph homophily property is completely reversed.
More importantly, we find that the direction of distribution shifts may be completely opposite for homophily and heterophily graphs, which makes it more challenging to solve the negative augmentation problem.
[Figure]
5.3 Ablation Study and Parameter Sensitivity
[Figure]
[Table]
Ablation on Student Model Designs.
The parameter-shared GNN shown in Fig. 3(c) is adopted as the student model by default in this paper for a fair comparison.
In this subsection, we delve into the applicability of the proposed KDGA framework to different student model designs.
Specifically, with the vanilla GCN as the base architecture and GraphAug as the graph augmentation method, we compare the performance of the parameter-shared model (w/ Param-S) in Fig. 3(c) and the parameter-independent model (w/ Param-I) in Fig. 3(b) on five datasets.
It can be seen from Table.
2 that although the Param-I model can also improve the performance of GraphAug overall, it may fail on a few datasets, such as a 0.5% accuracy drop on Citeseer; more importantly, its performance gain falls far behind the Param-S model on all five datasets.
The reason behind this may be that a parameter-independent model may be quickly fitted with the original graph structure while failing to take full advantage of the rich contextual information embedded in the augmented graphs.
Moreover, we also consider a variant of the Param-I model by directly taking a parameter-independent MLP (w/ MLP) as the student mode.
We find from
Table.
2 that even with a simple MLP, it can still benefit from the augmented graphs and achieves performance beyond that of its vanilla implementations.
Sensitivity Analysis on Hyperparameters.
We have evaluated the parameter sensitivity w.r.t two key hyperparameters: fusion factor α and loss weight κ .
However, due to space limitations, we have placed the corresponding results of sensitivity analysis in Appendix C .
Conclusion
In this paper, we identified a potential negative augmentation problem for graph augmentation, which is caused by overly severe distribution shifts between the original and augmented graphs.
To address this problem, we propose a novel Knowledge Distillation for Graph Augmentation (KDGA) framework by directly distilling contextual information from a teacher model trained on the augmented graphs into a partially parameter-shared student model.
Extensive experiments show that KDGA outperforms the vanilla implementations of existing augmentation methods and GNN architectures.
Limitations still exist, such as KDGA requires an initial raw graph structure for augmentation and cannot be applied to those structure-unknown scenarios, which will be left for future work.
Acknowledgement
This work is supported in part by Ministry of Science and Technology of the People’s Republic of
China (No. 2021YFA1301603) and National Natural Science Foundation of China (No. U21A20427).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We have read the guidelines and ensured that our paper conforms to them.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplementary material submitted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] The training details of the data splits and hyperparameter settings have been placed in Appendix A.1 and Appendix A.2 , respectively.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Each set of experiments is run five times with different random seeds, and the average accuracy and standard deviation are reported.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] The implementation uses the PyTorch library running on NVIDIA v
GPU, which is detailed in Appendix A.2 .
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] The assets in this paper come from some open-source platforms, such as Github, and follow the MIT license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] It does not contain such sensitive information.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/ A]
Appendix A Dataset Statistics
Eight publicly available graph datasets are used to evaluate the proposed KDGA framework.
An overview summary of the statistical characteristics of datasets is given in Tab.
A1.
Cora, Citeseet, and Pubmed.
These three are citation network benchmark datasets for node classification.
In these datasets, nodes represent papers, and edges denote citations of one paper by another.
Node features are the bag-of-words representation of papers, and the node label is the academic topic of a paper.
The data splittings of these datasets are the same as [21].
Cornell, Texas, and Wisconsin.
Cornell, Texas, and Wisconsin 1 are three sub-datasets of WebKB, which is a webpage dataset collected from computer science departments of various universities by
Carnegie Mellon University.
In these datasets, nodes represent web pages, and edges are hyperlinks between them.
Node features are the bag-of-words representation of web pages.
The nodes are manually classified into five categories: student, project, course, staff, and faculty.
Actor.
This dataset is a subgraph of the film-director-actor-writer network.
In this dataset, nodes represent actors, and edges are their co-occurrence on the same Wikipedia page.
Node features are some keywords in the Wikipedia pages.
The nodes are manually classified into five categories in terms of the words of the actor’s Wikipedia.
Chameleon and Squirrel.
Chameleon and Squirrel are two page-page networks on specific topics in
Wikipedia.
In these datasets, nodes represent web pages, and edges are mutual links between pages.
Node features are several informative nouns in the Wikipedia pages.
The nodes are classified into five categories in terms of the number of the average monthly traffic of the web page.
[Table]
B Hyperparameters and Search Space
[Table]
All baselines and our approach are implemented based on the standard implementation in the DGL library [45] using the PyTorch 1.6.0 library with Intel(R) Xeon(R)
Gold 6240R @ 2.40GHz CPU and NVIDIA
V100 GPU.
The following hyperparameters are set for all datasets: weight decay decay = 5e-4; Maximum Epoch E = 500; Layer number L = 2, sampling temperature τ 2 = 1.0.
The other dataset-specific hyperparameters are determined by a hyperparameter search tool - NNI for each dataset, including hidden dimension F , learning rate lr , loss weight λ and κ , fusion factor α , and distillation temperature τ 1 .
The hyperparameter search space is shown in Tab. A2, and the model with the highest validation accuracy is selected for testing.
The best hyperparameter choices are available in the supplementary.
C. Parameter Sensitivity Analysis
We have evaluated the parameter sensitivity w.r.t two key hyperparameters: fusion factor α and loss weight κ , and results are reported in Fig.
A1.
In practice, we can determine fusion factor α and loss weight κ by selecting the model with the highest accuracy on the validation set.
[Figure]
Fusion Factor α .
The parameter sensitivity w.r.t the fusion factor α defined in Eq.
(8) is reported in Fig.
1(a), from which we can observe that (1) When α is set to a small value, the performance gains for all four datasets are reduced, due to the insufficient contextual variations in the augmented graphs.
(2) When α is set to a large value, different types of graphs show different hyperparameter sensitivity to α .
For homophily graphs, such as Cora and Citeseer datasets, a too-large value of α hurts performance, as it may cause overly severe distribution shifts in the direction of reducing the homophily ratio.
For heterophily graphs, such as Texas and Wisconsin datasets, they generally reach the best performance at α = 1 , where the augmented graph tends to exhibit a higher homophily ratio than the original graph.
The difference in the parameter sensitivity of these two types of graphs comes mainly from the difference in their directions of the distribution shifts, as shown in Fig.
Loss Weight κ .
As can be observed from Fig. 1(b), the loss weight κ plays a critical role in the
KDGA framework.
If we set the loss weight κ = 0 , i.e., completely remove the GKD loss, the model performance will deteriorate to be the same as the vanilla implementations, resulting in the poorest performance compared to other settings.
In practice, we find that setting κ to a non-zero value, i.e., training with GKD loss, always achieves better performance than training without GKD loss (setting κ = 0 ), which demonstrates the effectiveness of the proposed GKD loss.
Moreover, we find that the model performance can be further improved as the loss weight κ increases, but the performance gains reduce when κ becomes too large, probably because a too large GKD loss weight κ tends to weaken the contribution of label information (i.e., the loss of L cla ) in the semi-supervised learning.