L OCAL A UGMENTATION FOR G RAPH N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Data augmentation has been widely used in image data and linguistic data but remains under-explored on graph-structured data.
Existing methods focus on augmenting the graph data from a global perspective and largely fall into two genres: structural manipulation and adversarial training with feature noise injection.
However, the structural manipulation approach suffers information loss issues while the adversarial training approach may downgrade the feature quality by injecting noise.
In this work, we introduce the local augmentation, which enhances node features by its local subgraph structures.
Speciﬁcally, we model the data augmentation as a feature generation process.
Given the central node’s feature, our local augmentation approach learns the conditional distribution of its neighbors’ features and generates the neighbors’ optimal feature to boost the performance of downstream tasks.
Based on the local augmentation, we further design a novel framework:
LA-GNN, which can apply to any GNN models in a plug-and-play manner.
Extensive experiments and analyses show that local augmentation consistently yields performance improvement for various GNN architectures across a diverse set of benchmarks.
I NTRODUCTION
Graph Neural Networks (GNNs) and their variants (Abu-El-Haija et al., 2019; Kipf & Welling, 2017; Veli ˇ ckovi ´ c et al., 2018) have achieved state-of-the-art performance on various tasks such as recommendation system (Ying et al., 2018) and trafﬁc prediction (Guo et al., 2019).
However, as Kipf & Welling (2017) points out, deep GCN are prone to over-ﬁtting due to limited labeled data (Shorten & Khoshgoftaar, 2019) and additional model parameters as the number of parameters increase with model depth.
Meanwhile, as a data-based solution of alleviating over-ﬁtting, data augmentation has been successfully adopted in many areas, such as computer vision (Cubuk et al., 2019) and natural language processing (Fadaee et al., 2017).
Therefore, graph data augmentation has been studied recently.
For explicit graph data augmentation, the existing methods can be divided into two categories: either topology-level (Rong et al., 2020;
Wang et al., 2020b; Zhao et al., 2021) or feature-level augmentation (Deng et al., 2019; Feng et al., 2019; Kong et al., 2020).
A standard operation on topology-level augmentation is to perturb the adjacency matrix, yielding different graph structures.
For instance, DropEdge (Rong et al., 2020) randomly removes a certain percentage of edges of the input graph for each training epoch and can be interpreted as a topologylevel augmenter.
NodeAug (Wang et al., 2020b) adds and removes edges on subgraphs based on their self-deﬁned importance score.
Zhao et al.
(2021) exploit their proposed neural edge predictor to promote intra-class edges and demote inter-class edges in a given graph structure.
While featurelevel augmentation recovers missing or interrupted vertex features using their context information.
Existing feature-level augmentation mainly exploits perturbations guided by adversarial training.
Some studies (Deng et al., 2019; Feng et al., 2019; Kong et al., 2020) inject adversarial noise to node attributes and regularize the training of GNNs.
Nevertheless, these feature-level augmentation techniques have three drawbacks.
They employ full-batch adversarial training for augmentation, which is unnecessary, and introduce some additional side effects such as over-smoothing (Li et al., 2018).
This type of feature-level augmentation is coarse-grained, which focuses on global augmentation and abandons the local information of the neighborhood. 3)
Injecting the perturbation into features degrades the quality of the feature representations.
Moreover, to our best knowledge,
[Figure]
none of the existing approaches combines both the feature representations and the graph topology, especially the local subgraph structures, for graph-level data augmentation.
In this work, to consider both structure and features, we propose a framework: L ocal A ugmentation for G raph N eural N etworks (LA-GNNs).
The term "local augmentation" refers to the generation of neighborhood information by conditioning on local structures and node features via a generative model.
Speciﬁcally, our proposed framework learns the conditional distribution of the connected neighbors’ representations given the representation of the central node, which has some similarities with the Skip-gram (Mikolov et al., 2013) and Deepwalk Perozzi et al. But our method does not focus on word or graph embedding.
The motivation behind this work concludes three-fold. 1)
Existing feature-level augmentation works primarily pay attention to global augmentation without considering the informative neighborhood.
The distribution of the representations of the neighbors has a close connection to the central node, making ample room for feature augmentation.
JKnet (Xu et al., 2018), APPNP (Klicpera et al., 2019) and GCNII (Chen et al., 2020) preserve the locality of the node representations to avoid the over-smoothing, yielding the demand of local augmentation.
And there are several beneﬁts in applying local augmentation for the GNN training.
First, local augmentation is essentially a data augmentation technique that can improve the generalization of the GNN models and prevent over-ﬁtting.
Second, we can recover some missing or interrupted feature information in an attributed graph via the generative model (Jia & Benson, 2020).
Third, our proposed framework is ﬂexible and can be equipped with various popular backbone networks such as GCN (Kipf & Welling, 2017), GAT (Veli ˇ ckovi ´ c et al., 2018), GCNII (Chen et al., 2020), and GRAND (Feng et al., 2020) to enhance their performance.
Extensive experimental results demonstrate that our proposed framework could improve the performance of GNN variants on benchmark datasets.
L OCAL A UGMENTATION
The local augmentation framework we proposed consists of three modules: the generator, the importance sampling module, and GNN as an auxiliary downstream task, as illustrated in Figure 1.
For each node, the pipeline utilizes the generative model to learn the conditional distribution of the feature representations of its connected neighbors.
We then exploit importance sampling to determine the optimal generated data to augment the training set, according to the dropout uncertainty of the downstream GNN predictor, which is subsequently trained on the augmented dataset.
2. N OTATIONS AND P RELIMINARIES
Notations.
Let G =
( V, E ) represent the graph, where V is the set of vertices { v 1 , · · · , v N } with | V | =
N and E is the set of edges.
The adjacency matrix is deﬁned as A ∈ { 0 , 1 } N × N , and A ij = 1 if and only if ( v i , v j ) ∈ E .
Let N i
= { v j |
A ij = 1 } denotes the neighborhood of node v i and D denote the diagonal degree matrix, where D ii = (cid:80) nj
=1 A ij .
The feature matrix is denoted as X ∈ R N × F where each node v is associated with a F -dimensional feature vector
X v .
Y ∈ { 0 , 1 } N × C denote the one-hot label matrix, where Y i ∈ { 0 , 1 } C is a one-hot vector and (cid:80)
Cj =1 Y ij = 1 for any v i ∈ V .
GNN.
Graph Neural Networks (GNNs) are neural network models, such as GCN and GAT (Kipf & Welling, 2017; Veliˇckovi´c et al., 2018), that capture the dependence of graphs via message passing between the nodes of a graph as
[Equation]
where f denotes the speciﬁc GNN layer for different models, H ( (cid:96) ) are the hidden vectors of the (cid:96) -th layer and H (0)
=
X .
For example, f ( A , H )
=
σ ( ˆ AHW ) for GCN, where ˆ A = ˜ D − 12 ˜
A ˜ D − 12 , ˜ D is the degree matrix of ˜ A , i.e., ˜ D ii
=
(cid:80)
j ˜
A ij , and ˜ A
=
A + I .
L EARNING T HE C ONDITIONAL D ISTRIBUTION
We start by reviewing the semi-supervised learning of GNNs.
Most existing GNN models (Kipf &
Welling, 2017; Veliˇckovi´c et al., 2018) are viewed as a classiﬁcation function to predict the class labels of the graph nodes.
In this work, we use a GNN classiﬁcation estimator P θ ( Y | A , X ) ( θ is the parameter) to model the conditional distribution of label Y with respect to the graph structure A and feature matrix X .
Given training samples { A , X , Y } , the parameter θ can be estimated using
Maximum Likelihood Estimation (MLE), by optimizing the following likelihood function:
[Equation]
where K is the set of node indices of the training dataset whose labels are visible during the semisupervised training.
To further boost the performance of GNN, a new model P θ ( Y , X | A , X ) is introduced, where X is generated features by feature-level augmentation (Kong et al., 2020; Zhu et al., 2020).
For this model, the MLE method needs to optimize a marginalized probability P θ over the generated feature matrix X :
[Equation]
For Bayesian tractability, we decompose P θ in Eq.(3) as a product of two posterior probabilities:
[Equation]
where P θ
( Y k | A , X ) and Q φ ( X | A , X ) denote the probabilistic distributions approximated by the downstream GNN and the (feature-level augmentation) generator respectively, parameterized by θ and φ .
There are two beneﬁts in the decomposition above.
First, it allows us to decouple the training of the downstream predictor P θ and the generator Q φ , enabling the generator to easily generalize to other downstream tasks.
Moreover, inspired by the successes of data augmentation via deep-learning-based generative modeling (Antoniou et al., 2017), the representation power of Eq.(4) is superior than that of a single predictor P
θ ( Y k | A , X ) without data augmentation.
Consequently, once a generator Q φ is trained very well, our training procedure can optimize
P θ ( Y k | A , X ) with samples X drawn from the ﬁxed conditional distribution Q φ .
Now, we show how to train the generator as follows.
Generator To learn a feature augmentation generator, a naive solution is to learn one single distribution for all the neighbors using the MLE method, i.e., solving the following optimization problem
[Equation]
where { X j | j ∈N i , X i } .
Then p ψ can be used to augment features for all the neighbors.
However, this method ignores the differences between all the neighbors, which may induce severe noise.
To overcome the limitation, we assume that each neighbor satisﬁes a different conditional distribution.
Speciﬁcally, there exists a conditional distribution p ( ·| X i , z j ) with latent random variable z j , such that we have X j ∼ p ( X | X i , z j ) for X j | j ∈N i .
Once we obtain p ( ·| X i , z j ) in some way, we can generate augmented features X , and then we can train P θ ( Y k | A , X ) instead of P θ ( Y k | A , X ) to improve the ﬁnal performance of P θ .
Below, we will present how to ﬁnd p ( ·| X i , z j ) , which will produce the generator Q φ .
X !"
X !
X $ X !
% (a) The Original Graph
X !"
X !
X $ X !
r e c o n s t u c t r e c o n s t u c t
X* $ loss (b) EP-B
X !"
X !
X $ X !
s i m i l a r
s
i m i l a r d i ss
i m i l a r (c) GraphSAGE
X !"
X !
X $ X !
% i n f e r i n f e r (d) Local Augmentation
To achieve our purpose, a suitable method is the conditional variational auto-encoder (CVAE) (Kingma & Welling, 2013; Sohn et al., 2015), which can help learn the distribution of the latent variable z j , and the conditional distribution p ( ·| X i , z j ) .
So, a CVAE model Q φ (cid:0) X | A , X (cid:1) is adopted as our generator, where φ = { ϕ, ψ } , ϕ denotes the variational parameters and ψ represents the generative parameters.
To derive the optimization problem for CVAE, log p ψ ( X j | X i ) can be written with latent variables z as follows, following previous work (Pandey & Dukkipati, 2017; Sohn et al., 2015):
[Equation]
and the evidence lower bound (ELBO) can be written as:
[Equation]
where the encoder q ϕ ( z | X j , X i ) =
N ( f ( X j , X i ) , g ( X j , X i )) and decoder p ψ ( X j | X i , z ) =
N ( h ( X i , z ) , cI ) .
The encoder is a two-layer MLP.
f
and g share the ﬁrst layer, and their second layers use different parameters.
The decoder h is two-layer MLP.
For simplicity and tractability, the implemented generator Q (cid:0) X | A , X (cid:1) uses the same parameters across all nodes v i ∈ V .
Here, we discuss how our proposed model distinguishes from the classical representation learning models on graphs.
Previous methods such as EP-B (García-Durán & Niepert, 2017) and
GraphSAGE (Hamilton et al., 2017) rely on reconstruction loss function between the central node and its neighbors’ embeddings.
EP-B aims to minimize the reconstruction error by optimizing the objective min (cid:80) u ∈ V \{ v } (cid:104) γ + d ( (cid:101)
X v , X v ) − d
(cid:101) X v , X u ) (cid:105) where X v represents the target node;
X u denotes the neighbor nodes; (cid:101)
X v =
AGG( X l | l ∈ N ( v ) ) indicates the reconstruction from neighbors; and γ refers to the bias.
Besides, GraphSAGE exploits the negative sampling to differentiate the representations of remote node-pairs.
GraphSAGE enforce nearby nodes to have similar representations and to enforce disparate nodes to be distinct by minimizing the objective min − E u ∼N ( v ) log (cid:0) ( σ ( X Tu X v ))
(cid:1) − λE v n ∼ P n ( v ) log (cid:0) ( σ ( − X Tv n X v )) (cid:1) where X v denotes target node; X u represents the neighbor node; X v n is disparate node; and P n ( v ) is the negative sampling.
These approaches build upon the assumption that adjacent nodes share similar attributes.
In contrast, our model does not rely on such assumption and instead generates the neighboring node features from the conditional distribution of central node representations.
Given the target node, X v , our aim is to learn the conditional distribution of the neighbor nodes, X u .
A comparison between the reconstruction-based representation learning on graphs and our proposed framework is illustrated in
Figure 2.
Optimization of the MLE Now, we present how to optimize the MLE Eq.(4) using the feature matrix produced from the generator.
Once the augmented feature matrix can be sampled from the generator, we can optimize the parameters of Eq.(4) in the following way.
Firstly, the parameter φ = { ψ, ϕ } can be optimized by maximizing the ELBO of the generator (6), i.e., we train the generator.
Secondly, the parameter θ is optimized by maximizing the MLE Eq.(4) with φ ﬁxed, which is the conditional distribution of Y k given A and X .
In this way, we iteratively and alternately optimize the conditional MLE and ELBO.
In this paper, the MLE is formulated by a GNN model as follows:
[Equation]
where L ( θ | A , X , φ )
= − (cid:80)
k ∈ T (cid:80)
Cf =1 Y kf ln (cid:16) softmax (cid:0) GNN( A , X ) (cid:1) kf (cid:17) .
In addition, a practical trick is introduced which ﬁrstly trains GNN parameter θ with A and the initial feature matrix
X for some iterations, and then updates it using A and the augmented feature matrix X .
Later, section 2.3 will further present a close relationship between the inference of the generator and the optimization of the MLE.
2. A CTIVE L EARNING
After the training of the generator ﬁnishes, it contains an issue of using Eq.(4) for inference because
Q may generate some rare samples from the side part of the distribution.
This critical question makes the inferences Eq.(4) inefﬁcient.
Inspired by the active learning (Nielsen & Okoniewski, 2019), we introduce importance sampling to capture the suitable generated feature matrix and the corresponding generator, which improves the inference efﬁciency and accelerates the optimization of the MLE.
During the importance sampling, the probability of each feature is proportional to its uncertainty evaluated by an acquisition function.
We adopt the Bayesian Active Learning by Disagreement (BALD) acquisition function (Houlsby et al., 2011) to sample the most important inferences with the approximation from the Monte Carlo (MC) dropout samples as
[Equation]
where N is the number of MC samples and ω n are the parameters of the network sampled for the n -th MC dropout sample.
A high BLAD score indicates a network with high uncertainty about the generated feature matrix.
So it tends to be selected to improve the GNN model.
Finally, the overall algorithm framework is summarized in Algorithm 2.3, which assists the optimization of Eq.(4) and generates a suitable generated feature matrix and the corresponding generator for downstream tasks.
2. T HE A RCHITECTURE OF LA-GNN
In this section, we discuss the details of how to modify backbone models to exploit the augmented feature matrix produced by the generator.
We use GCN, GAT, GCNII, and GRAND as the backbone network of the proposed framework and test them on semi-supervised node classiﬁcation tasks.
When the above steps ﬁnish, both the initial data and the generated feature matrix are useful for the training of the backbone networks.
Therefore, we mix the initial and generated feature representations using concatenation operation.
Then we feed the mixed representation to the next GNN layer to obtain our new model, named LA-GNN, where LA means local augmentation.
Speciﬁcally, a 2-layer
LA-GCN is deﬁned as follows:
[Equation]
where X i ( i = 1 , 2 , · · · , n ) is the augmented feature matrix produced by the generator, (cid:107) denotes an operator of column-wise concatenation, W (1)
i
( i = 1 , 2 , · · · , n ) denotes the parameters of the ﬁrst
LA-GCN layer, and W (2) denotes the parameters of the second LA-GCN layer.
Since GCNII (Chen et al., 2020) applies a fully-connected neural network on X to obtain a lower-dimensional initial representation H (0) before the forward propagation, we apply a fully-connected neural network on X and X to obtain H (0) for LA-GCNII as follows:
[Equation]
H (0) is fed into the next forward propagation layer.
LA-GAT has a similar network architecture with
LA-GCN, but the graph convolutional layer needs to be replaced by the graph attention layer.
2. D ISCUSSION
In this section, we discuss the motivation of this work and provide some analysis.
Local Augmentation vs. General Augmentation General image augmentation algorithms include geometric transformations, feature space augmentation, adversarial training, and generative adversarial networks (Shorten & Khoshgoftaar, 2019).
It is impossible to apply geometric transformations directly to graph data augmentation since graphs are sensitive to node permutation.
General adversarial training, feature space augmentation, and generative adversarial networks don’t take the graph structure into account.
Graphs consist of a set of identities with certain pairs of these identities connected by edges.
We need to consider node features and the graph structure when designing the graph data augmentation framework.
Our proposed method of local augmentation fully considers these two points.
By extracting the neighbors’ feature vectors, we have enough data points to learn the distribution.
There are two beneﬁts to designing local augmentation.
First, by taking the sub-graph structure and feature representation associated with this sub-graph structure as input for the generative model, we can learn the information of the sub-graph structure.
Second, the number of data points to learn the distribution depends on the node degree.
This assures that we have enough data points compared with the general feature augmentation and we can learn a better distribution.
Complementing missing information Jia & Benson (2020) points out that some attribute information might be missing on a subset of vertices.
By learning the distribution of node representations from the observed data, we can utilize the produced node representations from the generative model to complement the information missing in the nodes’ attributes, which boosts the robustness of downstream tasks.
And we show that our model still works in the scenario that nodes lose a certain percentage of attributes.
R ELATED W ORK
Graph Neural Networks In general, convolution in the graph domain involves non-spectral (spatial) and spectral approaches.
Non-spectral methods generalize convolutions operating on spatially close neighbors to the graph domain.
Duvenaud et al.
(2015) deﬁne the convolutional operation with a speciﬁc weight matrix for each node degree.
Diffusion-convolution neural networks (DCNNs) (Atwood & Towsley, 2016) employs the transition matrix power series to propagate features.
Niepert et al.
(2016) extract local features with a ﬁxed number of neighbors.
Monet (Monti et al., 2017) proposes a uniﬁed framework to generalize CNN architectures on non-Euclidean domains.
Spectral approaches deﬁne the convolution operations based on the spectral formulation.
Bruna et al.
(2014) specify the convolution operator with the spectrum of the graph Laplacian.
Defferrard et al.
(2016) approximate the ﬁlters by computing the Chebyshev polynomial recurrently, yielding fast and localized spectral ones.
Graph Convolutional Network (GCN) (Kipf & Welling, 2017), one of the most inﬂuential techniques, applies the ﬁrst-order approximation of localized spectral ﬁlters to operate the connected neighbors.
Recently, several methods (Abu-El-Haija et al., 2019; Liao et al., 2019) based on GCN have been proposed to obtain the higher-order ﬁlters.
Besides, GAT (Veliˇckovi´c et al., 2018), Graph U-Nets (Gao & Ji, 2019) combine attention networks and pooling operation with
GNN separately, which achieve state-of-the-art performance on node and link classiﬁcation tasks.
In this work, local augmentation can be applied on various backbone models to improve performance.
Graph Generative Models Generative models (Goodfellow et al., 2014; Kingma & Welling, 2013) are powerful tools of learning data distribution through unsupervised learning, and they have achieved tremendous success in various applications.
Recently, researchers have proposed several interesting generative models for graph data generation.
Variational graph auto-encoder (VGAE)
(Kipf &
Welling, 2016) makes use of latent variables and learns interpretable latent representations for undirected graphs.
Salha et al.
(2019) replace the GCN encoder in VGAE with a simple linear model and emphasize the effectiveness of a simple node encoding scheme.
Xu et al.
(2019) propose a generative model framework to learn node representations, by sampling graph generation sequences constructed from observed graph data.
ConDgen (Yang et al., 2019) exploits the GCN encoder to handle the inherent challenges of ﬂexible context-structure conditioning and permutation-invariant generation.
Besides, some methods have been proposed to apply the graph generative models in various applications such as graph matching (Simonovsky & Komodakis, 2018), molecule design (Liu et al., 2018), retrosynthesis prediction (Shi et al., 2020) and chemical design (Samanta et al., 2018).
Compared with these approaches mainly focusing on structure generation, our model takes full use of the power of the generative model for feature representation generation, which can serve as an enhanced technique for the downstream backbone models.
Topology-level Augmentation.
Topology-level augmentation usually perturbs A to generate different graph structures, which can be formulated as A (cid:48)
=
F ( A , X ) , where F ( · ) is a structure perturbation function.
For example, DropEdge (Rong et al., 2020) considers F ( A , X ) =
A − A s which is independent of X , where A s is a sparse matrix consists of a subset of the original edges E .
G AUG -O
(Zhao et al., 2021) employs a non-close-form formulation of the function and leverages their proposed neural edge predictors to produce a different structure A (cid:48)
where
[Equation]
τ is the temperature of Gumbel-Softmax distribution, G ∼ Gumbel(0 , 1) is a Gumbel random variate, and α is a hyperparameter mediating the inﬂuence of edge predictor on the original graph.
[Table]
Feature-level
Augmentation.
Besides, feature-level augmentation function can be deﬁnes as
X (cid:48)
=
H ( A , X ) , where H ( · ) is a feature perturbation function.
FLAG (Kong et al., 2020) deﬁnes the perturbation function as H ( A , X ) =
X + δ where perturbation δ is updated iteratively during the adversarial training phase.
G-GCN (plain) (Zhu et al., 2020) obtains the global attribute feature matrix X (
a ) ∈ R N × d a through minimizing the objective (cid:81) v ∈ V (cid:81) a ∈ CA
[Equation]
v )
[Equation]
a
[Equation]
a )
[Equation]
where U is the set of all attributes, CA ( v ) is the sampled context attributes of v , and
V ∈ R d a × F denotes the parameters.
Obviously, the perturbation function of G-GCN has no close-form solution.
In this work, we propose a novel feature perturbation function, named local augmentation, whose differences with existing methods are presented.
And the comparison of the details of various graph data augmentation techniques can be found in Table 1.
[Table]
E XPERIMENTS
[Table]
In this section
[Table]
we evaluate the performance of our proposed model on semisupervised node classiﬁcation tasks on a variety of public graph datasets and compare our model with the state-of-the-art graph neural networks.
We also carry out additional experiments to showcase the necessity of our design and its robustness to missing information.
4. D ATASETS
We utilize seven public graph datasets (Cora, Citeseer, Pubmed, Squirrel, Actor, Chameleon, and Cornell) for semisupervised node classiﬁcation tasks.
The details of these datasets can be found in the appendix.
[Figure]
Baselines and Experimental Setup.
We apply the standard ﬁxed splits (Yang et al., 2016) on three datasets Cora, Citeseer, and Pubmed, with 20 nodes per class for training, 500 nodes for validation, and 1,000 nodes for testing.
And we consider four backbones: GCN (Kipf &
Welling, 2017), GAT (Veliˇckovi´c et al., 2018),
GCNII (Chen et al., 2020), and GRAND (Feng et al., 2020) to evaluate our proposed framework and compare our model against stateof-the-art models including 1) backbone models: Chebyshev (Defferrard et al., 2016), GCN,
GAT, APPNP (Klicpera et al., 2019), Graph Unet (Gao & Ji, 2019), MixHop (Abu-El-Haija et al., 2019), GCNII, GSNN-M (Wang et al., 2020a), S 2 GC (Zhu & Koniusz, 2021), and
GRAND and 2) feature-level and topology-level augmentation models: G-GNNs (Zhu et al., 2020), DropEdge (Rong et al., 2020) and G AUGO (Zhao et al., 2021).
For four datasets Squirrel, Actor, Chameleon, and Cornell, we takerandom splits (Shchur et al., 2018) where 10%, 30%, and 60% of the date for training, validation, testing; measure the performance of GCN, GAT, GCNII, and corresponding modiﬁed models.
Results Figure 3 shows the distribution of the attributes of the original and inference neighbors, which can demonstrate our inference feature matrix follow the distribution of the initial feature matrix.
For three datasets Cora, Citeseer, and Pubmed, we report the mean classiﬁcation accuracy on the test nodes of all our models after 100 runs and report the values after running the experiments of their models with our server under their setting hyperparameters in their original papers.
The results of the evaluation experiments are summarized in Tables 2, 3, and in the appendix, which demonstrate that the backbone models equipped with our method achieve the best performance across all the datasets except the Cornell dataset.
More speciﬁcally, we can improve upon GCN by a margin of 2.5%, 2.2%, and 2.4% on Cora, Citeseer, and Pubmed respectively.
Moreover, LA-GNN outperforms other backbone models including GAT and GCNII as well as data augmentation models (Zhu et al., 2020;
Rong et al., 2020; Zhao et al., 2021) on these citation network datasets.
4. A BLATION S TUDY
[Table]
In this section, to demonstrate the effectiveness of our proposed generative framework, we conduct experiments that compare LA-GNN to several of its ablated variants without generative modeling.
The results are shown in Table 4.
"GCN + width" only increases the ﬁrst network layer width for GCN and GCNII to match LAGNN without giving generated samples as input.
"+ concatenation" only replaces the generated feature matrix of LA-GNN with the original feature matrix of the central node.
"+ plain neighborhood" replaces the generated feature matrix of LA-GNN with a neighborhood feature matrix where each row corresponds to the feature vector of the randomly sampled neighbor.
The results show that the ﬁrst two variants provide no notable improvement for the backbone models, and the third variant even results in degradation.
By eliminating the possibility that these confounding factors irrelevant to our core approach may contribute to the ﬁnal performance, it’s evident that the performance gain in Table 2 and 3 are due to our proposed generative local augmentation framework.
4. R OBUSTNESS TO M ISSING I NFORMATION
In this section
[Table]
we conduct experiments to verify that our proposed framework can robustify downstream tasks against missing information in the feature attributes.
Speciﬁcally, we mask a certain percentage of the attributes of each feature vector and use the same pipeline to do augmentation for the masked feature matrix.
As shown in Table 5, we can see that as the mask ratio increases, the gap of the performance between the GCN and LA-GCN enlarges in most cases in Cora and Citeseer, which corroborates our insight discussed in Section 2.5.
Since there exists large redundancy in the features of the Pubmed dataset, the performance of GCN and LA-GCN decreases little as the mask ratio increases.
And so the gap of the performance between the GCN and LA-GCN does not enlarge in the Pubmed dataset
[Table] [Table]
C ONCLUSION
We propose local augmentation, a brand-new technique that exploits the generative model to learn the conditional distribution of the central node’s neighbors’ feature representations given its representation.
We can feed the generated feature matrix from a well-trained generative model to some modiﬁed backbone GNN models to enhance model performance.
Experiments show that our model can improve performance across various GNN architectures and benchmark datasets.
Besides, our model achieves new state-of-the-art results on various semi-supervised node classiﬁcation tasks.
One limitation of our proposed framework is that we do not exploit the 2-hop neighbors or use the random walk to ﬁnd more related neighbors for the central node.
And one future work is that we can extract more 2/3-hop neighbors if the central node’s degree is small and learn the conditional distribution for random sampling nodes if the graph is large.
A P ROOFS OF E Q .(6) AND E Q .(7) A. P ROOF OF E Q .(6): T HE DERIVATION OF THE ELBO OF THE GENERATOR
We give more details of the derivation of the generator ELBO as follows:
[Equation]
A. P ROOF OF E Q .(7)
[Equation]
B R EPRODUCIBILITY B. D ATASETS D ETAILS
Cora, Citeseer, and Pubmed are standard citation network benchmark datasets Sen et al.
In these datasets, nodes represent documents, and edges denote citations; node feature corresponds to elements of a bag-of-words representation of a document, and node label corresponds to one of the academic topics.
Besides, we utilize four datasets used in Pei et al.
(2020) for evaluation.
Chameleon and squirrel are two page-page networks on speciﬁc topics in Wikipedia Rozemberczki et al.
In these datasets, nodes represent web pages, and edges denote mutual links between pages; node features correspond to several informative nouns in the Wikipedia pages and labels correspond to the number of the average monthly trafﬁc of the web page.
WebKB 1 is a webpage dataset collected from various universities.
We use the one subdataset of it, Cornell.
In this dataset, nodes represent web pages, and edges are hyperlinks between them; node features correspond to the bag-of-words representation of web pages and labels correspond to ﬁve categories, student, project, course, staff, and faculty.
Film dataset is the actor-only induced subgraph of the ﬁlm-directoractor-writer network Tang et al.
In this dataset, Nodes represent actors, and edges denote co-occurrence on the same
Wikipedia page; node features correspond to some keywords in the Wikipedia pages and labels correspond to ﬁve categories in terms of words of actor’s Wikipedia.
All the dataset statistics are summarized in Table 6.
[Table]
. I MPLEMENTATION D ETAILS
We use Pytorch (Paszke et al., 2019) to implement LA-GNNs.
The codes of S 2 GC (Zhu & Koniusz, 2021), LA-GCN, LA-GAT, LA-GCNII, LA-GRAND, and DropEdge-GCN are implemented referring to Pytorch implementation of S 2 GC 2 , GCN 3 (Kipf & Welling, 2017), GAT 4 (Veli ˇ ckovi ´
c et al., 2018),
GCNII 5 (Chen et al., 2020) GRAND 6 (Feng et al., 2020), and DropEdge-GCN 7 (Rong et al., 2020).
Besides, we implement APPNP (Klicpera et al., 2019) with DGL (Wang et al., 2019) version of
APPNP 8 .
The datasets Cora, Citeseer, Pubmed are downloaded from TensorFlow (Abadi et al., 2016) implementation of GCN 9 , and the datasets Chameleon, Squirrel, Actor, and Cornell are downloaded from the implementation of Geom-GCN 10 (Pei et al., 2020).
All the experiments in this work are conducted on a single NVIDIA Tesla V100 with 32GB memory size.
The operating system behind the Docker where the experiments are running is Red Hat 4.8.2-16.
And the software that we use for experiments are Python 3.6.8, numpy 1.19.2, sklearn 0.0, scipy 1.5.4, networkx 2.5.1, torch 1.6.0, torchvision 0.7.0, CUDA 10.2.89, and CUDNN 8.0.2.
B. H YPERPARAMETER D ETAILS
LA-GNNs introduce an additional parameter, that is the hidden layer for generated feature matrix X before concatenation.
The difference of architectures between GCN and LA-GCN can be found in
Figure 4, and the LA-GCNII architecture can be found in Figure 5.
[Figure]
[Figure]
The difference of hyperparameters between the GCN and LA-GCN is only the hidden layer size before concatenation.
For the LA-GCNII, LA-GAT, LA-GRAND, we tune the hyperparameters in the same way as described in their original papers with validation set.