U NIFYING G RAPH C ONVOLUTIONAL N EURAL N ETWORKS AND L ABEL P ROPAGATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Label Propagation (LPA) and Graph Convolutional Neural Networks (GCN) are both message passing algorithms on graphs.
Both solve the task of node classiﬁcation but LPA propagates node label information across the edges of the graph, while GCN propagates and transforms node feature information.
However, while conceptually similar, theoretical relation between LPA and GCN has not yet been investigated.
Here we study the relationship between LPA and GCN in terms of two aspects: (1) feature/label smoothing where we analyze how the feature/label of one node is spread over its neighbors; And, (2) feature/label inﬂuence of how much the initial feature/label of one node inﬂuences the ﬁnal feature/label of another node.
Based on our theoretical analysis, we propose an end-to-end model that uniﬁes GCN and LPA for node classiﬁcation.
In our uniﬁed model, edge weights are learnable, and the LPA serves as regularization to assist the GCN in learning proper edge weights that lead to improved classiﬁcation performance.
Our model can also be seen as learning attention weights based on node labels, which is more task-oriented than existing feature-based attention models.
In a number of experiments on real-world graphs, our model shows superiority over state-of-the-art GCN-based methods in terms of node classiﬁcation accuracy.
I NTRODUCTION
Consider the problem of node classiﬁcation in a graph, where the goal is to learn a mapping M :
V → L from nodes V to labels L .
Solution to this problem is widely applicable to various scenarios, e.g., inferring income of users in a social network or classifying scientiﬁc articles in a citation network.
Different from a generic machine learning problem where samples are independent from each other, nodes are connected by edges in the graph, which provide additional information and require more delicate modeling.
To capture the graph information, researchers have mainly designed models on the assumption that labels and features vary smoothly over the edges of the graph.
In particular, on the label side L , node labels are propagated and aggregated along edges in the graph, which is known as Label Propagation Algorithm (LPA) (Zhu et al., 2005;
Zhou et al., 2004; Zhang & Lee, 2007; Wang & Zhang, 2008; Karasuyama & Mamitsuka, 2013;
Gong et al., 2017; Liu et al., 2019a); On the node side V , node features are propagated along edges and transformed through neural network layers, which is known as Graph Convolutional Neural Networks (GCN) (Kipf &
Welling, 2017; Hamilton et al., 2017; Li et al., 2018; Xu et al., 2018;
Liao et al., 2019; Xu et al., 2019;
Qu et al., 2019).
GCN and LPA are related in that they propagate features and labels on the two sides of the mapping
M , respectively.
However, the relationship between GCN and LPA has not yet been investigated.
Speciﬁcally, what is the theoretical relationship between GCN and LPA, and how can they be combined to develop a more accurate model for node classiﬁcation in graphs?
Here we study the theoretical relationship between GCN and LPA from two viewpoints: (1)
Feature/label smoothing , where we show that the intuition behind GCN/LPA is smoothing features/labels of nodes across the edges of the graph, i.e., one node’s feature/label equals the weighted average of features/labels of its neighbors.
We prove that if edge weights smooth the node features, they also smooth the node labels with guaranteed upper bound on the approximation error.
And, (2) feature/label inﬂuence , where we quantify how much the initial feature/label of node v b inﬂuences the output feature/label of node v a in GCN/LPA by studying the Jacobian/gradient of node v b with respect to node v a , and then we also prove their quantitative relationship.
Based on the above theoretical analysis, we propose a uniﬁed model GCN-LPA for node classiﬁcation.
We show that the key to improving the performance of GCN is to enable nodes within the same class/label to connect more strongly with each other by making edge weights/strengths trainable.
Then we prove that increasing the strength of edges between the nodes of the same class is equivalent to increasing the accuracy of LPA’s predictions.
Therefore, we can ﬁrst learn the optimal edge weights by minimizing the loss of predictions in LPA, then plug the optimal edge weights into a GCN to learn node representations and do ﬁnal classiﬁcation.
In GCN-LPA, we further combine the two steps together and train the whole model in an end-to-end fashion, where the LPA part serves as regularization to assist the GCN part in learning proper edge weights that beneﬁt the separation of different classes.
It is worth noticing that GCN-LPA can also be seen as learning attention weights for edges based on node label information , which requires less handcrafting and is more task-oriented than existing work that learns attention weights based on node feature similarity (Veliˇckovi´c et al., 2018;
Thekumparampil et al., 2018; Zhang et al., 2018; Liu et al., 2019b).
Experiments on ﬁve datasets indicate that our model outperforms state-of-the-art methods in terms of classiﬁcation accuracy.
U NIFYING GCN AND LPA
In this section, we ﬁrst formulate the node classiﬁcation problem and brieﬂy introduce LPA and
GCN.
We then prove their relationship from the viewpoints of smoothing and inﬂuence.
Based on our theoretical analysis, we then propose a uniﬁed model GCN-LPA, and we also analyze the reason why our model performs better than GCN in node classiﬁcation.
2. P ROBLEM F ORMULATION AND P RELIMINARIES
We begin by describing the problem of node classiﬁcation on graphs and introducing notation.
Consider a graph G =
( V , A, X, Y ) , where V = { v 1 , · · · , v n } is the set of nodes, A ∈ R n × n is the adjacency matrix (self-loops are included), X is the feature matrix of nodes and Y is labels of nodes.
a ij (the ij -th entry of A ) is the weight of the edge connecting v i and v j .
N ( v ) denotes the set of immediate neighbors of node v in graph G .
Each node v
i has a feature vector x i which is the i -th row of X , while only the ﬁrst m nodes have labels y 1 , · · · , y m from a label set L = { 1 , · · · , c } .
The goal is to learn a mapping M : V → L and predict labels of unlabeled nodes.
Label Propagation Algorithm .
LPA assumes that two connected nodes are likely to have the same label, and thus it propagates labels iteratively along the edges.
Let Y ( k ) =
[ y ( k ) 1 , · · · , y ( k ) n ] (cid:62) ∈ R n × c be the soft label matrix in iteration k > 0 , in which the i -th row y ( k ) (cid:62) i denotes the predicted label distribution for node v i in iteration k . When k = 0 , the initial label matrix Y (0) = [ y (0)1 , · · · , y (0) n ] (cid:62) consists of one-hot label indicator vectors y (0) i for i = 1 , · · · , m (i.e., labeled nodes) or zero vectors otherwise (i.e., unlabeled nodes).
Let D be the diagonal degree matrix for A with entries d ii =
(cid:80) j a ij .
Then LPA (Zhu et al., 2005) in iteration k is formulated as the following two steps:
[Equation]
In Eq. (1), all nodes propagate their labels to their neighbors according to weights of edges. Then in Eq.
(2), labels of all labeled nodes are reset to their initial values, because LPA wants to persist labels of nodes which are labeled so that unlabeled nodes do not overpower the labeled ones as the initial labels would otherwise fade away.
Graph Convolutional Neural Network .
GCN is a multi-layer feedforward neural network that propagates and transforms node features across the graph.
The layer-wise propagation rule of GCN is X ( k +1)
=
σ ( D − 12 AD − 12 X ( k ) W ( k ) ) , where W ( k ) is trainable weight matrix in the k -th layer, σ ( · ) is an activation function such as ReLU, and X ( k ) =
[ x ( k ) 1 , · · · , x ( k ) n ] (cid:62) are the k -th layer node representations with X (0)
=
X .
To align with the above LPA, we use D − 1 A as the normalized adjacency matrix instead of the symmetric one D − 1 2 AD − 1 2 proposed by Kipf & Welling (2017).
Therefore, the feature propagation scheme of GCN in layer k is:
[Equation]
Notice similarity between Eqs.
(1) and (3).
Next we shall study and uncover the relationship between the two equations.
2. F EATURE S MOOTHING AND L ABEL S MOOTHING
The intuition behind both LPA and GCN is smoothing (Zhu et al., 2003;
Li et al., 2018):
In LPA, the ﬁnal label of a node is the weighted average of labels of its neighbors: y
[Equation]
i
[Equation]
in GCN, the ﬁnal node representation is also the weighted average of representations of its neighbors if we assume σ is identity function and W ( · ) are identity matrices
[Equation]
Next we show the relationship between feature smoothing and label smoothing:
Theorem 1 (Relationship between feature smoothing and label smoothing) Suppose that the latent ground-truth mapping M : x → y from node features to node labels is differentiable and satisﬁes L-Lipschitz constraint, i.e., |M ( x 1 ) − M ( x 2 ) | ≤ L (cid:107) x 1 − x 2 (cid:107) 2 for any x 1 and x 2 ( L is a constant).
If the edge weights { a ij } approximately smooth x i over its immediate neighbors with error (cid:15) i , i.e., x i = 1 d ii (cid:80) j ∈N ( i ) a ij x j + (cid:15) i , then the edge weights { a ij } also approximately smooth y i over its immediate neighbors with the following approximation error: (cid:12)(cid:12)
y
[Equation] (cid:107)
+ o (cid:0) max j ∈N ( i ) ( (cid:107) x
j − x i (cid:107) 2 ) (cid:1) , where o ( α ) denotes a higher order inﬁnitesimal than α .
Proof of Theorem 1 is in Appendix A. Theorem 1 indicates that label smoothing is theoretically guaranteed by feature smoothing.
Note that if we treat edge weights { a ij } learnable, then feature smoothing (i.e., (cid:15) i → 0 ) can be directly achieved by keeping node features x i ﬁxed while setting { a ij } appropriately, without resorting to feature propagation in a multi-layer GCN.
Therefore, a simple approach to exploit this theorem would be to learn { a ij } by reconstructing node feature x i from its neighbors, then use the learned { a ij } to reconstruct node labels y
i
(Karasuyama &
Mamitsuka, 2013).
This is equivalent to ﬁrst minimizing the difference between X (0) and X (1) by learning D − 1 A in a one-layer GCN where σ and W (0) are identity, then plug the learned D − 1 A in
LPA and run it for one iteration.
As shown in Theorem 1, the approximation error of labels is dominated by L (cid:107) (cid:15)
i (cid:107) 2 .
However, this error could be fairly large in practice because: (1)
The number of immediate neighbors for a given node may be too small to reconstruct its features perfectly, especially in the case where node features are high-dimensional and sparse.
For example, in a citation network where node features are one-hot bag-of-words vectors, the feature of one article can never be precisely reconstructed if none of its neighboring articles contains the speciﬁc word that appears in this article.
As a result, (cid:107) (cid:15) i (cid:107) 2 will be non-neglibible.
This explains why it is beneﬁcial to apply LPA and GCN for multiple iterations/layers in order to include information from farther away neighbors.
The ground-truth mapping M may not be sufﬁciently smooth due to the complex structure of latent manifold and possible noise, which fails to satisfy L -Lipschitz constraint.
In other words, the constant L will be extremely large.
2. F EATURE I NFLUENCE AND L ABEL I NFLUENCE
To address the above concerns and extend our analysis, we next consider GCN and LPA with multiple layers/iterations, and do not impose any constraint on the ground-truth mapping M .
Consider two nodes v a and v b in a graph.
Inspired by Koh & Liang (2017) and Xu et al.
(2018), we study the relationship between GCN and LPA in terms of inﬂuence, i.e., how the output feature/label of v a will change if the initial feature/label of v b is varied slightly.
Technically, the feature/label inﬂuence is measured by the Jacobian/gradient of the output feature/label of v a with respect to the initial feature/label of v b .
Denote x ( k ) a as the k -th layer representation vector of v a in GCN, and x b as the initial feature vector of v b .
We quantify the feature inﬂuence of v b on v a as follows:
Deﬁnition 1 (Feature inﬂuence)
The feature inﬂuence of node v b on node v a after k layers of GCN is the L1-norm of the expected Jacobian matrix ∂ x ( k ) a /∂ x b : I f ( v a , v b ; k ) =
(cid:13)(cid:13) E (cid:2) ∂
x ( k ) a /∂ x b (cid:3)(cid:13)(cid:13) 1 .
The normalized feature inﬂuence is then deﬁned as ˜ I
f ( v a , v b ; k ) =
I f ( v a , v b ; k ) / (cid:80) v i ∈V I f ( v a , v i ; k ) .
We also consider the label inﬂuence of node v b on node v a in LPA (this implies that v a is unlabeled and v b is labeled).
Since different label dimensions of y ( · ) i do not interact with each other in LPA, we assume that all y i and y ( · ) i are scalars within [0 , 1] (i.e., a binary classiﬁcation) for simplicity.
Label inﬂuence is deﬁned as follows:
Deﬁnition 2 (Label inﬂuence)
The label inﬂuence of labeled node v b on unlabeled node v a after k iterations of LPA is the gradient of y ( k ) a with respect to y b : I l ( v a , v b ; k )
= ∂y ( k ) a /∂y b .
The following theorem shows the relationship between feature inﬂuence and label inﬂuence:
Theorem 2 (Relationship between feature inﬂuence and label inﬂuence) Assume the activation function used in GCN is ReLU.
Denote v a as an unlabeled node, v b as a labeled node, and β as the fraction of unlabeled nodes.
Then the label inﬂuence of v b on v a after k iterations of LPA equals, in expectation, to the cumulative normalized feature inﬂuence of v b on v a after k layers of GCN:
[Equation]
Proof of Theorem 2 is in Appendix B. Intuitively, Theorem 2 shows that if v b has high label inﬂuence on v a , then the initial feature vector of v b will also affect the output feature vector of v a to a large extent.
Theorem 2 provides the theoretical guideline for designing our uniﬁed model in the next subsection.
2. T HE U NIFIED M ODEL
Before introducing the proposed model, we ﬁrst rethink the GCN method and see what an ideal node representation should be like.
Since we aim to classify nodes, the perfect node representation would be such that nodes with the same label are embedded close together, which would give a large separation between different classes.
Intuitively, the key to achieve this goal is to enable nodes within the same class to connect more strongly with each other, so that they are pushed together by the
GCN.
We can therefore make edge strengths/weights trainable, then learn to increase the intra-class feature inﬂuence for each class i : (cid:80) v a ,v b : y
a =
i,y b =
i ˜
I f ( v a , v b ) by adjusting edge weights.
However, this requires operating on Jacobian matrices with the size of d (0) ×
d ( K ) ( d (0) and d ( K ) are the dimensions of initial and output features, respectively), which is impractical if initial node features are high-dimensional.
Fortunately, we can turn to optimizing the intra-class label inﬂuence instead, i.e., (cid:80) v a ,v b : y
a =
i,y b
= i
I l ( v a , v b ) , according to Theorem 2.
We further show that, by the following theorem, the intra-class label inﬂuence for a given node v a is proportional to the probability that v a is classiﬁed correctly by LPA:
Theorem 3 (Relationship between label inﬂuence and LPA’s prediction) Consider a given node v a and its label y a .
If we treat node v a as unlabeled, then the total label inﬂuence of nodes with label y a on node v a is proportional to the probability that node v a is classiﬁed as y a by LPA:
[Equation]
where ˆ y lpaa is the predicted label of v a using a k -iteration LPA.
Proof of Theorem 3 is in Appendix C. Theorem 3 indicates that, if edge weights { a ij } maximize the probability that v a is correctly classiﬁed by LPA, then they also maximize the intra-class label inﬂuence for node v a .
We can therefore ﬁrst learn the optimal edge weights A ∗ by minimizing the loss of predicted labels by LPA:
[Equation]
where J is the cross-entropy loss, ˆ y lpaa and y a are the predicted label distribution of v a using LPA and the true one-hot label of v a , respectively. 2 a ≤ m means v a is labeled.
The optimal A ∗ maximize the probability that each node is correctly labeled by LPA, thus also increasing the intra-class label inﬂuence (by Theorem 3) and intra-class feature inﬂuence (by Theorem 2).
Then we can apply A ∗ and the corresponding D ∗ to a GCN to predict labels and learn optimal transformation matrices:
[Equation]
where ˆ y gcna (which is the a -th row of X ( K ) ) is the predicted label distribution of v a using the GCN speciﬁed in Eq.
In practice, it is generally better to combine the above two steps together and train the whole model in an end-to-end fashion:
[Equation]
where λ is the balancing hyper-parameter.
In this way, L lpa ( A ) serves as a regularization term that assists the learning of edge weights A , since it is hard for the GCN to learn both W and A simultaneously due to overﬁtting.
The proposed GCN-LPA approach can also be seen as learning the importance of edges that can be used to reconstruct node labels accurately by LPA, then transferring this knowledge from label space to feature space for the GCN.
From this perspective, GCN-LPA also connects to Theorem 1 except that the knowledge transfer is in the other direction.
It is also worth noticing how the optimal A ∗ is conﬁgured.
The principle here is that we do not modify the basic structure of the original graph (i.e., not adding or removing edges) but only adjusting weights of existing edges.
This is equivalent to learning a positive mask matrix M for the adjacency matrix A and taking the Hadamard product M ◦ A =
A ∗ .
Each element M ij can be set as either a free variable or a function of the nodes at edge endpoints, for example, M ij = log (cid:0) exp( x (cid:62) i
Hx j )
+ 1 (cid:1) where H is a learnable kernel matrix for measuring feature similarity.
2. A NALYSIS OF GCN-LPA M ODEL B EHAVIOR
In this subsection, we show beneﬁts of our uniﬁed model compared with GCN by analyzing properties of embeddings produced by the two models.
We ﬁrst analyze the update rule of GCN for node v i : x ( k +1)
i
[Equation]
, where ˜ a ij = a ij /d ii is the normalized weight of edge ( j, i ) .
This formula can be decomposed into the following two steps: (1) In aggregation step, we calculate the aggregated representation h ( k ) i of all neighborhoods N ( v i ) :
h ( k ) i
= (cid:80) v j ∈N ( v i )
˜ a ij x ( k ) j ; (2) In transformation step, the aggregated representation h ( k ) i is mapped to a new space by a transformation matrix and nonlinear function: x ( k +1)
i =
σ (cid:0) h ( k )
i W ( k ) (cid:1) .
We show by the following theorem that the aggregation step reduces the overall distance in the embedding space between the nodes that are connected in the graph:
Theorem 4 (Shrinking property in GCN) Let D ( x ) = 12 (cid:80) v i ,v j ˜ a ij (cid:107) x
i − x j (cid:107) 22 be a distance metric between the embeddings x of nodes.
Then we have D ( h ( k ) )
≤ D ( x ( k ) ) .
[Figure] Proof of Theorem 4 is in Appendix D. Theorem 4 indicates that the overall distance among connected nodes is reduced after taking one aggregation step, which implies that connected components in the graph “shrink” and nodes within each connected component get closer to each other in the embedding space.
In an ideal case where edges only connect nodes with the same label, the aggregation step will push nodes within the same class together, which greatly beneﬁts the step that acts like a hyperplane W ( k ) for classiﬁcation.
However two connected nodes may have different labels.
These “noisy” edges will impede the formation of clusters and make the inter
Fortunately, in GCN-LPA, edge weights are learned by minimizing the difference between groundtruth labels and labels reconstructed from multi-hop neighbors.
This will force the model to increase weight/bandwidth of possible paths that connect nodes with the same label, so that labels can “ﬂow” easily along these paths for the purpose of label reconstruction.
In this way, GCN-LPA is able to identify potential intra-class edges and increase their weights to assist learning clustering structures.
To illustrate this, we apply a two-layer untrained GCN with randomly initialized transformation matrices to the well-known Zachary’s karate club network (Zachary, 1977) as shown in Figure 1a, which contains 34 nodes of 2 classes and 78 unweighted edges (grey solid lines).
We then increase the weights of intra-class edges by ten times to simulate GCN-LPA.
We ﬁnd that GCN works well on this network (Figure 1b), but GCN-LPA performs even better than GCN because the node embeddings are completely linearly separable as shown in Figure 1c.
To further justify our claim, we randomly add 20 “noisy” inter-class edges (grey dotted lines) to the original network, from which we observe that GCN is misled by noise and mixes nodes of two classes together (Figure 1d), but GCNLPA still distinguishes the two clusters (Figure 1e) because it is better at “denoising” undesirable edges based on the supervised signal of labels.
C ONNECTION TO E XISTING W ORK
Edge weights play a key role in graph-based node classiﬁcation as well as representation learning.
In this section, we discuss three lines of related work that learn edge weights adaptively.
Locally linear embedding (LLE) (Roweis & Saul, 2000) and its variants (Zhang & Wang, 2007;
Kong et al., 2012) learn edge weights by constructing a linear dependency between a node and its neighbors, then use the learned edge weights to embed high-dimensional nodes into a lowdimensional space.
Our work is similar to LLE in the aspect of transferring the knowledge of edge importance from one space to another, but the difference is that LLE is an unsupervised dimension reduction method that learns the graph structure based on local proximity only, while our work is semi-supervised and explores high-order relationship among nodes.
Classical LPA (Zhu et al., 2005; Zhou et al., 2004) can only make use of node labels rather than node features.
In contrast, adaptive LPA considers node features by making edge weights learnable.
Typical techniques of learning edge weights include adopting kernel functions (Zhu et al., 2003;
Liu et al., 2019a)
(e.g., a ij = exp( − (cid:80) d ( x id − x jd ) 2 /σ 2 d ) where d is dimensionality of features), minimizing neighborhood reconstruction error (Wang & Zhang, 2008; Karasuyama & Mamitsuka, 2013), using leave-one-out loss (Zhang & Lee, 2007), or imposing sparseness on edge weights (Hong et al., 2009).
However, in these LPA variants, node features are only used to assist learning the graph structure rather than explicitly mapped to node labels, which limits their capability in node classiﬁcation.
Another notable difference is that adaptive LPA learns edge weights by introducing the regularizations above, while our work takes LPA itself as regularization to learn edge weights.
Our method is also conceptually connected to attention mechanisms on graphs (Veliˇckovi´c et al., [-
-]2018;
Thekumparampil et al., 2018; Zhang et al., 2018; Liu et al., 2019b), in which an attention weight α ij is learned between node v i and v j .
For example, α ij = LeakyReLU
( a (cid:62)
[ W x i ||
W x j ]) in GAT (Veliˇckovi´c et al., 2018), α ij = a · cos( W x i , W x j ) in AGNN (Thekumparampil et al., 2018), α ij =
( W 1 x i ) (cid:62) W 2 x j in GaAN (Zhang et al., 2018), and α ij = a (cid:62) tanh
( W 1 x i + W 2 x j ) in GeniePath (Liu et al., 2019b), where a and W are trainable variables.
A signiﬁcant difference between these attention mechanisms and our work is that attention weights are learned based merely on feature similarity, while we propose that edge weights should be consistent with the distribution of labels on the graph, which requires less handcrafting of the attention function and is more taskoriented.
Nevertheless, all the above formulas for calculating attentions can also be used in our model as the implementation of edge weights.
E XPERIMENTS
We evaluate our model and present its performance on ﬁve datasets including citation networks and coauthor networks.
We also study the hyper-parameter sensitivity and provide training time analysis.
4. D ATASETS
We use the following ﬁve datasets in our experiments:
Citation networks : We consider three citation network datasets (Sen et al., 2008): Cora , Citeseer , and Pubmed .
In these datasets , nodes correspond to documents, edges correspond to citation links, and each node has a sparse bag-of-words feature vector as well as a class label.
Coauthor networks :
We also use two co-authorship networks (Shchur et al., 2018), Coauthor-CS and Coauthor-Phy , based on Microsoft Academic Graph from the KDD Cup 2016 challenge.
Here nodes are authors and an edge indicates that two authors co-authored a paper.
Node features represent paper keywords for each author’s papers, and class labels indicate most active ﬁelds of study for each author.
Statistics of the ﬁve datasets are shown in Table 1.
We also calculate the intra-class edge rate (the fraction of edges that connect two nodes within the same class), which is signiﬁcantly higher than inter-class edge rate in all networks.
The ﬁnding supports our claim in Section 2.5 that node classiﬁcation beneﬁts from intra-class edges in a graph.
4. B ASELINES
We compare against the following baselines in our experiments: Multi-layer Perceptron (MLP) and Logistic Regression (LR) are feature-based methods that do not consider the graph structure.
Label Propagation (LPA) (Zhu et al., 2005), on the other hand, only consider the graph structure and ignore node features.
The rest of baselines are GCN-based methods: Graph Convolutional Network (GCN) (Kipf & Welling, 2017) proposes a ﬁrst-order approximation to spectral graph convolutions.
Graph Attention Network (GAT) (Veliˇckovi´c et al., 2018) propose an attention mechanism to treat neighbors differently in the aggregation step.
Jumping Knowledge
Networks (JK-Net) (Xu et al., 2018) leverages different neighborhood ranges for each node to en- [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Figure]
[Figure]
r a c y
[Figure]
{+1K 10K 100K 1M # nodes 
+}
T r a i n i n g t i m e p e r
e
p
o c h ( s ) GCN-LPAGCN able structure-aware representation. We use concat as the aggregator for JK-Net. Graph Sampling and Aggregation (GraphSAGE) (Hamilton et al., 2017) is a mini-batch implementation of GCN that uses neighborhood sampling strategy and different aggregation schemes.
We use mean as the aggregator for GraphSAGE.
4. E XPERIMENTAL S ETUP
Our experiments focus on the transductive setting where we only know labels of part of nodes but have access to the entire graph as well as features of all nodes.
The ratio of training, validation, and test set are set as 6 : 2 : 2 .
The weight of each edge is treated as a free variable during training.
We train our model for 200 epochs using Adam (Kingma & Ba, 2015) and report the test set accuracy when validation set accuracy is maximized.
Each experiment is repeated three times and we report the mean and the 95% conﬁdence interval.
We initialize weights according to Glorot & Bengio (2010) and row-normalize input features.
During training, we apply L2 regularization to the transformation matrices and use the dropout technique (Srivastava et al., 2014).
The settings of all other hyper-parameters can be found in Appendix F.
4. R ESULTS
The results of node classiﬁcation are summarized in Table 2.
Table 2 indicates that only using node features (MLP, LR) or graph structure (LPA) will lead to information loss and cannot fully exploit datasets in general.
The results demonstrate that our proposed GCN-LPA model surpasses stateof-the-art GCN/GNN baselines.
We note that JK-Net is a strong baseline on Cora, but it does not perform consistently well on other datasets.
We investigate the inﬂuence of the number of LPA iterations and the training weight of LPA loss term λ on the performance of classiﬁcation.
The results on Citeseer dataset are plotted in Figures 2 and 3, respectively, where each line corresponds to a given number of GCN layers in GCN-LPA.
From Figure 2 we observe that the performance is boosted at ﬁrst when the number of LPA iterations increases, then the accuracy stops increasing and decreases since a large number of LPA iterations will include more noisy nodes.
Figure 3 shows that training without the LPA loss term (i.e., λ = 0 ) is more difﬁcult than the case where λ = 1 ∼ 5 , which justiﬁes our aforementioned claim that it is [Table] hard for the GCN part to learn both transformation matrices W and edge weights A simultaneously without the assistance of LPA regularization.
To further show how much the LPA impacts the performance, we vary the ratio of labeled nodes in LPA from 100% to 0% during training, and report the result of acuracy on Citeseer dataset in
Table 3.
From Table 3 we observe that the performance of GCN-LPA gets worse when the ratio of labeled nodes in LPA decreases. In addition, using more labeled nodes in LPA also helps improve the model stability. Note that a ratio of 0% does not mean that GCN-LPA is equivalent to GCN (Kipf & Welling, 2017) because the edge weights in GCN-LPA is still trainable, which increases the risk of overﬁtting the training data. We study the training time of GCN-LPA on random graphs.
We use the one-hot identity vector as feature and 0 as label for each node.
The size of training set and validation set is 100 and 200, respectively, while the rest is test set.
The average number of neighbors for each node is set as 5, and the number of nodes is varied from one thousand to one million.
We run GCN-LPA and GCN for 100 epochs on a Microsoft Azure virtual machine with 1 NVIDIA Tesla M60 GPU, 12 Intel Xeon
CPUs (E5-2690 v3 @2.60GHz), and 128GB of RAM, using the same hyper-parameter setting as in
Cora.
The training time per epoch of GCN-LPA and GCN is presented in Figure 4.
Our result shows that GCN-LPA requires only 9 . 2% extra training time on average compared to GCN.
C ONCLUSION AND F UTURE W ORK
In this paper, we studied the theoretical relationship between two types of well-known graph-based models for node classiﬁcation, label propagation algorithm and graph convolutional neural networks, from the perspectives of feature/label smoothing and feature/label inﬂuence.
We then propose a uniﬁed model GCN-LPA, which learns transformation matrices and edge weights simultaneously in GCN with the assistance of LPA regularizer.
We also analyze why our uniﬁed model performs better than traditional GCN in node classiﬁcation.
Experiments on ﬁve datasets demonstrate that our model outperforms state-of-the-art baselines, and it is also highly time-efﬁcient with respect to the size of a graph.
We point out two avenues of possible directions for future work.
First, our proposed model focuses on transductive setting where all node features and the entire graph structure are given.
An interesting problem is whether it can be applied to inductive setting where we have no access to test nodes during training.
Second, the question of how to generalize the idea of our model to GNNs with different aggregation functions (e g., concatenation or max-pooling) is also a promising direction.
A PPENDIX A P ROOF OF T HEOREM
Proof.
Denote ˜ a ij = a ij /d ii as the normalized weight of edge ( j, i ) .
It is clear that (cid:80) j ∈N ( i ) ˜ a ij = 1 .
Given that M is differentiable
[Equation]
we perform a ﬁrst-order Taylor expansion with Peano’s form of remainder at x i for (cid:80) j ∈N ( i ) ˜ a ij y j :
[Equation]
i
[Equation]
According to Cauchy-Schwarz inequality and L -Lipschitz property
[Equation]
we have
[Equation]
Therefore
[Equation]
the approximation of y i is bounded by
[Equation]
B P ROOF OF T HEOREM
Before proving Theorem
[Equation]
we ﬁrst give two lemmas that demonstrate the exact form of feature inﬂuence and label inﬂuence deﬁned in this paper.
The relationship between feature inﬂuence and label inﬂuence can then be deduced from their exact forms.
Lemma 1 Assume that the nonlinear activation function in GCN is ReLU.
Let P a → b k be a path [ v ( k ) , v ( k − 1) , · · · , v (0) ] of length k from node v a to node v b , where v ( k )
= v a , v (0) = v b , and v ( i − 1)
∈ N ( v ( i ) ) for i
= k, · · · , 1 .
Then we have
[Equation]
where ˜ a v ( i − 1) ,v ( i ) is the normalized weight of edge ( v ( i ) , v ( i − 1) ) .
Proof.
See Xu et al.
(2018) for the detailed proof.
[Equation]
The product term in Eq.
(13) is the probability of a given path P a → b k .
Therefore, the right hand side in Eq.
(13) is the sum over probabilities of all possible paths of length k from v a to v b , which is the probability that a random walk starting at v a ends at v b after taking k steps. [Figure] 𝒗 [Figure]
% 𝑣
[Figure] 𝑣 ’ 𝑣 ( (a [Figure] Iteration𝒗 𝒂 𝒗 𝒃 𝑣 % 𝑣 & 𝑣 ’ 𝑣 ( (b) Iteration𝒗 𝒂 𝒗 𝒃 𝑣 % 𝑣 & 𝑣 ’ 𝑣 ( (c) Iteration𝒗 𝒂 𝒗 𝒃 𝑣 % 𝑣 & 𝑣 ’
[Figure]
[Figure]
Lemma 2 Let U a →
b
j be a path [ v ( j ) , v ( j − 1) , · · · , v (0) ] of length j from node v a to node v b , where v ( j )
= v a , v (0) = v b , v ( i − 1)
∈
N ( v ( i ) ) for i
= j, · · · , 1 , and all nodes along the path are unlabeled except v (0) .
Then we have
[Equation]
(cid:88)
U a → b j 1 (cid:89) i =
j
[Equation] a [Equation] ( i − 1) ,v ( i ) , (14) where ˜ a v ( i − 1) ,v ( i ) is the normalized weight of edge ( v ( i ) , v ( i − 1) ) .
To intuitively understand this lemma, note that there are two differences between Lemma 1 and
Lemma 2: (1) In Lemma 1, ˜ I f ( v a , v b ; k ) sums over all paths from v a to v b of length k , but in
Lemma 2, I l ( v a , v b ; k ) sums over all paths from v a to v b of length no more than k .
The is because in
LPA, v b ’s label is reset to its initial value after each iteration, which means that the label of v b serves as a constant signal that begins propagating in the graph again and again after each iteration. In Lemma 1 we consider all possible paths from v a to v b , but in Lemma 2, the paths are restricted to contain unlabeled nodes only. The reason here is the same as above: Since the labels of labeled nodes are reset to their initial values after each iteration in LPA, the inﬂuence of v b ’s label will be absorbed in labeled nodes, and the propagation of v b ’s label will be cut off at these nodes.
Therefore, v b ’s label can only ﬂow to v a along the paths with unlabeled nodes only.
See Figure 5 for an illustrating example showing the label propagation in LPA. Proof. As mentioned above, a signiﬁcant difference between LPA and GCN is that all labeled nodes are reset to its original labels after each iteration in LPA.
This implies that the initial label y b of node v b appears not only as y (0) b , but also as every y ( j ) b for j = 1 , · · · , k − 1 .
Therefore, the inﬂuence of y b on y ( k ) a is the cumulative inﬂuence of y ( j ) b on y ( k ) a for j = 0 , 1 , · · · , k − 1 :
[Equation]
According to the updating rule of LPA, we have
[Equation]
In the above equation, the derivative [Equation] is decomposed into the weighted average of ∂y
[Equation]
where v z traverses all neighbors of v a . For those v z ’s that are initially labeled, y ( k − 1) z is reset to their initial labels in each iteration.
Therefore, they are always constant and independent of y ( j ) b , meaning that their derivatives w.r.t.
y ( j ) b are zero. So we only need to consider the terms where v z is an unlabeled node:
[Equation]
where z > m means v z is unlabeled.
To intuitively understand Eq.
(17), one can imagine that we perform a random walk starting from node v a for one step, where the “transition probability” is the edge weights ˜ a , and all nodes in this random walk are restricted to unlabeled nodes only.
Note that we can further decompose every y ( k − 1) z in Eq.
(17) in the way similar to what we do for y ( k ) a in
Eq. So the expansion in Eq. (17) can be performed iteratively until the index k decreases to j . This is equivalent to performing all possible random walks for k − j steps starting from v a , where all nodes but the last in the random walk are restricted to be unlabeled nodes:
[Equation]
where v z in the ﬁrst summation term is the end node of a random walk, U a → z k − j in the second summation term is an unlabeled-nodes-only path from v a to v z of length k − j , and the product term is the probability of a given path U a → z k − j .
Consider the last term
[Equation] in Eq. We know that ∂y ( j ) z ∂y ( j ) b =for all z (cid:54) =
b and ∂y ( j ) z ∂y ( j ) b = 1 for z = b , which means that only those random-walk paths that end exactly at v b (i.e., the end node v z is exactly v b ) count for the computation in Eq.
Therefore, we have
[Equation]
where U a → b k − j is a path from v a to v b of length k − j containing only unlabeled nodes except v b .
Substituting the right hand term of Eq.
(15) with Eq.
(19), we obtain that
[Equation]
Now Theorem 2 can be proved by combining Lemma 1 and Lemma 2:
Proof.
Suppose that whether a node is labeled or not is independent of each other for the given graph.
Then we have
[Equation]
is an unlabeled-nodes-only path
[Equation]
C P ROOF OF T HEOREM
Proof.
Denote the set of labels as L .
Since different label dimensions in y ( · ) a do not interact with each other when running LPA, the value of the y a -th dimension in y ( · ) a (denoted by y ( · ) a [ y a ] ) comes only from the nodes with initial label y a .
It is clear that
[Equation]
22) which equals (cid:80) v b : y b =
y a I l ( v a , v b ; k ) according to Lemma 2.
Therefore, we have
[Equation]
D P ROOF OF T HEOREM
In this proof we assume that the dimension of node representations is one, but note that the conclusion can be easily generalized to the case of multi-dimensional representations since the function
D ( x ) can be decomposed into the sum of one-dimensional cases.
In the following of this proof, we still use bold notations x ( k ) i and h ( k ) i to denote node representations, but keep in mind that they are scalars rather than vectors.
We give two lemmas before proving Theorem 4.
The ﬁrst one is about the gradient of D ( x ) :
Lemma
[Equation]
Proof.
[Equation]
It is interesting to see from Lemma 3 that the aggregation step in GCN is equivalent to running gradient descent for one step with a step size of one.
However, this is not able to guarantee that
D ( h ( k ) ) ≤ D ( x ( k ) )
because the step size may be too large to reduce the value of D .
The second lemma is about the Hessian of D ( x ) :
Lemma 4 ∇ 2 D ( x ) (cid:22) 2 I , or equivalently, 2 I − ∇ 2 D ( x ) is a positive semideﬁnite matrix.
Proof.
We ﬁrst calculate the Hessian of D (
[Equation]
Therefore, 2 I − ∇ 2 D ( x ) =
I + D − 1 A .
Since D − 1 A is Markov matrix (i.e., each entry is nonnegative and the sum of each row is one), its eigenvalues are within the range [-1, 1], so the eigenvalues of I + D − 1 A are within the range [0, 2].
Therefore, I + D − 1 A is a positive semideﬁnite matrix, and we have ∇ 2 D ( x ) (cid:22) 2 I .
[Equation]
We can now prove Theorem 4:
Proof.
Since D is a quadratic function, we perform a second-order Taylor expansion of D around x ( k ) and obtain the following inequality:
[Equation]
[Figure]
E M ORE V ISUALIZATION R ESULTS ON K ARATE C LUB N ETWORK
Figure 6 illustrates more visualization of GCN and GCN-LPA on karate club network.
In each subﬁgure, we vary the number of layers from 1 to 4 to examine how the learned representations evolve.
The initial node features are one-hot identity vectors, and the dimension of hidden layers and output layer is 2.
The transformation matrices are uniformly initialized within range [-1, 1].
We use sigmoid function as the nonlinear activation function.
Comparing the four ﬁgures in each row, we conclude that the aggregation step and transformation step in GCN and GCN-LPA do beneﬁt the separation of different classes.
Comparing Figure 6a and 6c (or Figure 6b and 6d), we conclude that more inter-class edges will make the separation harder for GCN (or GCN-LPA).
Comparing Figure 6a and 6b (or Figure 6c and 6d), we conclude that GCN-LPA is more noise-resistant than GCN, therefore, GCN-LPA can better differentiate classes and identify clustering substructures.
F H YPER - PARAMETER S ETTINGS
The detailed hyper-parameter settings for all datasets are listed in Table 4.
In GCN-LPA, we use the same dimension for all hidden layers.
Note that the number of GCN layers and the number of
LPA iterations can actually be different since GCN and LPA are implemented as two independent modules.
We use grid search to determine hyper-parameters on Cora, and perform ﬁne-tuning on
[Table] [Table]
other datasets, i.e., varying one hyper-parameter per time to see if the performance can be further improved.
The search spaces for hyper-parameters are as follows:
• Dimension of hidden layers: { 8 , 16 , 32 } ; • # GCN layers: { 1 , 2 , 3 , 4 , 5 , 6 } ; • # LPA iterations: { 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 } ; • L2 weight: { 10 − 7 , 2 × 10 − 7 , 5 × 10 − 7 , 10 − 6 , 2 × 10 − 6 , 5 × 10 − 6 , 10 − 5 , 2 × 10 − 5 , 5 × 10 − 5 , 10 − 4 , 2 × 10 − 4 , 5 × 10 − 4 , 10 − 3 } ; • LPA weight ( λ ): { 0 , 1 , 2 , 5 , 10 , 15 , 20 } ; • Dropout rate: { 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } ; • Learning rate: { 0 . 01 , 0 .