S TOCHASTIC T RAINING OF G RAPH C ONVOLUTIONAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph convolutional networks (GCNs) are powerful deep neural networks for graph-structured data.
However, GCN computes nodes’ representation recursively from their neighbors, making the receptive ﬁeld size grow exponentially with the number of layers.
Previous attempts on reducing the receptive ﬁeld size by subsampling neighbors do not have any convergence guarantee, and their receptive ﬁeld size per node is still in the order of hundreds.
In this paper, we develop a preprocessing strategy and two control variate based algorithms to further reduce the receptive ﬁeld size.
Our algorithms are guaranteed to converge to GCN’s local optimum regardless of the neighbor sampling size.
Empirical results show that our algorithms have a similar convergence speed per epoch with the exact algorithm even using only two neighbors per node.
The time consumption of our algorithm on the Reddit dataset is only one ﬁfth of previous neighbor sampling algorithms.
I NTRODUCTION
Graph convolution networks (GCNs) (Kipf & Welling, 2017) generalize convolutional neural networks (CNNs) (LeCun et al., 1995) to graph structured data.
The “graph convolution” operation applies same linear transformation to all the neighbors of a node, followed by mean pooling.
By stacking multiple graph convolution layers, GCNs can learn nodes’ representation by utilizing information from distant neighbors.
GCNs have been applied to semi-supervised node classiﬁcation (Kipf &
Welling, 2017), inductive node embedding (Hamilton et al., 2017a), link prediction (Kipf & Welling, 2016; Berg et al., 2017) and knowledge graphs (Schlichtkrull et al., 2017), outperforming multi-layer perceptron (MLP) models that do not use the graph structure and graph embedding approaches (Perozzi et al., 2014;
Tang et al., 2015; Grover & Leskovec, 2016) that do not use node features.
However, the graph convolution operation makes it difﬁcult to train GCN efﬁciently.
A node’s representation at layer L is computed recursively by all its neighbors’ representations at layer L − 1 .
Therefore, the receptive ﬁeld of a single node grows exponentially with respect to the number of layers, as illustrated in Fig. 1(a).
Due to the large receptive ﬁeld size, Kipf & Welling (2017) proposed training GCN by a batch algorithm, which computes the representation for all the nodes altogether.
However, batch algorithms cannot handle large scale datasets because of their slow convergence and the requirement to ﬁt the entire dataset in GPU memory.
Hamilton et al. (2017a) made an initial attempt on developing stochastic algorithms to train GCNs, which is referred as neighbor sampling (NS) in this paper.
Instead of considering all the neighbors, they randomly subsample D ( l ) neighbors at the l -th layer.
Therefore, they reduce the receptive ﬁeld size to (cid:81) l D ( l ) , as shown in Fig. 1(b).
They found that for two layer GCNs, keeping D (1) = 10 and
D (2) = 25 neighbors can achieve comparable performance with the original model.
However, there is no theoretical guarantee on the predictive performance of the model learnt by NS comparing with the original algorithm.
Moreover, the time complexity of NS is still D (1) D (2) =
250 times larger than training an MLP, which is unsatisfactory.
In this paper, we develop novel stochastic training algorithms for GCNs such that D ( l ) can be as low as two, so that the time complexity of training GCN is comparable with training MLPs.
Our methods are built on two techniques.
First, we propose a strategy which preprocesses the ﬁrst graph convolution layer, so that we only need to consider all neighbors within L − 1 hops instead of L hops.
This is signiﬁcant because most GCNs only have L = 2 layers (Kipf & Welling, 2017; Hamilton
[Figure]
Layer
Layer(a)
Exact
Input
Layer
Layer(b) Neighbour sampling
Input
Layer
Layer(c) Control variate
Latest activation Historical activation
Input
GraphConv
Dropout Dropout
GraphConv (1) H (2) H
GraphConv
GraphConv (1)  (2)  (d) CVD network et al., 2017a).
Second, we develop two control variate (CV) based stochastic training algorithms.
We show that our CV-based algorithms have lower variance than NS, and for GCNs without dropout, our algorithm provably converges to a local optimum of the model regardless of D ( l ) .
We empirically test on six graph datasets, and show that our techniques signiﬁcantly reduce the bias and variance of the gradient from NS with the same receptive ﬁeld size.
Our algorithm with
D ( l ) = 2 achieves the same predictive performance with the exact algorithm in comparable number of epochs on all the datasets, while the training time is 5 times shorter on our largest dataset.
B ACKGROUNDS
We now brieﬂy review graph convolutional networks (GCNs) (Kipf & Welling, 2017) and the neighbor sampling (NS) algorithm
(Hamilton et al., 2017a).
2. G RAPH CONVOLUTIONAL NETWORKS
The original GCN was presented in a semi-supervised node classiﬁcation task (Kipf & Welling, 2017).
We follow this setting throughout this paper.
Generalization of GCN to other tasks can be found in Kipf & Welling (2016);
Berg et al.
; Schlichtkrull et al.
(2017) and Hamilton et al.
(2017b).
In the node classiﬁcation task, we have an undirected graph G =
( V , E ) with V = |V| vertices and E = |E| edges, where each vertex v consists of a feature vector x v and a label y v .
The label is only observed for some vertices V L
and we want to predict the label for the rest vertices
V U := V\V L .
The edges are represented as a symmetric V × V adjacency matrix A , where A v,v (cid:48) is the weight of the edge between v and v (cid:48) , and the propagation matrix P is a normalized version of
A : ˜ A =
A + I , ˜ D
vv = (cid:80) v (cid:48)
˜
A vv (cid:48) , and P =
˜ D − 12 ˜
A ˜ D − 12 .
A graph convolution layer is deﬁned as
[Equation]
where H ( l ) is the activation matrix in the l -th layer, whose each row is the activation of a graph node.
H (0) =
X is the input feature matrix, W ( l ) is a trainable weight matrix, σ ( · ) is an activation function, and Dropout p ( · ) is the dropout operation (Srivastava et al., 2014) with keep probability p .
Finally, the loss is deﬁned as L =
[Equation]
f
[Equation]
y v , Z
[Equation]
L ) v ) , where f ( · , · ) can be the square loss, cross entropy loss, etc., depending on the type of the label.
When P =
I , GCN reduces to a multi-layer perceptron (MLP) model which does not use the graph structure.
Comparing with MLP, GCN is able to utilize neighbor information for node classiﬁcation.
We deﬁne n ( v, L ) as the set of all the L -neighbors of node v , i.e., the nodes that are reachable from v within L hops.
It is easy to see from Fig.
1(a) that in an L -layer GCN, a node uses the information from all its L -neighbors.
This makes GCN more powerful than MLP, but also complicates the stochastic training, which utilizes an approximated gradient ∇L ≈
[Equation]
where V B ⊂ V L is a minibatch of training data.
The large receptive ﬁeld size | ∪ v ∈V B n ( v, L ) | per minibatch leads to high time complexity, space complexity and amount of IO.
See Table 1 for the average number of 1- and 2-neighbors of our datasets.
2. A LTERNATIVE NOTATION
We introduce alternative notations to help compare different algorithms.
Let U ( l ) =
P ˜ H ( l ) , or u
[Equation]
l ) v =
[Equation]
P
[Equation]
˜
[Equation] [Equation]
, we focus on studying how u v is computed based on node v ’s neighbors.
To keep notations simple, we omit all the subscripts and tildes, and exchange the ID of nodes
such
[Table]
Citeseer
[Table] [Table]
[Table]
that n ( v, 1) =
[ D ] + , 1 where D = | n ( v, 1) | is the number of neighbors.
We get the propagation rule u
= (cid:80) Dv =1 p v h v , which is used interchangeably with the matrix form U ( l ) =
P ˜ H ( l ) .
2. N EIGHBOR SAMPLING
To reduce the receptive ﬁeld size, Hamilton et al. (2017a) propose a neighbor sampling (NS) algorithm.
On the l -th layer, they randomly choose D ( l ) neighbors for each node, and develop an estimator u NS of u based on Monte-Carlo approximation u ≈ u NS = DD ( l ) (cid:80) v ∈ D ( l ) p v h v , where
D ( l ) ⊂
[ D ] + is a subset of D ( l ) neighbors.
In this way, they reduce the receptive ﬁeld size from | ∪ v ∈V B n ( v, L ) | to O ( |V B | (cid:81)
Ll =1 D ( l ) ) .
Neighbor sampling can also be written in a matrix form as ˜ H ( l ) NS =
Dropout p
( H ( l ) NS ) ,
Z ( l
[Equation] [Equation]
NS =
[Equation]
P (
[Equation]
where ˆ P ( l ) is a sparser unbiased estimator of P , i.e., E ˆ P ( l )
= P .
The approximate prediction Z ( L ) NS used for testing and for computing stochastic gradient 1 |V B | (cid:80) v ∈V B ∇ f ( y v , Z ( L ) CV,v ) during training.
The NS estimator u NS is unbiased.
However it has a large variance, which leads to biased prediction and gradients after the non-linearity in subsequent layers.
Due to the biased gradients, training with NS does not converge to the local optimum of GCN.
When D ( l ) is moderate, NS may has some regularization effect like dropout (Srivastava et al., 2014), where it drops neighbors instead of features.
However, for the extreme ease D ( l ) = 2 , the neighbor dropout rate is too high to reach high predictive performance, as we will see in Sec. 5.4.
Intuitively, making prediction solely depends on one neighbor is inferior to using all the neighbors.
To keep comparable prediction performance with the original GCN, Hamilton et al.
(2017a) use relatively large D (1)
= 10 and D (2) = 25 .
Their receptive ﬁeld size D (1) × D (2) = 250 is still much larger than MLP, which is 1 .
P REPROCESSING FIRST LAYER
We ﬁrst present a technique to preprocess the ﬁrst graph convolution layer, by approximating
A Dropout p ( X ) with Dropout p ( AX ) .
The model becomes
[Equation]
This approximation does not change the expectation because E (cid:2)
A Dropout p ( X ) (cid:3)
[Equation]
E (cid:2)
Dropout p ( AX ) (cid:3) , and it does not affect the predictive performance, as we shall see in Sec. 5.1.
The advantage of this modiﬁcation is that we can preprocess U (0) =
PH (0) = PX and takes U (0) as the new input.
In this way, the actual number of graph convolution layers is reduced by one — the ﬁrst layer is merely a fully connected layer instead of a graph convolution one.
Since most GCNs only have two graph convolution layers (Kipf & Welling, 2017; Hamilton et al., 2017a), this gives a signiﬁcant reduction of the receptive ﬁeld size from the number of L -neighbors | ∪ v ∈V B n ( v, L ) | to the number of L − 1 -neighbors | ∪ v ∈V B n ( v, L − 1) | .
The numbers are reported in Table 1.
C ONTROL VARIATE BASED STOCHASTIC APPROXIMATION
We now present two novel control variate based estimators that have smaller variance as well as stronger theoretical guarantees than NS.
4. C ONTROL VARIATE BASED ESTIMATOR
We assume that the model does not have dropout for now and will address dropout in Sec.
The idea is that we can approximate u = (cid:80) Dv =1 p v h v better if we know the latest historical activations ¯ h v of the neighbors, where we expect ¯ h v and h v are similar if the model weights do not change too fast during the training.
With the historical activations, we approximate
[Equation]
where v (cid:48) is a random neighbor, and ∆ h v (cid:48)
=
h v (cid:48) − ¯ h v (cid:48) .
For the ease of presentation, we assume that we only use the latest activation of one neighbor, while the implementation also include the node itself besides the random neighbor, so D ( l )
= 2 .
Using historical activations is cheap because they need not to be computed recursively using their neighbors’ activations, as shown in Fig.
1(c).
Unlike NS, we apply Monte-Carlo approximation on (cid:80) v p v ∆ h v instead of (cid:80) v p v h v .
Since we expect h v and ¯ h v to be close, ∆ h v will be small and u CV should have a smaller variance than u NS .
Particularly, if the model weight is kept ﬁxed, ¯ h v should be eventually equal with h v , so
that u CV =
+ (cid:80) Dv =1 p v ¯ h v =
(cid:80)
Dv =1 p v h v = u , i.e., the estimator has zero variance.
The term CV = u CV − u NS = − Dp v (cid:48)
¯ h v (cid:48)
+ (cid:80) Dv =1 p v ¯ h v is a control variate (Ripley, 2009, Chapter 5), which has zero mean and large correlation with u NS , to reduce its variance.
We refer this stochastic approximation algorithm as CV, and we will formally analyze the variance and prove the convergence of the training algorithm using CV for stochastic gradient in subsequent sections.
In matrix form, CV computes the approximate predictions as follows, where we explicitly write down the iteration number i and add the subscript CV to the approximate activations
[Equation]
where ¯ h ( l ) CV,i,v stores the latest activation of node v on layer l computed before time i .
Formally, let m ( l ) i ∈
R V × V be a diagonal matrix, and ( m ( l ) i ) vv = 1 if ( ˆ P ( l ) i ) v (cid:48) v > 0 for any v (cid:48) .
After ﬁnishing one iteration we update history ¯ H with the activations computed in that iteration as Eq.
4. C ONTROL VARIATE FOR DROPOUT
With dropout, the activations H are no longer deterministic.
They become random variables whose randomness come from different dropout conﬁgurations.
Therefore, ∆ h v = h v − ¯ h v is not necessarily small even if h v and ¯ h v have the same distribution.
We develop another stochastic approximation algorithm, control variate for dropout (CVD), that works well with dropout.
Our method is based on the weight scaling procedure (Srivastava et al., 2014) to approximately compute the mean µ v :=
E [ h v ] .
That is, along with the dropout model, we can run a copy of the model with no dropout to obtain the mean µ v , as illustrated in Fig. 1(d).
With the mean, we can obtain a better stochastic approximation by separating the mean and variance u =
D (cid:88) [Equation] =
[Equation]
where ¯ µ v is the historical mean activation, obtained by storing µ v instead of h v , and ∆ µ = µ v − ¯ µ v .
u CV D an unbiased estimator of u because the term √ Dp v (cid:48)
( h v (cid:48) − µ v (cid:48) ) has zero mean, and the
Monte-Carlo approximation (cid:80) Dv =1 p v ( µ v − ¯ µ v )
≈
Dp v (cid:48)
∆ µ v (cid:48) does not change the mean.
The approximation (cid:80) Dv =1 p v ( h v − µ v )
≈ √
Dp v (cid:48)
( h v (cid:48) − µ v (cid:48) ) is made by assuming h v ’s to be independent
Gaussians, which we will soon clarify.
The pseudocodes for CV and CVD are in Appendix E.
4. V ARIANCE ANALYSIS
NS, CV and CVD are all unbiased estimators of u = (cid:80) v p v h v .
We analyze their variance in a simple independent Gaussian case, where we assume that activations are Gaussian random variables
[Table]
h v ∼ N ( µ v , s 2 v ) following Wang & Manning (2013).
Without loss of generality, we assume that all the activations
h v are one dimensional.
We also assume that all the activations h 1 , . .
. , h D and historical activations ¯ h 1 , . . .
, ¯ h D are independent, where the historical activations ¯ h v ∼ N (¯ µ v , ¯ s 2 v ) .
We introduce a few more notations.
∆ µ v and ∆ s 2 v are the mean and variance of ∆ h v = h v − ¯ h v , where ∆ µ v = µ v − ¯ µ v and ∆ s 2 v = s 2 v + ¯ s 2 v .
µ and s 2 are the mean and variance of (cid:80) v p v h v , where µ = (cid:80) v p v µ v and s 2 = (cid:80) v p 2 v s 2 v .
Similarly, ∆ µ , ∆ s 2 , ¯ µ and ¯ s 2 are the mean and variance of (cid:80) v p v ∆ h v and (cid:80) v p v ¯ h v , respectively.
With these assumptions and notations, we list the estimators and variances in Table 2, where the derivations can be found in Appendix C.
We decompose the variance as two terms: variance from
Monte-Carlo approximation (VMCA) and variance from dropout (VD).
If the model has no dropout, the activations have zero variance, i.e., s v = ¯ s v = 0 , and the only source of variance is VMCA.
We want VMCA to be small.
As in Table 2, the VMCA for the exact estimator is 0.
For the NS estimator, VMCA is 12 (cid:80) v,v (cid:48)
( p v µ v − p v (cid:48) µ v (cid:48) ) 2 , whose magnitude depends on the pairwise difference ( p v µ v − p v (cid:48) µ v (cid:48) ) 2 , and VMCA is zero
if and only if p v µ v = p v (cid:48) µ v (cid:48) for all v, v (cid:48) .
Similarly, VMCA for both CV and CVD estimators is 12 (cid:80) v,v (cid:48)
( p v ∆ µ v − p v (cid:48)
∆ µ v (cid:48) ) 2 , which should be smaller than NS estimator’s VMCA if ( p v ∆ µ v − p v (cid:48)
∆ µ v (cid:48) )
2 < ( p v µ v − p v (cid:48) µ v (cid:48) ) 2 , which is likely because ∆ µ v should be smaller than µ v .
Since CV and CVD estimators have the same
VMCA, we adopt the CV estimator for models without dropout, due to its simplicity.
The VD of the exact estimator is s 2 , which is overestimated by both NS and CV.
NS overestimates
VD by D times, and CV has even larger VD.
Meanwhile, the VD of the CVD estimator is the same as the exact estimator, indicating CVD to be the best estimator for models with dropout.
4. E XACT TESTING
Besides smaller variance, CV also has stronger theoretical guarantees than NS.
We can show that during testing, CV’s prediction becomes exact after a few testing epochs.
For models without dropout, we can further show that training using the stochastic gradients obtained by CV converges to GCN’s local optimum.
We present these results in this section and Sec. 4.5.
Note that the analysis does not need the independent Gaussian assumption.
Given a model W , we compare the exact predictions (Eq. 1) and CV’s approximate predictions (Eq. 5,6) during testing, which uses the deterministic weight scaling procedure.
To make predictions, we run forward propagation by epochs.
In each epoch, we randomly partition the vertex set V as I minibatches V 1 , . .
, V I and in the i -th iteration, we run a forward pass to compute the prediction for nodes in V i .
Note that in each epoch we scan all the nodes instead of just testing nodes, to ensure that the activation of each node is computed at least once per epoch.
The following theorem reveals the connection of the exact predictions and gradients, and their approximate versions by CV.
Theorem 1.
For a ﬁxed W and any i > LI we have: (1) (Exact Prediction)
The activations computed by CV are exact, i.e., Z ( l ) CV,i =
Z ( l ) for each l ∈
[ L ] and H ( l ) CV,i = H ( l ) for each l ∈
[ L − 1] .
(2) (Unbiased Gradient)
The stochastic gradient g CV,i ( W ) :
= 1 |V B | (cid:80) v ∈V B ∇ W f
( y v , z ( L ) CV,i,v ) is an unbiased estimator of GCN’s gradient, i.e., E ˆ P, V B g CV,i ( W )
= ∇ W 1 | V | (cid:80) v ∈V f ( y v , z ( L ) v ) .
Theorem 1 shows that at testing time, we can run forward propagation with CV for L epoches and get the exact prediction.
This outperforms NS, which cannot recover the exact prediction.
Comparing with directly making exact predictions by a batch algorithm, CV is more scalable because it does not need to load the entire graph into memory.
The proof can be found in Appendix A.
4. C ONVERGENCE GUARANTEE
The following theorem shows that for a model without dropout, training using CV’s approximated gradients converges to a local optimum of GCN, regardless of the neighbor sampling size D ( l ) .
Therefore, we can choose arbitrarily small D ( l ) without worrying about the convergence.
Theorem 2.
Assume that (1) all the activations are ρ -Lipschitz, (2) the gradient of the cost function ∇ z f ( y, z ) is ρ -Lipschitz and bounded, (3) (cid:107) g CV ( W ) (cid:107) ∞ and (cid:107)
g
( W ) (cid:107) ∞
=
(cid:107)∇L ( W ) (cid:107) ∞ are bounded by G > 0 for all ˆ P, V B and W .
The loss L ( W ) is ρ -smooth, i.e., |L ( W 2 ) − L ( W [Equation] − (cid:104)∇ L ( W 1 ) , W
[Equation] − W 1 (cid:105)| ≤ [Equation] 2 (cid:107) W 2 − W 1 (cid:107) 2 ∀ W 1 , W 2 , where (cid:104) A, B (cid:105) =
tr ( A (cid:62) B ) is the inner product of matrix A and matrix B .
We randomly run SGD for R ≤ N iterations, where
P R ( R = i ) =
[Equation] Then, for the updates W i +
=
W i − γ
i g CV ( W i ) and step sizes γ i = min { 1 ρ , 1 √ N } , there exists constants K 1 and K 2 which are irrelevant with N , s.t. ∀ N > LI ,
[Equation]
The proof can be found in Appendix B. Particularly, lim N →∞ E R ∼ P R E ˆ P, V B (cid:107)∇L ( W R ) (cid:107) 2 = 0 . Therefore, our algorithm converges to a local optimum.
4. T IME COMPLEXITY AND IMPLEMENTATION DETAILS
Finally we discuss the time complexity of different algorithms.
We decompose the time complexity as sparse time complexity for sparse-dense matrix multiplication such as P ˜ H ( l ) , and dense time complexity for dense-dense matrix multiplication such as U ( l ) W ( l ) .
Assume that the node feature is K -dimensional and the ﬁrst hidden layer is A -dimensional, the batch GCN has O ( EK ) sparse and O ( V KA ) dense time complexity per epoch.
NS has O ( V (cid:81)
Ll =1 D ( l ) K ) sparse and
O ( V (cid:81)
Ll =2 D ( l ) KA ) dense time complexity per epoch.
The dense time complexity of CV is the same as NS.
The sparse time complexity depends on the cost of computing the sum (cid:80) v p v ¯ µ v .
There are V (cid:81)
Ll =2 D ( l ) such sums to compute on the ﬁrst graph convolution layer, and overall cost is not larger than O ( V D (cid:81)
Ll =2 D ( l ) K ) , if we subsample the graph such that the max degree is D , following Hamilton et al. (2017a).
The sparse time complexity is D/D (1) times higher than NS.
Our implementation is similar as Kipf & Welling (2017).
We store the node features in the main memory, without assuming that they ﬁt in GPU memory as Hamilton et al.
(2017a), which makes our implementation about 2 times slower than theirs.
We keep the histories in GPU memory for efﬁciency since they are only LH < K dimensional.
E XPERIMENTS
We examine the variance and convergence of our algorithms empirically on six datasets, including
Citeseer, Cora, PubMed and NELL from Kipf & Welling (2017) and Reddit, PPI from Hamilton et al.
(2017a), as summarized in Table 1.
To measure the predictive performance, we report Micro-F1 for the multi-label PPI dataset, and accuracy for all the other multi-class datasets.
We use the same model architectures with previous papers but slightly different hyperparameters (see Appendix D for the details).
We repeat the convergence experiments 10 times on Citeseer, Cora, PubMed and
NELL, and 5 times on Reddit and PPI.
The experiments are done on a Titan X (Maxwell) GPU.
5. I MPACT OF PREPROCESSING
We ﬁrst examine the approximation in Sec. 3 that switches the order of dropout and aggregating the neighbors.
Let M0 be the original model (Eq. 1) and M1 be our approximated model (Eq. 3), we compare three settings: (1) M0, D ( l ) = ∞ is the exact algorithm without any neighbor sampling.
M1+PP, D ( l ) = ∞ changes the model from M0 to M1.
Preprocessing does not affect the training for
D ( l ) = ∞ .
(3) M1+PP, D ( l ) = 20 uses NS with a relatively large number of neighbors.
In Tablewe can see that all the three settings performs similarly, i.e., our approximation does not affect the predictive performance.
Therefore, we use M1+PP, D ( l ) = 20 as the exact baseline in following convergence experiments because it is the fastest among these three settings.
[Figure]
Citeseer
Cora
PubMed
NELL
PPI
[Figure]
[Figure] [Figure]
0 ± .
[Figure]
9 ± .96 .
2 ± .
M1+PP, D ( l ) = ∞ 70 .
9 ± .82 . 0 ± .78 .
7 ± .64 .
9 ± 1 .97 .
8 ± .96 .
3 ± .
M1+PP, D ( l ) =70 .
9 ± .81 .
9 ± .78 .
[Table]
± .64 .
2 ±
[Table] [Table]
± .96 .
3 ± .
[Figure]
5. C ONVERGENCE WITH NO DROPOUT
We now study how fast our algorithms converge with a very small neighbor sampling size D ( l ) = 2 .
We compare the following algorithms: (1) Exact, which is M1+PP, D ( l ) = 20 in Sec.
5.1 as a surrogate of the exact algorithm.
(2) NS, which is the NS algorithm with no preprocessing and
D ( l ) = 2 .
(3) NS+PP, which is same with NS but uses preprocessing.
(4) CV+PP, which replaces the NS estimator in NS+PP with the CV estimator.
(5) CVD+PP, which uses the CVD estimator.
We ﬁrst validate Theorem 2, which states that CV+PP converges to a local optimum of Exact, for models without dropout, regardless of D ( l ) .
We disable dropout and plot the training loss with respect to number of epochs as Fig.
We can see that CV+PP can always reach the same training loss with Exact, which matches the conclusion of Theorem 2.
Meanwhile, NS and NS+PP have a higher training loss because their gradients are biased.
5. C ONVERGENCE WITH DROPOUT
Next, we compare the predictive accuracy obtained by the model trained by different algorithms, with dropout turned on.
We use different algorithms for training and the same Exact algorithm for testing, and report the validation accuracy at each training epoch.
The result is shown in Fig.
We ﬁnd that CVD+PP is the only algorithm that is able to reach comparable validation accuracy with Exact on all datasets.
Furthermore, its convergence speed with respect to the number of epochs is comparable with Exact despite its D ( l ) is 10 times smaller.
Note that CVD+PP performs much better than Exact on the PubMed dataset; we suspect it ﬁnds a better local optimum.
[Figure]
Alg.
Valid.
Epochs
Time
Sparse
[Figure]
acc
[Figure]
GFLOP
TFLOP
Exact 96.
7.17 NS 94.
[Figure]
PP 96.
7.36 CV+PP 96.
CVD+PP 96.
[Figure]
[Figure]
Meanwhile, simper algorithms CV+PP and NS+PP work acceptably on most of the datasets.
CV+PP reaches a comparable accuracy with Exact for all datasets except PPI.
NS+PP works slightly worse but the ﬁnal validation accuracy is still within 2%.
These algorithms can be adopted if there is no strong need for predictive performance.
We however emphasize that exact algorithms must be used for making predictions, as we will show in Sec. 5.4.
Finally, the algorithm NS without preprocessing works much worse than others, indicating the signiﬁcance of our preprocessing strategy.
5. F URTHER A NALYSIS ON T IME COMPLEXITY , T ESTING ACCURACY AND V ARIANCE
Table 4 reports the average number of epochs, time, and total number of ﬂoating point operations to reach a given 96% validation accuracy on the largest Reddit dataset.
Sparse and dense computations are deﬁned in Sec. 4.6.
We found that CVD+PP is about 5 times faster than Exact due to the signiﬁcantly reduced receptive ﬁeld size.
Meanwhile, simply setting D ( l ) = 2 for NS does not converge to the given accuracy.
We compare the quality of the predictions made by different algorithms, using the same model trained by Exact in Fig.
As Thm.
1 states, CV reaches the same testing accuracy as Exact, while
NS and NS+PP perform much worse.
Testing using exact algorithms (CV or Exact) corresponds to the weight scaling algorithm for dropout (Srivastava et al., 2014).
Finally, we compare the average bias and variance of the gradients per dimension for ﬁrst layer weights relative to the weights’ magnitude in Fig.
For models without dropout, the gradient of
CV+PP is almost unbiased.
For models with dropout, the bias and variance of CV+PP and CVD+PP are ususally smaller than NS and NS+PP, as we analyzed in Sec. 4.3.
C ONCLUSIONS
The large receptive ﬁeld size of GCN hinders its fast stochastic training.
In this paper, we present a preprocessing strategy and two control variate based algorithms to reduce the receptive ﬁeld size.
Our algorithms can achieve comparable convergence speed with the exact algorithm even the neighbor sampling size D ( l ) = 2 , so that the per-epoch cost of training GCN is comparable with training
MLPs.
We also present strong theoretical guarantees, including exact prediction and convergence to
GCN’s local optimum, for our control variate based algorithm.
A P ROOF OF T HEOREM
Proof.
We prove by induction.
After the ﬁrst epoch the activation h (0) i,v is at least computed once for each node v , so ¯ H (0) CV,i = H (0) CV,i = H (0) for all
i > I .
Assume that we have ¯ H ( l ) CV,i = H ( l ) CV,i =
H ( l ) for all i > ( l + 1) I .
Then for all i > ( l + 1)
I
[Equation]
After one more epoch, all the activations h ( l +1) CV,i,v are computed at least once for each v , so ¯ H ( l +1) CV,i =
H ( l +1) CV,i = H ( l +1) for all i > ( l + 2) I .
By induction, we know that after LI steps, we have ¯ H ( L − 1) CV,i =
H ( L − 1) CV,i =
H ( L − 1) .
By Eq.
7 we have ¯ Z ( L ) CV,i =
Z ( L ) .
We omit the time subscript i and denote f CV,v :=
f ( y v , z ( L ) CV,v ) .
By back propagation, the approximated gradients by CV can be computed as follows
[Equation]
where ◦ is the element wise product and σ (cid:48)
( Z ( l ) CV ) is the element-wise derivative.
Similarly, denote f v :=
f ( y v , Z ( l ) v ) , the exact gradients can be computed as follows
[Equation]
Applying E ˆ P =
E ˆ P (1) ,..., ˆ P ( L ) to both sides of Eq.
[Equation]
and utilizing
• 1’s conclusion that after L epoches, Z ( l ) CV = Z ( l ) , so ∇ Z ( L ) CV f CV,v is also determinstic.
• E
ˆ P [ ∇ Z ( l ) f CV,v ] =
E ˆ P ( l ) ,..., ˆ P ( L ) [ ∇ Z ( l ) f CV,v ] .
• E
ˆ P [ ∇ H ( l ) f CV,v ] =
E ˆ P ( l ) ,..., ˆ P ( L ) [ ∇ H ( l ) f CV,v ] .
we have
[Equation]
Comparing Eq.
10 and Eq.
9 we get
[Equation]
so
[Equation]
B P ROOF OF T HEOREM
We proof Theorem 2 in 3 steps:
Lemma 1: For a sequence of weights W (1) , . . .
, W ( N ) which are close to each other, CV’s approximate activations are close to the exact activations.
Lemma 2:
For a sequence of weights W (1) , . . .
, W ( N ) which are close to each other, CV’s gradients are close to be unbiased.
Theorem 2: An SGD algorithm generates the weights that changes slow enough for thez gradient bias goes to zero, so the algorithm converges.
The following proposition is needed in our proof
Proposition 1.
Let (cid:107) A (cid:107) ∞ =
max ij | A ij | , then
• (cid:107) AB (cid:107) ∞ ≤ col ( A ) (cid:107) A (cid:107) ∞ (cid:107) B (cid:107) ∞ , where col ( A ) is the number of columns of the matrix A . • (cid:107) A ◦ B (cid:107) ∞ ≤ (cid:107) A (cid:107) ∞ (cid:107) B (cid:107) ∞ .
• (cid:107) A + B (cid:107) ∞ ≤ (cid:107) A (cid:107) ∞ + (cid:107) B (cid:107) ∞ . Proof. [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
We deﬁne C := max { col ( P ) , col ( H (0) ) , . .
, col ( H ( L ) ) } .
B. P ROOF OF L EMMA
Proposition 2.
There are a series of T inputs X 1 , . . .
, X T , X CV, 1 , . . .
, X CV,T and weights
W 1 , . .
, W T feed to an one-layer GCN with CV
Z CV,i = (cid:16) ˆ P
i
( X i − ¯ X i )
+ P ¯ X
i (cid:17) W i ,
H CV,i =
σ ( Z CV,i ) , ¯ H CV,i +
=
s i H CV,i +(1 −
s i )
¯ H CV,i .
and an one-layer exact GCN
Z i = PX
i W i ,
H i =
σ ( Z i ) .
If
The activation σ ( · ) is ρ -Lipschitz; 2.
(cid:107) X CV,i − X CV,j (cid:107) ∞ < (cid:15) and (cid:107) X CV,i − X
i (cid:107) ∞ < (cid:15) for all i, j ≤ T and (cid:15) > 0 .
Then there exists some K > 0 , s.t., (cid:107) H CV,i − H CV,j (cid:107) ∞ <
K(cid:15) and (cid:107) H CV,i − H
i (cid:107) ∞ < K(cid:15) for all
I < i, j ≤ T , where I is the number of iterations per epoch.
Proof.
Because for all i > I , the elements of ¯ X CV,i are all taken from previous epochs, i.e.,
X CV, 1 , . . .
, X CV,i − 1 , we know that
[Equation]
By triangular inequality, we also know
[Equation]
Since (cid:107) X CV, 1 (cid:107) ∞ , . . .
, (cid:107) X CV,T (cid:107) ∞ are bounded, (cid:13)(cid:13) ¯ X CV,i (cid:13)(cid:13) ∞ is also bounded for i > I .
Then, (cid:107) H CV,i − H CV,j (cid:107) ∞ ≤ ρ (cid:107) Z CV,i − Z CV,j (cid:107) ∞ [Equation]
[Table]
ˆ P j (cid:13)(cid:13)(cid:13) ∞ (cid:107) W i (cid:107) ∞ + (cid:13)(cid:13)(cid:13)
ˆ P j (cid:13)(cid:13)(cid:13) ∞ (cid:107) W i −
W j (cid:107) ∞ + 3 (cid:13)(cid:13)(cid:13)
ˆ P j (cid:13)(cid:13)(cid:13) ∞ (cid:107) W
i (cid:107) ∞ + (cid:13)(cid:13)(cid:13) ˆ P j (cid:13)(cid:13)(cid:13) ∞ (cid:13)(cid:13) ¯ X CV,j (cid:13)(cid:13) ∞ ] =
K 1 (cid:15), and (cid:107) H CV,i − H
i (cid:107)
[Equation]
ρ (cid:107) Z CV,i − Z i (cid:107) ∞ ≤ ρ (cid:13)(cid:13)(cid:13)(cid:16) [Equation] [Equation] CV,
[Equation]
− ¯ X CV,
i
[Equation]
CV
[Equation]
i
[Equation]
(cid:17)(cid:13)(cid:13)(cid:13)
[Equation]
W i
[Equation]
≤ K
[Equation]
The following lemma bounds CV’s approximation error of activations
Lemma 1.
Given a sequence of model weights W 1 , . . .
, W T .
If (cid:107) W i − W j (cid:107) ∞ < (cid:15), ∀ i, j , and all the activations are ρ -Lipschitz, there exists K > 0 , s.t.,
• (cid:13)(cid:13)
H li − H lCV,i (cid:13)(cid:13) ∞ <
K(cid:15), ∀ i > LI, l = 1 , . . .
, L − 1 , • (cid:13)(cid:13)
Z li − Z lCV,i (cid:13)(cid:13) ∞ <
K(cid:15), ∀ i > LI, l = 1 , . . .
, L .
Proof.
We prove by induction.
Because H
= X is constant, ¯ H 0 CV,i =
H
i after I iterations.
So
H 1 CV,i =
σ ( (cid:16) ˆ P
i ( H 0 CV,i − ¯ H 0 CV,i ) + P ¯ H 0 CV,i (cid:17) W 0 i =
σ ( PXW 0 i ) =
H 1 i , and [Equation] ,i [Equation]
∞
[Equation]
Repeatedly apply Proposition B.1 for L − 1 times, we get the intended results.
B. P ROOF OF L EMMA
The following lemma bounds the bias of CV’s approximate gradient
Lemma 2.
Given a sequence of model weights W 1 , . . .
, W T , if
(cid:107) W i − W j (cid:107) ∞ < (cid:15), ∀ i, j , 2.
all the activations are ρ -Lipschitz, 3.
the gradient of the cost function ∇ z f ( y, z ) is ρ -Lipschitz and bounded,
then there exists K > 0 , s.t.,
[Equation]
Proof.
By Lipschitz continuity of ∇ z f ( y, z ) and Lemma 1, there exists K > 0 , s.t.,
[Equation]
−
[Equation]
Assume that (cid:13)(cid:13)(cid:13) E ˆ P ∇ Z ( l +1) CV f CV,v − ∇ Z ( l +1)
f v (cid:13)(cid:13)(cid:13) ∞ < K 1 (cid:15) , we now prove that there exists K > 0 , s.t., (cid:13)(cid:13)(cid:13)
E
ˆ P ∇ Z ( l ) CV f CV,v − ∇ Z ( l ) f v (cid:13)(cid:13)(cid:13) ∞ < K(cid:15) .
By Eq.
9, Eq.
10 and Lemma 1, we have
[Equation]
[Table]
and
[Table]
By induction we know that for l = 1 , . . .
, L there exists K , s.t.,
[Table]
Again by Eq. 9, Eq.
[Table]
and Lemma 1,
[Equation]
[Table]
Finally,
[Equation]
B. P ROOF OF T HEOREM
Proof.
This proof is a modiﬁcation of Ghadimi & Lan (2013), but using biased stochastic gradients instead.
We assume the algorithm is already warmed-up for LI steps with the initial weights W 0 , so that Lemma 2 holds for step i > 0 .
Denote δ i =
g CV ( W i ) − ∇L ( W i ) .
By smoothness we have
L ( W i +1 ) ≤ L ( W i )
+ (cid:104)∇L ( W i ) , W i +1 − W i (cid:105)
{++ ρ 
+}
{+γ 
+}
i (cid:107) g CV
[Equation]
W i ) (cid:107)= L ( W i
[Equation]
− γ i (cid:104)∇L ( W
[Equation]
[Equation]
g CV
[Equation]
W
[Equation]
) (cid:105)
+
[Equation]
γ
[Equation]
(cid:107) g CV ( W i ) (cid:107)
[Equation] Consider the sequence of LI + 1 weights W i − LI , . .
, W i .
[Equation]
By Lemma [Equation] there exists K > 0 , s.t.
[Equation]
Assume that W is D -dimensional,
[Equation]
where K 1 = KLIDG 2 and K 2 = DK
L 2 B 2 G 2 .
Taking E ˆ P, V B to both sides of Eq.
14 we have
[Equation]
Summing up the above inequalities and re-arranging the terms [Equation] we obtain,
[Equation]
Dividing both sides by (cid:80) Ni =1 ( γ i − ργ 2 i 2 ) , [Equation] −
[Equation]
Taking γ i = [-
-]γ :=
min { 1 ρ , 1 √ N } , for all i = 1 , . . , N , we have [Equation] Particularly, when N → ∞ , we have E R ∼ P R E ˆ P, V B (cid:107)∇L ( W R ) (cid:107) 2 = 0 , which implies that the gradient is asymptotically unbiased.
C D ERIVATION OF THE VARIANCE
[Equation]
[Table]
[Equation]
Var [ u CV D ] [Equation] E
[Equation]
D E XPERIMENT SETUP
In this sections we describe the details of our model architectures.
We use the Adam optimizer Kingma & Ba (2014) with learning rate 0.01.
• Citeseer, Cora, PubMed and NELL:
We use the same architecture as Kipf & Welling (2017): two graph convolution layers with one linear layer per graph convolution layer.
We use 32 hidden units, 50% dropout rate and 5 × 10 − 4 L2 weight decay for Citeseer, Cora and PubMed and 64 hidden units, 10% dropout rate and 10 − 5 L2 weight decay for NELL.
• PPI and Reddit: We use the mean pooling architecture proposed by Hamilton et al. (2017a).
We use two linear layers per graph convolution layer.
We set weight decay to be zero, dropout rate to be 0.2%, and adopt layer normalization (Ba et al., 2016) after each linear layer.
We use 512 hidden units for PPI and 128 hidden units for Reddit.
We ﬁnd that our architecture can reach 97.8% testing micro-F1 on the PPI dataset, which is signiﬁcantly higher than 59.8% reported by Hamilton et al.
(2017a).
We ﬁnd the improvement is from wider hidden layer, dropout and layer normalization. minibatch V B ⊂ V do
Randomly sample propagation matrices ˆ P (0) , . . .
, ˆ P ( L − 1)
Compute the receptive ﬁelds m (0) , . . .
, m ( L − 1) (Forward propgation) for each layer l ← 0 to L − 1 do
Z ( l +1) ← (cid:16) ˆ P ( l ) ( H ( l ) − ¯ H ( l ) + P ¯ H ( l ) (cid:17) W ( l )
H ( l +1) ← σ ( Z ( l +1) ) end for
Compute the loss L = 1 |V B | (cid:80) v ∈V B f ( y v , Z ( L ) v ) (Backward propagation)
W ← W − γ i ∇ W L (Update historical activations) for each layer l ← 0 to L − 1 do ¯
H ( l ) ← m ( l ) H ( l ) + (1 − m ( l ) )
¯ H ( l ) end for end for for each
V B ⊂ V do
Randomly sample propagation matrices ˆ P (0) , . . .
, ˆ P ( L − 1)
Compute the receptive ﬁelds m (0) , . . .
, m ( L − 1) (Forward propgation) for each layer l ← 0 to L − 1 do
U ← (cid:16) ¯ P ( l ) ( H ( l ) − µ ( l ) )
+
ˆ P ( l ) ( µ ( l ) − ¯ µ ( l ) )
+ P ¯ H ( l ) (cid:17)
H ( l +1) ← σ ( Dropout p ( U ) W ( l ) ) µ ( l +1) ← σ ( UW ( l ) ) end for
Compute the loss L = 1 |V B | (cid:80) v ∈V B f ( y v , H ( L ) v ) (Backward propagation)
W ← W − γ i ∇ W L (Update historical activations) for each layer l ← 0 to L − 1 do ¯ µ ( l ) ← m ( l ) end end for
E
P SEUDOCODE
E. T RAINING WITH THE CV ESTIMATOR
Alg. 1 depicts the training algorithm using the CV estimator in Sec. 4.1.
We perform forward propagation according to Eq.
(5,6), compute the stochastic gradient, and then update the historical activations ¯ H ( l ) according to Eq.
We omit the subscripts CV and the iteration number i for concise.
Let W =
( W (0) , . . .
, W ( L − 1) ) be all the trainable parameters, the gradient ∇ W L is computed automatically by frameworks such as TensorFlow.
The diagonal matrix m ( l ) denotes the receptive ﬁeld at layer l , i.e., the nodes that need to be computed in order to approximate z ( l ) v for v in the minibatch V B .
We only need to compute and update the activations H ( l ) , Z ( l ) , ¯ H ( l ) for nodes in m ( l ) .
[Figure]
[Table]
E. T RAINING WITH THE CVD ESTIMATOR
Training with the CVD estimator is similar with the CV estimator, except it runs two versions of the network, with and without dropout, to compute the samples H and their mean µ of the activation.
The matrix ¯ P v,v (cid:48)
=
ˆ P v,v (cid:48) /
(cid:112) | n ( v, 1) | , where | n ( v, 1) | is the degree of node v .
F E XPERIMENT FOR 3- LAYER GCN S
We test 3-layer GCNs on the Reddit dataset.
The settings are the same with 2-layer GCNs in Sec.
To ensure the exact algorithm can run in a reasonable amount of time, we subsample the graph so that the maximum degree is 10.
The convergence result is shown as Fig. 6, which is similar with the two-layer models.
The time consumption to reach 0.94 testing accuracy is shown in Table 5.
G J USTIFICATION OF THE INDEPENDENT G AUSSIAN ASSUMPTION
We justify the independent Gaussian assumption in Sec.
4.3 by showing that for a 2-layer GCN with the ﬁrst layer pre-processed, the neighbor’s activations are independent.
Without loss of generality, suppose that we want to compute z (2)1 , and the neighbors of node 1 are 1 , . . .
, D .
By Eq.
(1), h (1) v = ( φ v ◦ u (0) v ) W (0) is a random variable with respect to φ v , where φ v ∼ Bernoulli ( p ) is the dropout mask and u (0) v =
( PH (0) ) v .
The indepedent Gaussian assumption states that
h (1) v is a Gaussian random variable with diagonal covariance; 2.
h (1) v and
h (1) v
(cid:48) are independent
for
v (cid:54) =
v (cid:48) .
Assumption 1 is not GCN-speciﬁc and is discussed in Wang & Manning (2013), we now prove assumption 2 by the following lemma.
Lemma 3.
If a and b are independent random variables, then their transformations f 1 ( a ) and f 2 ( b ) are independent.
Because for any event A and B , P ( f 1 ( a ) ∈ f 1 ( A ) , f 2 ( b ) ∈ f 2 ( B ))
=
P ( a ∈ A, b ∈ B ) =
P ( a ∈ A ) P ( b ∈ B ) =
P ( f 1 ( a ) ∈ f 1 ( A )) P ( f 2 ( B ) ∈ f 2 ( B )) , where f 1 ( A ) = { f 1 ( a ) | a ∈ A } and f 2 ( B ) = { f 2 ( b )
| b ∈ B } .
Let h (1) v = f 1 ( φ v ) :
= σ (cid:16) ( φ v ◦ u (0) v ) W (0) (cid:17) and h (1) v (cid:48) =
f 1 ( φ v (cid:48) ) :
= σ (cid:16) ( φ v (cid:48) ◦ u (0) v (cid:48) )
W (0) (cid:17) , because φ v and φ v (cid:48) are independent Bernoulli random variables, h (