GraphSAINT : G RAPH S AMPLING B ASED I NDUCTIVE
L EARNING M ETHOD
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph Convolutional Networks (GCNs) are powerful models for learning representations of attributed graphs.
To scale GCNs to large graphs, state-of-the-art methods use various layer sampling techniques to alleviate the “neighbor explosion” problem during minibatch training.
We propose GraphSAINT , a graph sampling based inductive learning method that improves training efﬁciency and accuracy in a fundamentally different way.
By changing perspective, GraphSAINT constructs minibatches by sampling the training graph, rather than the nodes or edges across
GCN layers.
Each iteration, a complete GCN is built from the properly sampled subgraph.
Thus, we ensure ﬁxed number of well-connected nodes in all layers.
We further propose normalization technique to eliminate bias, and sampling algorithms for variance reduction.
Importantly, we can decouple the sampling from the forward and backward propagation, and extend GraphSAINT with many architecture variants (e.g., graph attention, jumping connection).
GraphSAINT demonstrates superior performance in both accuracy and training time on ﬁve large graphs, and achieves new state-of-the-art F1 scores for PPI (0.995) and Reddit (0.970).
I NTRODUCTION
Recently, representation learning on graphs has attracted much attention, since it greatly facilitates tasks such as classiﬁcation and clustering (Wu et al., 2019;
Cai et al., 2017).
Current works on Graph
Convolutional Networks (GCNs) (Hamilton et al., 2017; Chen et al., 2018b; Gao et al., 2018; Huang et al., 2018; Chen et al., 2018a) mostly focus on shallow models (2 layers) on relatively small graphs.
Scaling GCNs to larger datasets and deeper layers still requires fast alternate training methods.
In a GCN, data to be gathered for one output node comes from its neighbors in the previous layer.
Each of these neighbors in turn, gathers its output from the previous layer, and so on.
Clearly, the deeper we back track, the more multi-hop neighbors to support the computation of the root.
The number of support nodes (and thus the training time) potentially grows exponentially with the GCN depth.
To mitigate such “neighbor explosion”, state-of-the-art methods use various layer sampling techniques.
The works by Hamilton et al.
; Ying et al.
(2018a)
; Chen et al.
(2018a) ensure that only a small number of neighbors (typically from 2 to 50) are selected by one node in the next layer.
Chen et al.
(2018b)
and Huang et al.
(2018) further propose samplers to restrict the neighbor expansion factor to 1, by ensuring a ﬁxed sample size in all layers.
While these methods signiﬁcantly speed up training, they face challenges in scalability, accuracy or computation complexity.
Present work We present GraphSAINT (Graph SAmpling based INductive learning meThod) to efﬁciently train deep GCNs.
GraphSAINT is developed from a fundamentally different way of minibatch construction.
Instead of building a GCN on the full training graph and then sampling across the layers, we sample the training graph ﬁrst and then build a full GCN on the subgraph.
Our method is thus graph sampling based.
Naturally, GraphSAINT resolves “neighbor explosion”, since every GCN of the minibatches is a small yet complete one.
On the other hand, graph sampling based method also brings new challenges in training.
Intuitively, nodes of higher inﬂuence on each other should have higher probability to form a subgraph.
This enables the sampled nodes to “support” each other without going outside the minibatch.
Unfortunately, such strategy results in non-identical node sampling probability, and introduces bias in the minibatch estimator.
To address this issue, we develop normalization techniques so that the feature learning does not give preference to nodes more frequently sampled.
To further improve training quality, we perform variance reduction analysis, and design light-weight sampling algorithms by quantifying “inﬂuence” of neighbors.
Experiments on
GraphSAINT using ﬁve large datasets show signiﬁcant performance gain in both training accuracy and time.
We also demonstrate the ﬂexibility of GraphSAINT by integrating our minibatch method with popular GCN architectures such as JK-net
(Xu et al., 2018) and GAT (Veliˇckovi´c et al., 2017).
The resulting deep models achieve new state-of-the-art F1 scores on PPI (0.995) and Reddit (0.970).
R ELATED W ORK
A neural network model that extends convolution operation to the graph domain is ﬁrst proposed by Bruna et al.
Further, Kipf & Welling (2016); Defferrard et al.
(2016) speed up graph convolution computation with localized ﬁlters based on Chebyshev expansion.
They target relatively small datasets and thus the training proceeds in full batch.
In order to scale GCNs to large graphs, layer sampling techniques (Hamilton et al., 2017; Chen et al., 2018b; Ying et al., 2018a; Chen et al., 2018a; Gao et al., 2018; Huang et al., 2018) have been proposed for efﬁcient minibatch training.
All of them follow the three meta steps: 1.
Construct a complete GCN on the full training graph.
Sample nodes or edges of each layer to form minibatches.
Propagate forward and backward among the sampled GCN.
Steps (2) and (3) proceed iteratively to update the weights via stochastic gradient descent.
The layer sampling algorithm of GraphSAGE (Hamilton et al., 2017) performs uniform node sampling on the previous layer neighbors.
It enforces a pre-deﬁned budget on the sample size, so as to bound the minibatch computation complexity.
Ying et al.
(2018a) enhances the layer sampler of Hamilton et al.
by introducing an importance score to each neighbor.
The algorithm presumably leads to less information loss due to weighted aggregation.
S-GCN (Chen et al., 2018a) further restricts neighborhood size by requiring only two support nodes in the previous layer.
The idea is to use the historical activations in the previous layer to avoid redundant re-evaluation.
FastGCN
(Chen et al., 2018b) performs sampling from another perspective.
Instead of tracking down the inter-layer connections, node sampling is performed independently for each layer.
It applies importance sampling to reduce variance, and results in constant sample size in all layers.
However, the minibatches potentially become too sparse to achieve high accuracy.
Huang et al.
(2018) improves
FastGCN by an additional sampling neural network.
It ensures high accuracy, since sampling is conditioned on the selected nodes in the next layer.
Signiﬁcant overhead may be incurred due to the expensive sampling algorithm and the extra sampler parameters to be learned.
Apart from layer sampling, recently, ClusterGCN (Chiang et al., 2019) proposes graph clustering based minibatch construction.
During pre-processing, the training graph is partitioned into densely connected clusters.
During training, clusters are randomly selected to form minibatches, and intracluster edge connections remain unchanged.
Similar to GraphSAINT , ClusterGCN does not sample the layers and thus “neighbor explosion” is avoided.
Unlike GraphSAINT , ClusterGCN is heuristic based, and does not account for bias towards speciﬁc structures in the pre-determined clusters.
Another line of research focuses on improving model capacity.
Applying attention on graphs, the architectures of Veliˇckovi´c et al.
; Zhang et al.
; Lu et al.
(2019) better capture neighbor features by dynamically adjusting edge weights.
Klicpera et al.
(2018) combines PageRank with
GCNs to enable efﬁcient information propagation from many hops away.
To develop deeper models, “skip-connection” is borrowed from CNNs (He et al., 2015; Huang et al., 2017) into the GCN context.
In particular, JK-net Xu et al.
(2018) demonstrates signiﬁcant accuracy improvement on GCNs with more than two layers.
Note, however, that JK-net (Xu et al., 2018) follows the same sampling strategy as GraphSAGE (Hamilton et al., 2017).
Thus, its training cost is high due to neighbor explosion.
In addition, high order graph convolutional layers (Zhou, 2017; Lee et al., 2018; Abu-El-Haija et al., 2019) also help propagate long-distance features.
With the numerous architectural variants developed, the question of how to train them efﬁciently via minibatches still remains to be answered.
P ROPOSED M ETHOD : GraphSAINT
Graph sampling based method is motivated by the challenges in scalability (in terms of model depth and graph size).
We analyze the bias (Section 3.2) and variance (Section 3.3) introduced by graph sampling, and thus, propose feasible sampling algorithms (Section 3.4).
We show the applicability of
GraphSAINT to other architectures, both conceptually (Section 4) and experimentally (Section 5.2).
In the following, we deﬁne the problem of interest and the corresponding notations.
A GCN learns representation of an un-directed, attributed graph G ( V , E ) , where each node v ∈ V has a length- f attribute x v .
Let A be the adjacency matrix and (cid:101)
A be the normalized one (i.e., (cid:101)
A = D − 1 A , and D is the diagonal degree matrix).
Let the dimension of layer- (cid:96) input activation be f ( (cid:96) ) .
The activation of node v is x ( (cid:96) ) v ∈ R f
( (cid:96) ) , and the weight matrix is W ( (cid:96) )
∈ R f ( (cid:96) ) × f
( (cid:96) +1) .
Note that x v = x (1) v .
Propagation rule of a layer is deﬁned as follows:
[Equation]
where (cid:101)
A v,u is a scalar, taking an element of (cid:101) A .
And σ ( · ) is the activation function (e.g., ReLU).
We use subscript “s” to denote parameterd of the sampled graph (e.g., G s , V s , E s ).
GCNs can be applied under inductive and transductive settings.
While GraphSAINT is applicable to both, in this paper, we focus on inductive learning.
It has been shown that inductive learning is especially challenging (Hamilton et al., 2017) — during training, neither attributes nor connections of the test nodes are present.
Thus, an inductive model has to generalize to completely unseen graphs.
3. M INIBATCH BY G RAPH S AMPLING
G s = SAMPLE ( G )
[Figure]
GCN on G s
5: { y v v
V s } ← Forward propagation of { x v | v ∈ V s } , normalized by α 6:
Backward propagation from λ -normalized loss L ( y v , y v ) .
Update weights.
7: end for
GraphSAINT follows the design philosophy of directly sampling the training graph G , rather than the corresponding GCN.
Our goals are to 1.
extract appropriately connected subgraphs so that little information is lost when propagating within the subgraphs, and 2.
combine information of many subgraphs together so that the training process overall learns good representation of the full graph.
Figure 1 and Algorithm 1 illustrate the training algorithm.
Before training starts, we perform light-weight pre-processing on G with the given sampler SAMPLE .
The pre-processing estimates the probability of a node v ∈ V and an edge e ∈ E being sampled by SAMPLE .
Such probability is later used to normalize the subgraph neighbor aggregation and the minibatch loss (Section 3.2).
Afterwards, training proceeds by iterative weight updates via SGD.
Each iteration starts with an independently sampled G s (where |V s | (cid:28) |V| ).
We then build a full GCN on G s to generate embedding and calculate loss for every v ∈ V s .
In Algorithm 1, node representation is learned by performing node classiﬁcation in the supervised setting, and each training node v comes with a ground truth label y v .
Intuitively, there are two requirements for SAMPLE : 1.
Nodes having high inﬂuence on each other should be sampled in the same subgraph.
Each edge should have non-negligible probability to be sampled.
For requirement (1), an ideal SAMPLE would consider the joint information from node connections as well as attributes.
However, the resulting algorithm may have high complexity as it would need to infer the relationships between features.
For simplicity, we deﬁne “inﬂuence” from the graph connectivity perspective and design topology based samplers.
Requirement (2) leads to better generalization since it enables the neural net to explore the full feature and label space.
3. N ORMALIZATION
A sampler that preserves connectivity characteristic of G will almost inevitably introduce bias into minibatch estimation.
Below we present normalization techniques to eliminate biases.
Analysis of the complete multi-layer GCN is difﬁcult due to non-linear activations.
Thus, we analyze the embedding of each layer independently.
This is similar to the treatment of layers independently by prior work (Chen et al., 2018b; Huang et al., 2018).
Consider a layer- ( (cid:96) + 1) node v and a layer- (cid:96) node u .
If v is sampled (i.e., v ∈ V s ), we can compute the aggregated feature of v as:
[Equation]
where ˜ x ( (cid:96) )
u = (cid:0) W ( (cid:96) ) (cid:1)
T x ( (cid:96) ) u , and 1 u | v ∈ { 0 , 1 } is the indicator function given v is in the subgraph (i.e., 1 u | v =
if v ∈ V s ∧ ( u, v ) (cid:54)∈ E s ; 1 u | v = 1 if ( u, v ) ∈
E s ; 1 u | v not deﬁned if v (cid:54)∈ V s ).
We refer to the constant α u,v as aggregator normalization .
Deﬁne p u,v = p v,u as the probability of an edge ( u, v ) ∈
E being sampled in a subgraph, and p v as the probability of a node v ∈ V being sampled.
Proposition 3.1.
ζ ( (cid:96) +1) v is an unbiased estimator of the aggregation of v in the full ( (cid:96) + 1) th GCN layer, if α u,v = p u,
[Equation]
i
[Equation]
[Equation]
Assuming that each layer independently learns an embedding, we use Proposition 3.1 to normalize feature propagation of each layer of the GCN built by GraphSAINT .
Further, let L v be the loss on v in the output layer.
The minibatch loss is calculated as L batch =
(cid:80) v ∈G s L v /λ v , where λ v is a constant that we term loss normalization .
We set λ v = |V| · p v so that:
[Equation]
Feature propagation within subgraphs thus requires normalization factors α and λ , which are computed based on the edge and node probability p u,v , p v .
In the case of random node or random edge samplers, p u,v and p v can be derived analytically.
For other samplers in general, closed form expression is hard to obtain.
Thus, we perform pre-processing for estimation.
Before training starts, we run the sampler repeatedly to obtain a set of N subgraphs G .
We setup a counter C v and C u,v for each v ∈ V and ( u, v ) ∈ E , to count the number of times the node or edge appears in the subgraphs of G .
Then we set α u,v = C u,v C v =
C v,u C v and λ v = C
v N .
The subgraphs G s ∈ G can all be reused as minibatches during training.
Thus, the overhead of pre-processing is small (see Appendix D.2).
3. V ARIANCE
We derive samplers for variance reduction.
Let e be the edge connecting u , v , and b ( (cid:96) )
e = (cid:101)
A v,u ˜ x
( (cid:96) − 1) u + (cid:101)
A u,v ˜ x ( (cid:96) − 1) v .
It is desirable that variance of all estimators ζ ( (cid:96) ) v is small.
With this objective, we deﬁne:
[Equation]
where 1 e = 1 if e ∈ E s ; 1 e =
if e (cid:54)∈ E s .
And 1 v = 1 if v ∈ V s ; 1 v =
if v (cid:54)∈ V s .
The factor p u in the ﬁrst equality is present so that ζ is an unbiased estimator of the sum of all node aggregations at all layers: E ( ζ )
=
(cid:80) (cid:96) (cid:80) v ∈V E (cid:16) ζ ( (cid:96) ) v (cid:17) .
Note that 1 ( (cid:96) )
e = 1 e , ∀ (cid:96) , since once an edge is present in the sampled graph, it is present in all layers of our GCN.
We deﬁne the optimal edge sampler to minimize variance for every dimension of ζ .
We restrict ourselves to independent edge sampling.
For each e ∈ E , we make independent decision on whether it should be in G s or not.
The probability of including e is p e .
We further constrain (cid:80) p
e
= m , so that the expected number of sampled edges equals to m .
The budget m is a given sampling parameter.
Theorem 3.2.
Under independent edge sampling with budget m , the optimal edge probabilities to minimize the sum of variance of each dimension of ζ is given by: p e =
[Table]
To prove Theorem 3.2, we make use of the independence among graph edges, and the dependence among layer edges to obtain the covariance of 1 ( (cid:96) )
e .
Then using the fact that sum of p e is a constant, we use the Cauchy-Schwarz inequality to derive the optimal p e .
Details are in Appendix A.
Note that calculating b ( (cid:96) )
e requires computing ˜ x
( (cid:96) − 1) v , which increases the complexity of sampling.
As a reasonable simpliﬁcation, we ignore ˜ x
( (cid:96) ) v to make the edge probability dependent on the graph topology only.
Therefore, we choose p e ∝ (cid:101)
A v,u + (cid:101)
A u,v = 1 deg ( u ) + 1 deg ( v ) .
The derived optimal edge sampler agrees with the intuition in Section 3.1.
If two nodes u , v are connected and they have few neighbors, then u and v are likely to be inﬂuential to each other.
In this case, the edge probability p u,v = p v,u should be high.
The above analysis on edge samplers also inspires us to design other samplers, which are presented in Section 3.4.
Remark We can also apply the above edge sampler to perform layer sampling.
Under the independent layer sampling assumption of Chen et al.
(2018b), one would sample a connection (cid:0) u
( (cid:96) ) , v ( (cid:96) +1) (cid:1) with probability p ( (cid:96) )
u,v ∝ 1 deg ( u ) + 1 deg ( v ) .
For simplicity, assume a uniform degree graph (of degree d ).
Then p ( (cid:96) ) e = p .
For an already sampled u ( (cid:96) ) to connect to layer (cid:96) + 1 , at least one of its edges has to be selected by the layer (cid:96) + 1 sampler.
Clearly, the probability of an input layer node to “survive” the
L number of independent sampling process is (cid:16) 1 − (1 − p ) d (cid:17) L − 1 .
Such layer sampler potentially returns an overly sparse minibatch for L > 1 .
On the other hand, connectivity within a minibatch of
GraphSAINT never drops with GCN depth.
If an edge is present in layer (cid:96) , it is present in all layers.
3. S AMPLERS
Based on the above variance analysis, we present several light-weight and efﬁcient samplers that
GraphSAINT has integrated.
Detailed sampling algorithms are listed in Appendix B.
Random node sampler We sample |V s | nodes from V randomly, according to a node probability distribution P ( u ) ∝ (cid:13)(cid:13)(cid:13) (cid:101)
A : ,u (cid:13)(cid:13)(cid:13) 2 .
This sampler is inspired by the layer sampler of Chen et al.
(2018b).
Random edge sampler We perform edge sampling as described in Section 3.3.
Random walk based samplers Another way to analyze graph sampling based multi-layer GCN is to ignore activations.
In such case, L layers can be represented as a single layer with edge weights given by B = (cid:101)
A L .
Following a similar approach as Section 3.3, if it were possible to pick pairs of nodes (whether or not they are directly connected in the original (cid:101) A ) independently, then we would set p u,v ∝ B u,v + B v,u , where B u,v can be interpreted as the probability of a random walk to start at u and end at v in L hops (and B v,u vice-versa).
Even though it is not possible to sample a subgraph where such pairs of nodes are independently selected, we still consider a random walk sampler with walk length L as a good candidate for L -layer GCNs.
There are numerous random walk based samplers proposed in the literature (Ribeiro & Towsley, 2010; Leskovec & Faloutsos, 2006; Hu & Lau, 2013; Li et al., 2015).
In the experiments, we implement a regular random walk sampler (with r root nodes selected uniformly at random and each walker goes h hops), and also a multi-dimensional random walk sampler deﬁned in Ribeiro & Towsley (2010).
For all the above samplers, we return the subgraph induced from the sampled nodes.
The induction step adds more connections into the subgraph, and empirically helps improve convergence.
D ISCUSSION Extensions
GraphSAINT admits two orthogonal extensions.
First, GraphSAINT can seamlessly integrate other graph samplers.
Second, the idea of training by graph sampling is applicable to many
GCN architecture variants: 1.
Jumping knowledge (Xu et al., 2018): since our GCNs constructed during training are complete, applying skip connections to GraphSAINT is straightforward.
On the other hand, for some layer sampling methods (Chen et al., 2018b; Huang et al., 2018), extra modiﬁcation to their samplers is required, since the jumping knowledge architecture requires layer- (cid:96) samples to be a subset of layer- ( (cid:96) − 1) samples ∗ .
Attention (Veliˇckovi´c et al., 2017; Fey, 2019;
Zhang et al., 2018): while explicit variance reduction is hard due to the dynamically updated attention values, it is reasonable to apply attention within the subgraphs which are considered as representatives of the full graph.
Our loss and aggregator normalizations are also applicable † .
Others : To support high order layers (Zhou, 2017; Lee et al., 2018; Abu-El-Haija et al., 2019) or even more complicated networks for the task of graph classiﬁcation (Ying et al., 2018b), we replace the full adjacency matrix
A with the (normalized) one for the subgraph A s to perform layer propagation.
Comparison
GraphSAINT enjoys: 1. high scalability and efﬁciency, 2.
high accuracy, and 3.
low training complexity.
Point (1) is due to the signiﬁcantly reduced neighborhood size compared with
Hamilton et al.
; Ying et al.
(2018a)
; Chen et al.
(2018a).
Point (2) is due to the better interlayer connectivity compared with Chen et al.
(2018b), and unbiased minibatch estimator compared with Chiang et al.
Point (3) is due to the simple and trivially parallelizable pre-processing compared with the sampling of Huang et al.
(2018) and clustering of Chiang et al.
E XPERIMENTS
Setup Experiments are under the inductive, supervised learning setting.
We evaluate GraphSAINT on the following tasks: 1.
classifying protein functions based on the interactions of human tissue proteins (PPI), 2.
categorizing types of images based on the descriptions and common properties of online images (Flickr), 3.
predicting communities of online posts based on user comments (Reddit), 4.
categorizing types of businesses based on customer reviewers and friendship (Yelp), and 5.
classifying product categories based on buyer reviewers and interactions (Amazon).
For PPI, we use the small version for the two layer convergence comparison (Table 2 and Figure 2), since
Hamilton et al.
(2017) and Chen et al.
(2018a) report accuracy for this version in their original papers.
We use the large version for additional comparison with Chiang et al.
(2019) to be consistent with its reported accuracy.
All datasets follow “ﬁxed-partition” splits.
Appendix C.2 includes further details.
[Table]
We open source GraphSAINT anonymously ‡ .
We compare with six baselines: 1. vanilla GCN (Kipf & Welling, 2016), 2.
GraphSAGE (Hamilton et al., 2017),
3. FastGCN
(Chen et al., 2018b), 4.
S-GCN (Chen et al., 2018a), 5.
AS-GCN (Huang et al., 2018), and 6.
ClusterGCN (Chiang et al., 2019).
All baselines are executed with their ofﬁcially released code (see Appendix C.3 for downloadable URLs
When applying GraphSAINT to GAT (Veliˇckovi´c et al., 2017), we remove the softmax step which normalizes attention values within the same neighborhood, as suggested by Huang et al.
See Appendix C.3.
and commit numbers).
Baselines and GraphSAINT are all implemented in Tensorﬂow with Python3.
We run experiments on a NVIDIA Tesla P100 GPU (see Appendix C.1 for hardware speciﬁcation).
5. C OMPARISON WITH S TATE - OF - THE -A RT
Table 2 and Figure 2 show the accuracy and convergence comparison of various methods.
All results correspond to two-layer GCN models (for GraphSAGE, we use its mean aggregator).
For a given dataset, we keep hidden dimension the same across all methods.
We describe the detailed architecture and hyperparameter search procedure in Appendix C.3.
The mean and conﬁdence interval of the accuracy values in Table 2 are measured by three runs under the same hyperparameters.
The training time of Figure 2 excludes the time for data loading, pre-processing, validation set evaluation and model saving.
Our pre-processing incurs little overhead in training time.
See Appendix D.2 for cost of graph sampling.
For GraphSAINT , we implement the graph samplers described in Section 3.4.
In
Table 2, “Node” stands for random node sampler; “Edge” stands for random edge sampler; “RW” stands for random walk sampler; “MRW” stands for multi-dimensional random walk sampler.
[Table]
[Figure]
Clearly, with appropriate graph samplers, GraphSAINT achieves signiﬁcantly higher accuracy on all datasets.
For GraphSAINT -Node, we use the same node probability as FastGCN.
Thus, the accuracy improvement is mainly due to the switching from layer sampling to graph sampling (see “Remark” in
Section 3.3).
Compared with AS-GCN, GraphSAINT is signiﬁcantly faster.
The sampler of AS-GCN is expensive to execute, making its overall training time even longer than vanilla GCN.
We provide detailed computation complexity analysis on the sampler in Appendix D.2.
For S-GCN on Reddit, it achieves similar accuracy as GraphSAINT , at the cost of over 9 × longer training time.
The released code of FastGCN only supports CPU execution, so its convergence curve is dashed.
[Table]
t
[Table]
1 - m i c r o
[Table]
Amazon
[Table]
Table 3 presents additional comparison with ClusterGCN.
We use L × f to specify the architecture, where L and f denote GCN depth and hidden dimension, respectively.
The four architectures are the ones used in the original paper (Chiang et al., 2019).
Again, GraphSAINT achieves signiﬁcant accuracy improvement.
To train models with L > 2 often requires additional architectural tweaks.
ClusterGCN uses its diagonal enhancement technique for the 5-layer PPI and 4-layer Reddit models.
GraphSAINT uses jumping knowledge connection (Xu et al., 2018) for 4-layer Reddit.
Evaluation on graph samplers From Table 2, random edge and random walk based samplers achieve higher accuracy than the random node sampler.
Figure 3 presents sensitivity analysis on parameters of “RW”.
We use the same hyperparameters (except the sampling parameters) and network architecture as those of the “RW” entries in Table 2.
We ﬁx the length of each walker to 2 (i.e., GCN depth), and vary the number of roots r from 250 to 2250 .
For PPI, increasing r from 250 tosigniﬁcantly improves accuracy.
Overall, for all datasets, accuracy stabilizes beyond r = 750 .
5. GraphSAINT ON A RCHITECTURE V ARIANTS AND D EEP M ODELS
In Figure 4, we train a 2-layer and a 4-layer model of GAT (Veliˇckovi´c et al., 2017) and JK-net (Xu et al., 2018), by using minibatches of GraphSAGE and GraphSAINT respectively.
On the two 4-layer architectures, GraphSAINT achieves two orders of magnitude speedup than GraphSAGE, indicating much better scalability on deep models.
From accuracy perspective, 4-layer GAT-SAGE and JKSAGE do not outperform the corresponding 2-layer versions, potentially due to the smoothening effect caused by the massive neighborhood size.
On the other hand, with minibatches returned by our edge sampler, increasing model depth of JK- SAINT leads to noticeable accuracy improvement (from 0.966 of 2-layer to 0.970 of 4-layer).
Appendix D.1 contains additional scalability results.
C ONCLUSION
We have presented GraphSAINT , a graph sampling based training method for deep GCNs on large graphs.
We have analyzed bias and variance of the minibatches deﬁned on subgraphs, and proposed normalization techniques and sampling algorithms to improve training quality.
We have conducted extensive experiments to demonstrate the advantage of GraphSAINT in accuracy and training time.
An interesting future direction is to develop distributed training algorithms using graph sampling based minibatches.
The graph sampling naturally partitions data, and training on the self-contained subgraphs can signiﬁcantly reduce the communication cost.
Especially, since the graph data may reside on geographically dispersed computation nodes, the property that graph sampling is decoupled from the training helps parallelization, and ﬁts very well the distributed computing environments.
A P ROOFS
Proof of Proposition 3.1.
Under the condition that v is sampled in a subgraph:
[Equation]
where the second equality is due to linearity of expectation, and the third equality (conditional edge probability) is due to the initial condition that v is sampled in a subgraph.
It directly follows that, when α u,v = p u
[Equation]
[Equation]
Proof of Theorem 3.2.
Below, we use Cov ( · ) to denote covariance and Var ( · ) to denote variance.
For independent edge sampling as deﬁned in Section 3.3, Cov (cid:16) 1 ( (cid:96)
[Equation]
= 0 , ∀ e
[Equation]
=
e 2 .
And for a full GCN on the subgraph, Cov (cid:16) 1 ( (cid:96) 1 ) e ,
[Equation]
(cid:17) =
p e − p 2 e .
To start the proof, we ﬁrst assume that the b ( (cid:96) ) e is one dimensional (i.e., a scalar) and denote it by b ( (cid:96) )
e .
Now,
[Equation]
Let a given constant m = (cid:80) e p e be the expected number of sampled edges.
By Cauchy-Schwarz inequality: (cid:80) e (
[Equation]
2 p e
[Equation]
√ p
[Equation]
2 ≥ (cid:16)(cid:80) e,(cid:96)
b ( (cid:96) )
e (cid:17) 2 .
The equality is achieved when (cid:12) (cid:12) (cid:12) (cid:80) (cid:96) b
( (cid:96) )
e √ p e (cid:12) (cid:12) (cid:12) ∝ √ p e .
i.e., variance is minimized when p e ∝ (cid:12) (cid:12) (cid:12)(cid:80) (cid:96) b ( (cid:96) )
e (cid:12) (cid:12) (cid:12) .
It directly follows that:
[Equation]
For the multi-dimensional case of b ( (cid:96) )
e , following similar steps as above, it is easy to show that the optimal edge probability to minimize (cid:80) i Var ( ζ i ) (where i is the index for ζ ’s dimensions) is:
[Equation]
m
[Equation]
[Table]
[Equation]
[Table]
B S AMPLING A LGORITHM
Input: Training graph G ( V , E ) ; Sampling parameters: node budget n ; edge budget m ; number of roots r ; random walk length h
Output: Sampled graph G s
( V s , E s ) 1: function N ODE ( G , n ) (cid:46)
Node sampler 2:
P ( v ) :
= (cid:13) (cid:13)(cid:13) (cid:101)
A : ,v (cid:13) (cid:13)(cid:13) 2 / (cid:80) v (cid:48) ∈V (cid:13)(cid:13)(cid:13) (cid:101)
A : ,v (cid:48) (cid:13)(cid:13)(cid:13)3:
V s ← n nodes randomly sampled (with replacement) from V according to P 4:
G s ← Node induced subgraph of G from V s 5: end function 6: function E DGE ( G , m ) (cid:46)
Edge sampler (approximate version) 7:
P (( u, v )) :
= (cid:16) 1 deg ( u ) + 1 deg ( v ) (cid:17) / (cid:80) ( u (cid:48) ,v (cid:48) )
∈E (cid:16) 1 deg ( u (cid:48) )
+ 1 deg ( v (cid:48) ) (cid:17) 8:
E s ← m edges randomly sampled (with replacement) from E according to P 9:
V s ← Set of nodes that are end-points of edges in E s 10:
G s ← Node induced subgraph of G from V s 11: end function 12: function RW( G , r , h ) (cid:46)
Random walk sampler 13:
V root ← r root nodes sampled uniformly at random (with replacement) from V 14:
V s ← V root 15: for v ∈ V root do 16: u ← v 17: for d = 1 to h do 18: u ← Node sampled uniformly at random from u ’s neighbor 19:
V s ← V s ∪ { u } 20: end for 21: end for 22:
G s ← Node induced subgraph of G from V s 23: end function 24: function MRW( G , n , r ) (cid:46) Multi-dimensional random walk sampler 25:
V FS ← r root nodes sampled uniformly at random (with replacement) from V 26:
V s ← V FS 27: for i =
r + 1 to n do 28:
Select u ∈ V FS with probability deg ( u ) / (cid:80) v ∈V FS deg ( v ) 29: u (cid:48) ← Node randomly sampled from u ’s neighbor 30:
V FS ← ( V FS \ { u } ) ∪ { u (cid:48) } 31:
V s ← V s ∪ { u } 32: end for 33:
G s ← Node induced subgraph of G from V s 34: end function
Algorithm 2 lists the four graph samplers we have integrated into GraphSAINT .
The naming of the samplers follows that of Table 2.
Note that the sampling parameters n and m specify a budget rather than the actual number of nodes and edges in the subgraph G s .
Since certain nodes or edges in the training graph G may be repeatedly sampled under a single invocation of the sampler, we often have |V s | < n for node and MRW samplers, |V s | < 2 m for edge sampler, and |V s | < r · h for RW sampler.
[Figure]
PPI
Flickr
Reddit
[Figure]
Also note that the edge sampler presented in Algorithm 2 is an approximate version of the independent edge sampler deﬁned in Section 3.4.
Complexity (excluding the subgraph induction step) of the original version in Section 3.4 is O ( |E| ) , while complexity of the approximate one is O ( m ) .
When m (cid:28) |E| , the approximate version leads to identical accuracy as the original one, for a given m .
C D ETAILED E XPERIMENTAL S ETUP C. H ARDWARE S PECIFICATION AND E NVIRONMENT
We run our experiments on a single machine with Dual Intel Xeon CPUs (E5-2698 v4 @ 2.2Ghz),
one NVIDIA Tesla P100 GPU (16GB of HBM2 memory) and 512GB DDR4 memory.
The code is written in Python 3.6.8 (where the sampling part is written with Cython 0.29.2).
We use Tensorﬂow 1.12.0 on CUDA 9.2 with CUDNN 7.2.1 to train the model on GPU.
Since the subgraphs are sampled independently, we run the sampler in parallel on 40 CPU cores.
C. A DDITIONAL D ATASET D ETAILS
Here we present the detailed procedures to prepare the Flickr, Yelp and Amazon datasets.
The Flickr dataset originates from NUS-wide § .
The SNAP website ¶ collected Flickr data from four different sources including NUS-wide, and generated an un-directed graph.
One node in the graph represents one image uploaded to Flickr.
If two images share some common properties (e.g., same geographic location, same gallery, comments by the same user, etc.)
, there is an edge between the nodes of these two images.
We use as the node features the 500-dimensional bag-of-word representation of the images provided by NUS-wide.
For labels, we scan over the 81 tags of each image and manually merged them to 7 classes.
Each image belongs to one of the 7 classes.
The Yelp dataset is prepared from the raw json data of businesses, users and reviews provided in the open challenge website (cid:107) .
For nodes and edges, we scan the friend list of each user in the raw json ﬁle of users.
If two users are friends, we create an edge between them.
We then ﬁlter out all the reviews by each user and separate the reviews into words.
Each review word is converted to a 300-dimensional vector using the Word2Vec model pre-trained on GoogleNews ∗∗ .
The word vectors of each node are added and normalized to serve as the node feature (i.e., x v ).
As for the node labels, we scan the raw json ﬁle of businesses, and use the categories of the businesses reviewed by a user v as the multi-class label of v .
For the Amazon dataset, a node is a product on the Amazon website and an edge ( u, v ) is created if products u and v are bought by the same customer.
Each product contains text reviews (converted to 4-gram) from the buyer.
We use SVD to reduce the dimensionality of the 4-gram representation to 200, and use the obtained vectors as the node feature.
The labels represent the product categories (e.g., books, movies, shoes).
Figure 5 shows the degree distribution of the ﬁve graphs.
A point ( k, p ) in the plot means the probability of a node having degree at least k is p .
C. A DDITIONAL D ETAILS IN E XPERIMENTAL C ONFIGURATION
[Table]
Table 4 summarizes the URLs to download the baseline codes.
The optimizer for GraphSAINT and all baselines is Adam (Kingma & Ba, 2014).
For all baselines and datasets, we perform grid search on the hyperparameter space deﬁned by:
• Hidden dimension: { 128 , 256 , 512 } • Dropout: { 0 . 0 , 0 . 1 , 0 . 2 , 0 .
3 } • Learning rate: { 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 }
The hidden dimensions used for Table 2, Figure 2, Figure 3 and Figure 4 are: 512 for PPI, 256 for
Flickr, 128 for Reddit, 512 for Yelp and 512 for Amazon.
All methods terminate after a ﬁxed number of epochs based on convergence.
We save the model producing the highest validation set F1-micro score, and reload it to evaluate the test set accuracy.
For vanilla GCN and AS-GCN, we set the batch size to their default value 512.
For GraphSAGE, we use the mean aggregator with the default batch size 512.
For S-GCN, we set the ﬂag -cv -cvd (which stand for “control variate” and “control variate dropout”) with pre-computation of the ﬁrst layer aggregation.
According to the paper (Chen et al., 2018a), such pre-computation signiﬁcantly reduces training time without affecting accuracy.
For S-GCN, we use the default batch size 1000, and for FastGCN, we use the default value 400.
For ClusterGCN, its batch size is determined by two parameters: the cluster size and the number of clusters per batch.
We sweep the cluster size from 500 to 10000 with step 500 , and the number of clusters per batch from { 1 , 10 , 20 , 40 } to determine the optimal conﬁguration for each dataset / architecture.
Considering that for ClusterGCN, the cluster structure may be sensitive to the cluster size, and for FastGCN, the minibatch connectivity may increase with the sample size, we present additional experimental results to reveal the relation between accuracy and batch size in Appendix D.3.
Conﬁguration of GraphSAINT to reproduce Table 2 results is shown in Table 5.
Conﬁguration of
GraphSAINT to reproduce Table 3 results is shown in Table 6.
Below we describe the conﬁguration for Figure 4.
The major difference between a normal GCN and a JK-net (Xu et al., 2018) is that JK-net has an additional ﬁnal layer that aggregates all the output hidden features of graph convolutional layers 1 to
L .
Mathematically, the additional aggregation layer outputs the ﬁnal embedding x JK as follows:
[Equation]
[Table] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
GCN depth
N o r m a li
ze d t r a i n i ng ti m e
[Table]
PP
I
F li c k r
R e dd it
Y e l p
A m az on 0 . 1 .
F r ac ti on o f t r a i n i ng
[Table]
where based on Xu et al.
(2018), (cid:76) is the vector aggregation operator: max-pooling, concatenation or LSTM (Hochreiter & Schmidhuber, 1997) based aggregation.
The graph attention of GAT (Veli ˇ ckovi ´ c et al., 2017) calculates the edge weights for neighbor aggregation by an additional neural network.
With multi-head ( K ) attention, the layer- ( (cid:96) − 1) features propagate to layer- ( (cid:96) ) as follows:
[Equation]
where (cid:107) is the vector concatenation operation, and the coefﬁcient α is calculated with the attention weights a k by:
[Equation]
Note that the α calculation is slightly different from the original equation in Veliˇckovi´c et al.
Namely, GAT- SAINT does not normalize α by softmax across all neighbors of v .
We make such modiﬁcation since under the minibatch setting, node v does not see all its neighbors in the training graph.
The removal of softmax is also seen in the attention design of Huang et al.
Note that during the minibatch training, GAT- SAINT further applies another edge coefﬁcient on top of attention for aggregator normalization.
Table 7 shows the conﬁguration of the GAT- SAINT and JK- SAINT curves in Figure 4.
D A DDITIONAL E XPERIMENTS D. T RAINING E FFICIENCY ON D EEP M ODELS
We evaluate the training efﬁciency for deeper GCNs.
We only compare with S-GCN, since implementations for other layer sampling based methods have not yet supported arbitrary model depth.
The batch size and hidden dimension are the same as Table 2.
On the two large graphs (Reddit and Yelp), we increase the number of layers and measure the average time per minibatch execution.
In Figure 6, training cost of GraphSAINT is approximately linear with GCN depth.
Training cost of S-GCN grows dramatically when increasing the depth.
This reﬂects the “neighbor explosion” phenomenon (even though the expansion factor of S-GCN is just 2).
On Yelp, S-GCN gives “out-of-memory” error for models beyond 5 layers.
D. C OST OF S AMPLING AND P RE -P ROCESSING
Cost of graph samplers of GraphSAINT Graph sampling introduces little training overhead.
Let t s be the average time to sample one subgraph on a multi-core machine.
Let t t be the average time to perform the forward and backward propagation on one minibatch on GPU.
Figure 7 shows the ratio t s /t t for various datasets.
The parameters of the samplers are the same as Table 2.
For
Node, Edge and RW samplers, we observe that time to sample one subgraph is in most cases less than 25% of the training time.
The MRW sampler is more expensive to execute.
Regarding the complete pre-processing procedure, we repeatedly run the sampler for N = 50 · |V| / |V s | times before training, to estimate the node and edge probability as discussed in Section 3.2 (where |V s | is the average subgraph size).
These sampled subgraphs are reused as training minibatches.
Thus, if training runs for more than N iterations, the pre-processing is nearly zero-cost.
Under the setting of Table 2, pre-processing on PPI and Yelp and Amazon does not incur any overhead in training time.
Pre-processing on Flickr and Reddit (with RW sampler) takes less than 40% and 15% of their corresponding total training time.
Cost of layers sampler of AS-GCN AS-GCN uses an additional neural network to estimate the conditional sampling probability for the previous layer.
For a node v already sampled in layer (cid:96) , features of layer- ( (cid:96) − 1) corresponding to all v ’s neighbors need to be fed to the sampling neural network to obtain the node probability.
For sake of analysis, assume the sampling network is a single layer MLP, whose weight W MLP has the same shape as the GCN weights W ( (cid:96) ) .
Then we can show, for a L -layer GCN on a degree- d graph, per epoch training complexity of AS-GCN is approximately γ = ( d · L ) / (cid:80) L − 1 (cid:96) =
d (cid:96) times that of vanilla GCN.
For L = 2 , we have γ ≈ 2 .
This explains the observation that AS-GCN is slower than vanilla GCN in Figure 2.
Additional, Table 8 shows the training time breakdown for AS-GCN.
Clearly, its sampler is much more expensive than the graph sampler of GraphSAINT .
[Table]
Cost of clustering of ClusterGCN ClusterGCN uses the highly optimized METIS software †† to perform clustering.
Table 9 summarizes the time to obtain the clusters for the ﬁve graphs.
On the large and dense Amazon graph, the cost of clustering increase dramatically.
The pre-processing time of ClusterGCN on Amazon is more than 4 × of the total training time.
On the other hand, the sampling cost of GraphSAINT does not increase signiﬁcantly for large graphs (see Figure 7). [Table]
Taking into account the pre-processing time
[Table]
sampling time and training time altogether
[Table]
we summarize the total convergence time of GraphSAINT and ClusterGCN in Table 10 (corresponding to
Table 2 conﬁguration).
On graphs that are large and dense (e
[Table]
g., Amazon), GraphSAINT achieves signiﬁcantly faster convergence.
Note that both the sampling of GraphSAINT and clustering of
ClusterGCN can be performed ofﬂine.
[Table]
D. E FFECT OF B ATCH S IZE
[Table]
S
[Table]
— ¶ — ¶ — ‡ 1000 ∗ 0.
AS-
[Table]
— ‡ — ‡ 512 ∗ 0.
ClusterGCN 0.
[Table]
[Table]
[Table] [Table] [Table] Table 11 shows the change of test set accuracy with batch sizes.
For each row of Table 11, we ﬁx the batch size, tune the other hyperparameters according to Appendix C.3, and report the highest test set accuracy achieved.
For GraphSAGE, S-GCN and AS-GCN, their default batch sizes (512,1000 and 512, respectively) lead to the highest accuracy on all datasets.
For FastGCN, increasing the default batch size (from 400 to 4000) leads to noticeable accuracy improvement.
For ClusterGCN, different datasets correspond to different optimal batch sizes.
Note that the accuracy in Section 5.1 is already tuned by identifying the optimal batch size on a per graph basis.
For FastGCN, intuitively, increasing batch size may help with accuracy improvement since the minibatches may become better connected.
Such intuition is veriﬁed by the rows of 400 and 2000.
However, increasing the batch size from 2000 to 4000 does not further improve accuracy signiﬁcantly.
For ClusterGCN, the optimal batch size depends on the cluster structure of the training graph. For
PPI, small batches are better, while for Amazon, batch size does not have signiﬁcant impact on accuracy. For GraphSAGE, overly large batches may have negative impact on accuracy due to neighbor explosion. Approximately, GraphSAGE expand 10 × more neighbors per layer. For a 2-layer GCN, a size 2 × 10 3 minibatch would then require the support of 2 × 10 5 nodes from the input layer.
Note that the full training graph size of Reddit is just around 1 . 5 × 10 5 . Thus, no matter which nodes are sampled in the output layer, GraphSAGE would almost always propagate features within the full training graph for initial layers. We suspect this would lead to difﬁculties in learning.
For S-GCN, with batch size of 500, it fails to learn properly on Reddit and Yelp.
The accuracy ﬂuctuates in a region of very low value, even after appropriate hyperparameter tuning.
For AS-GCN, its accuracy is not sensitive to the batch size, since AS-GCN addresses neighbor explosion and also ensures good inter-layer connectivity within the minibatch.