F AST GCN: F AST L EARNING WITH G RAPH C ONVOLUTIONAL N ETWORKS VIA I MPORTANCE S AMPLING
Anonymous authors
Paper under double-blind review
A BSTRACT
The graph convolutional networks (GCN) recently proposed by Kipf and Welling are an effective graph model for semi-supervised learning.
Such a model, however, is transductive in nature because parameters are learned through convolutions with both training and test data.
Moreover, the recursive neighborhood expansion across layers poses time and memory challenges for training with large, dense graphs.
To relax the requirement of simultaneous availability of test data, we interpret graph convolutions as integral transforms of embedding functions under probability measures.
Such an interpretation allows for the use of Monte Carlo approaches to consistently estimate the integrals, which in turn leads to a batched training scheme as we propose in this work—FastGCN.
Enhanced with importance sampling, FastGCN not only is efﬁcient for training but also generalizes well for inference.
We show a comprehensive set of experiments to demonstrate its effectiveness compared with GCN and related models.
In particular, training is orders of magnitude more efﬁcient while predictions remain comparably accurate.
I NTRODUCTION
Graphs are universal representations of pairwise relationship.
Many real world data come naturally in the form of graphs; e.g., social networks, gene expression networks, and knowledge graphs.
To improve the performance of graph-based learning tasks, such as node classiﬁcation and link prediction, recently much effort is made to extend well-established network architectures, including recurrent neural networks (RNN) and convolutional neural networks (CNN), to graph data; see, e.g.,
Bruna et al.
; Duvenaud et al.
; Li et al.
; Jain et al.
; Henaff et al.
Niepert et al.
; Kipf & Welling (2016a;b).
Whereas learning feature representations for graphs is an important subject among this effort, here, we focus on the feature representations for graph vertices.
In this vein, the closest work that applies a convolution architecture is the graph convolutional network (GCN) (Kipf & Welling, 2016a;b).
Borrowing the concept of a convolution ﬁlter for image pixels or a linear array of signals, GCN uses the connectivity structure of the graph as the ﬁlter to perform neighborhood mixing.
The architecture may be elegantly summarized by the following expression:
[Equation]
where ˆ A is some normalization of the graph adjacency matrix, H ( l ) contains the embedding (rowwise) of the graph vertices in the l th layer, W ( l ) is a parameter matrix, and σ is nonlinearity.
As with many graph algorithms, the adjacency matrix encodes the pairwise relationship for both training and test data.
The learning of the model as well as the embedding is performed for both data simultaneously, at least as the authors proposed.
For many applications, however, test data may not be readily available, because the graph may be constantly expanding with new vertices (e.g. new members of a social network, new products to a recommender system, and new drugs for functionality tests).
Such scenarios require an inductive scheme that learns a model from only a training set of vertices and that generalizes well to any augmentation of the graph.
A more severe challenge for GCN is that the recursive expansion of neighborhoods across layers incurs expensive computations in batched training.
Particularly for dense graphs and powerlaw graphs, the expansion of the neighborhood for a single vertex quickly ﬁlls up a large portion of the graph.
Then, a usual mini-batch training will involve a large amount of data for every batch, even with a small batch size.
Hence, scalability is a pressing issue to resolve for GCN to be applicable to large, dense graphs.
To address both challenges, we propose to view graph convolutions from a different angle and interpret them as integral transforms of embedding functions under probability measures.
Such a view provides a principled mechanism for inductive learning, starting from the formulation of the loss to the stochastic version of the gradient.
Speciﬁcally, we interpret that graph vertices are iid samples of some probability distribution and write the loss and each convolution layer as integrals with respect to vertex embedding functions.
Then, the integrals are evaluated through Monte Carlo approximation that deﬁnes the sample loss and the sample gradient.
One may further alter the sampling distribution (as in importance sampling) to reduce the approximation variance.
The proposed approach, coined FastGCN, not only rids the reliance on the test data but also yields a controllable cost for per-batch computation.
At the time of writing, we notice a newly published work GraphSAGE (Hamilton et al., 2017) that proposes also the use of sampling to reduce the computational footprint of GCN.
Our sampling scheme is more economic, resulting in a substantial saving in the gradient computation, as will be analyzed in more detail in Section 3.3.
Experimental results in Section 4 indicate that the per-batch computation of FastGCN is more than an order of magnitude faster than that of GraphSAGE, while classiﬁcation accuracies are highly comparable.
R ELATED W ORK
Over the past few years, several graph-based convolution network models emerged for addressing applications of graph-structured data, such as the representation of molecules (Duvenaud et al., 2015).
An important stream of work is built on spectral graph theory (Bruna et al., 2013; Henaff et al., 2015; Defferrard et al., 2016).
They deﬁne parameterized ﬁlters in the spectral domain, inspired by graph Fourier transform.
These approaches learn a feature representation for the whole graph and may be used for graph classiﬁcation.
Another line of work learns embeddings for graph vertices, for which Goyal & Ferrara (2017) is a recent survey that covers comprehensively several categories of methods.
A major category consists of factorization based algorithms that yield the embedding through matrix factorizations; see, e.g.,
Roweis & Saul (2000); Belkin & Niyogi (2001);
Ahmed et al.
; Cao et al.
; Ou et al.
These methods learn the representations of training and test data jointly.
Another category is random walk based methods (Perozzi et al., 2014; Grover & Leskovec, 2016) that compute node representations through exploration of neighborhoods.
LINE (Tang et al., 2015) is also such a technique that is motivated by the preservation of the ﬁrst and second-order proximities.
Meanwhile, there appear a few deep neural network architectures, which better capture the nonlinearity within graphs, such as SDNE (Wang et al., 2016).
As motivated earlier, GCN (Kipf & Welling, 2016a) is the model on which our work is based.
The most relevant work to our approach is GraphSAGE (Hamilton et al., 2017), which learns node representations through aggregation of neighborhood information.
One of the proposed aggregators employs the GCN architecture.
The authors also acknowledge the memory bottleneck of GCN and hence propose an ad hoc sampling scheme to restrict the neighborhood size.
Our sampling approach is based on a different and more principled formulation.
The major distinction is that we sample vertices rather than neighbors.
The resulting computational savings are analyzed in Section 3.3.
T RAINING AND I NFERENCE THROUGH S AMPLING
One striking difference between GCN and many standard neural network architectures is the lack of independence in the sample loss.
Training algorithms such as SGD and its batch generalization are designed based on the additive nature of the loss function with respect to independent data samples.
For graphs, on the other hand, each vertex is convolved with all its neighbors and hence deﬁning a sample gradient that is efﬁcient to compute is beyond straightforward.
Concretely, consider the standard SGD scenario where the loss is the expectation of some function g with respect to a data distribution D :
L
[Equation]
E
[Equation]
D [ g ( W
[Equation]
x )] .
Here, W denotes the model parameter to be optimized.
Of course, the data distribution is generally unknown and one instead minimizes the empirical loss through accessing n iid samples x 1 , . . .
, x n :
[Equation]
In each step of SGD, the gradient is approximated by ∇ g ( W ; x i ) , an (assumed) unbiased sample of ∇ L .
One may interpret that each gradient step makes progress toward the sample loss g
( W ; x i ) .
The sample loss and the sample gradient involve only one single sample x i .
For graphs, one may no longer leverage the independence and compute the sample gradient ∇ g ( W ; x i ) by discarding the information of i ’s neighboring vertices and their neighbors, recursively.
We therefore seek an alternative formulation.
In order to cast the learning problem under the same sampling framework, let us assume that there is a (possibly inﬁnite) graph G (cid:48) with the vertex set V (cid:48) associated with a probability space ( V (cid:48) , F, P ) , such that for the given graph G , it is an induced subgraph of G (cid:48) and its vertices are iid samples of V (cid:48)
according to the probability measure
P .
For the probability space, V (cid:48) serves as the sample space and F may be any event space (e.g., the power set F = 2 V (cid:48) ).
The probability measure P deﬁnes a sampling distribution.
To resolve the problem of lack of independence caused by convolution, we interpret that each layer of the network deﬁnes an embedding function of the vertices (random variable) that are tied to the same probability measure but are independent.
See Figure 1.
Speciﬁcally, recall the architecture of
GCN
[Equation]
For the functional generalization, we write
[Equation]
Here, u and v are independent random variables, both of which have the same probability measure
P .
The function h ( l ) is interpreted as the embedding function from the l th layer, parameterized by W ( l ) .
The embedding functions from two consecutive layers are related through convolution, expressed as an integral transform.
The loss is the expectation of g ( h ( M ) ) for the ﬁnal embedding h ( M ) .
Note that the integrals are not the usual Riemann–Stieltjes integrals, because the variables u and v are graph vertices but not real numbers; however, this distinction is only a matter of formalism.
Writing GCN in the functional form allows for evaluating the integrals in the Monte Carlo manner, which leads to a batch training algorithm and also to a natural separation of training and test data, as in inductive learning.
For each layer l , we use t l iid samples u ( l ) 1 , . . .
, u ( l ) t l ∼ P to approximately evaluate the integral transform (2); that is,
[Equation]
Then, the loss L in (3) admits an estimator
[Equation]
The follow result establishes that the estimator is consistent.
The proof is a recursive application of the law of large numbers and the continuous mapping theorem; it is given in the appendix.
Theorem 1.
If g and σ are continuous, then
[Equation]
probability
[Equation]
[Figure]
In practical use, we are given a graph whose vertices are already assumed to be samples.
Hence, we will need bootstrapping to obtain a consistent estimate.
In particular, for the network architecture (1), the output H ( M ) is split into batches as usual.
We will still use u ( M ), . .
, u ( M ) t M to denote a batch of vertices, which come from the given graph.
For each batch, we sample (with replacement) uniformly each layer and obtain samples u ( l ) i , i = 1 , . . .
, t l , l = 0 , . . .
, M − 1 .
Such a procedure is equivalent to uniformly sampling the rows of H ( l ) for each l .
Then, we obtain the batch loss
[Equation]
where, recursively,
[Equation]
Here, the n inside the activation function σ is the number of vertices in the given graph and is used to account for the normalization difference between the matrix form (1) and the integral form (2).
The corresponding batch gradient may be straightforwardly obtained through applying the chain rule on each H ( l ) .
See Algorithm 1.
Algorithm
1: for each batch do 2:
For each layer l , sample uniformly t l vertices u ( l ) 1 , . . .
, u ( l ) t l 3: for each layer
l do (cid:46) Compute batch gradient ∇ L batch 4:
If v is sampled in the next layer,
[Equation]
5: end for 6:
W ← W − η ∇ L batch (cid:46) SGD
[Equation]
7: end for
3. V ARIANCE REDUCTION
As for any estimator, one is interested in improving its variance.
Whereas computing the full variance is highly challenging because of nonlinearity in all the layers, it is possible to consider each single layer and aim at improving the variance of the embedding function before nonlinearity.
Speciﬁcally, consider for the l th layer, the function ˜
h
( l +1)
t l +1 ( v ) as an approximation to the convolution (cid:82) ˆ a ( v, u ) h ( l ) t l ( u ) dP ( u ) .
When taking t l +1 samples v = u ( l +1), . .
, u ( l +1) t l +1 , the sample average of ˜ h
( l +1)
t l +1 ( v ) admits a variance that captures the deviation from the eventual loss contributed by this layer.
Hence, we seek an improvement of this variance.
Now that we consider each layer separately, we will do the following change of notation to keep the expressions less cumbersome:
Function
Samples
Num.
samples
Layer l + 1 ; random variable v
[Equation]
l
[Equation]
random variable
[Equation]
Under the joint distribution of v and u , the aforementioned sample average is
[Equation]
First, we have the following result.
Proposition 2.
The variance of G admits
[Equation]
where
[Equation]
and e ( v ) =
(cid:90) ˆ a ( v, u ) x ( u ) dP ( u ) .
The variance (6) consists of two parts.
The ﬁrst part R leaves little room for improvement, because the sampling in the v space is not done in this layer.
The second part (the double integral), on the other hand, depends on how the u j ’s in this layer are sampled.
The current result (6) is the consequence of sampling u j ’s by using the probability measure P .
One may perform importance sampling, altering the sampling distribution to reduce variance.
Speciﬁcally, let Q ( u ) be the new probability measure, where the u j ’s are drawn from.
We hence deﬁne the new sample average approximation
[Equation]
and the quantity of interest
[Equation]
Clearly, the expectation of G Q is the same as that of G , regardless of the new measure Q .
The following result gives the optimal Q .
Theorem 3.
If
[Equation]
where
[Equation]
then the variance of G Q admits
[Equation]
where R is deﬁned in Proposition 2.
The variance is minimum among all choices of Q .
A drawback of deﬁning the sampling distribution Q in this manner is that it involves | x ( u ) | , which constantly changes during training.
It corresponds to the product of the embedding matrix H ( l ) and the parameter matrix W ( l ) .
The parameter matrix is updated in every iteration; and the matrix product is expensive to compute.
Hence, the cost of computing the optimal measure Q is quite high.
As a compromise, we consider a different choice of Q , which involves only b ( u ) .
The following proposition gives the precise deﬁnition.
The resulting variance may or may not be smaller than (6).
In practice, however, we ﬁnd that it is almost always helpful.
Proposition 4.
If
[Equation]
where b ( u ) is deﬁned in (7) , then the variance of G Q admits
[Equation]
where R is deﬁned in Proposition 2.
With this choice of the probability measure Q , the ratio dQ ( u ) /dP
( u ) is proportional to b ( u ) 2 , which is simply the integral of ˆ a ( v, u ) 2 with respect to v .
In practical use, for the network architecture (1), we deﬁne a probability mass function for all the vertices in the given graph:
[Equation]
and sample t vertices u 1 , . . .
, u t according to this distribution.
From the expression of q , we see that it has no dependency on l ; that is, the sampling distribution is the same for all layers.
To summarize, the batch loss L batch in (4) now is recursively expanded as
[Equation]
The major difference between (5) and (10) is that the former obtains samples uniformly whereas the latter according to q .
Accordingly, the scaling inside the summation changes.
The corresponding batch gradient may be straightforwardly obtained through applying the chain rule on each H ( l ) .
See
Algorithm 2.
do do
∇
[Equation] [Equation]
3. I NFERENCE
The sampling approach described in the preceding subsection clearly separates out test data from training.
Such an approach is inductive, as oppose to transductive that is common for many graph algorithms.
The essence is to cast the set of graph vertices as iid samples of a probability distribution, so that the learning algorithm may use the gradient of a consistent estimator of the loss to perform parameter update.
Then, for inference, the embedding of a new vertex may be either computed by using the full GCN architecture (1), or approximated through sampling as is done in parameter learning.
Generally, using the full architecture is more straightforward and easier to implement.
3. C OMPARISON WITH G RAPH SAGE
GraphSAGE (Hamilton et al., 2017) is a newly proposed architecture for generating vertex embeddings through aggregating neighborhood information.
It shares the same memory bottleneck with
GCN, caused by recursive neighborhood expansion.
To reduce the computational footprint, the authors propose restricting the immediate neighborhood size for each layer.
Using our notation for the sample size, if one samples t l neighbors for each vertex in the l th layer, then the size of the expanded neighborhood is, in the worst case, the product of the t l ’s. On the other hand, FastGCN samples vertices rather than neighbors in each layer.
Then, the total number of involved vertices is at most the sum of the t l ’s, rather than the product.
See experimental results in Section 4 for the order-of-magnitude saving in actual computation time.
E XPERIMENTS
We follow the experiment setup in Kipf & Welling (2016a) and Hamilton et al.
(2017) to demonstrate the effective use of FastGCN, comparing with the original GCN model as well as GraphSAGE, on the following benchmark tasks: (1) classifying research topics using the Cora citation data set (McCallum et al., 2000); (2) categorizing academic papers with the Pubmed database; and (3) predicting the community structure of a social network modeled with Reddit posts.
These data sets are downloaded from the accompany websites of the aforementioned references.
The graphs have increasingly more nodes and higher node degrees, representative of the large and dense setting under which our method is motivated.
Statistics are summarized in Table 1.
We adjusted the training/validation/test split of Cora and Pubmed to align with the supervised learning scenario.
[Table]
Implementation details are as following.
All networks (including those under comparison) contain two layers as usual.
The code of GraphSAGE is downloaded from the accompany website, whereas
GCN is self implemented.
Testing with FastGCN is done with the full GCN network, as mentioned in Section 3.2.
Further details are contained in the appendix.
We ﬁrst consider the use of sampling in FastGCN.
The left part of Table 2 (columns under “Sampling”) lists the time and classiﬁcation accuracy as the number of samples increases.
For illustration purpose, we equalize the sample size on both layers.
Clearly, with more samples, the per-epoch training time increases, but the accuracy (as measured by using micro F1 scores) also improves generally.
Sampling
Precompute
[Table]
Time
F
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table] [Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure] [Figure]
An interesting observation is that given input features H (0) , the product ˆ AH (0) in the bottom layer does not change, which means that the chained expansion of the gradient with respect to W (0) in the last step is a constant throughout training.
Hence, one may precompute the product rather than sampling this layer to gain efﬁciency.
The compared results are listed on the right part of Table(columns under “Precompute”).
One sees that the training time substantially decreases while the accuracy is comparable.
Hence, all the experiments that follow use precomputation.
Next, we compare the sampling approaches for FastGCN: uniform and importance sampling.
Figure 2 summarizes the prediction accuracy under both approaches.
It shows that importance sampling consistently yields higher accuracy than does uniform sampling.
Since the altered sampling distribution (see Proposition 4 and Algorithm 2) is a compromise alternative of the optimal distribution that is impractical to use, this result suggests that the variance of the used sampling indeed is smaller than that of uniform sampling; i.e., the term (9) stays closer to (8) than does (6).
A possible reason is that b ( u ) correlates with | x ( u ) | .
Hence, later experiments will apply importance sampling.
[Figure]
Cora
Pubmed
Reddit
FastGCN 0.
GraphSAGE-GCN 0.
GraphSAGE-mean 0.
GCN (batched) 0. GCN (original) 0.
NA refers to using batched learning, as oppose to the original version that is nonbatched; for more details of the implementation, see the appendix.
The nonbatched version of GCN runs out of memory on the large graph Reddit.
The sample sizes for FastGCN are 400, 100, and 400, respectively for the three data sets.
We now demonstrate that the proposed method is signiﬁcantly faster than the original GCN as well as
GraphSAGE, while maintaining comparable prediction performance.
See Figure 3.
The bar heights indicate the per-batch training time.
One sees that GraphSAGE is a substantial improvement of GCN for large and dense graphs (e.g., Reddit), although for smaller ones (Cora and Pubmed), GCN trains faster.
FastGCN is the fastest, with at least an order of magnitude improvement compared with the runner up (except for Cora), and approximately two orders of magnitude speed up compared with the slowest.
Here, the training time of FastGCN is with respect to the sample size that achieves the best prediction accuracy.
As seen from the table on the right, this accuracy is highly comparable with the best of the the other two methods.
C ONCLUSIONS
We have presented FastGCN, a fast improvement of the GCN model recently proposed by Kipf &
Welling (2016a) for learning graph embeddings.
It generalizes transductive training to an inductive manner and also addresses the memory bottleneck issue of GCN caused by recursive expansion of neighborhoods.
The crucial ingredient is a sampling scheme in the reformulation of the loss and the gradient, well justiﬁed through an alternative view of graph convoluntions in the form of integral transforms of embedding functions.
We have compared the proposed method with additionally
GraphSAGE (Hamilton et al., 2017), a newly published work that also proposes using sampling to restrict the neighborhood size, although the two sampling schemes substantially differ in both algorithm and cost.
Experimental results indicate that our approach is orders of magnitude faster than GCN and GraphSAGE, while maintaining highly comparable prediction performance with the two.
A P ROOFS
Proof of Theorem 1.
Because the samples u (0) j are iid, by the strong law of large numbers,
[Equation]
converges almost surely to ˜ h
(1) ( v ) .
Then, because the activation function σ is continuous the continuous mapping theorem implies that h (1) t 1 ( v ) =
σ (˜ h (1) t 1 ( v )) converges almost surely to h (1) ( v )
=
σ (˜ h (1) ( v )) .
Thus, (cid:82) ˆ a ( v, u ) h (1) t 1 ( u ) dP ( u ) converges almost surely to ˜ h
(2) ( v ) =
(cid:82) ˆ a ( v, u ) h (1) ( u ) dP ( u ) , where note that the probability space is with respect to the 0 th layer and hence has nothing to do with that of the variable u or v in this statement.
Similarly,
[Equation]
converges almost surely to (cid:82) ˆ a ( v, u ) h
(1) t 1 ( u ) dP ( u ) and thus to ˜ h
(2) ( v ) .
A simple induction completes the rest of the proof.
Proof of Proposition 2.
Conditioned on v , the expectation of y ( v ) is
[Equation]
and the variance is 1 /t times that of ˆ a ( v, u ) x ( u ) , i.e.,
[Equation]
Instantiating (11) and (12) with iid samples v 1 , . .
. , v s ∼ P and taking variance and expectation in the front, respectively, we obtain
Var
[Equation]
v ) dP
[Equation]
v
[Equation]
, and
E
[Equation]
Var
[Equation]
Then, applying the law of total variance
Var
[Equation]
we conclude the proof.
Proof of Theorem 3.
Conditioned on v , the variance of y Q ( v ) is 1 /t times that of
[Equation]
where u ∼ Q ) , i.e.,
[Equation]
Then, following the proof of Proposition 2, the overall variance is
[Equation]
Hence, the optimal dQ ( u ) must be proportional to b ( u ) | x ( u ) | dP ( u ) .
Because it also must integrate to unity, we have
[Equation]
in which case
[Equation]
Proof of Proposition 4.
Conditioned on v , the variance of y Q ( v ) is 1 /t times that of
[Equation]
i.e.,
[Equation]
ˆ a ( v, u )b ( u ) 2 dQ ( u ) − e ( v )(cid:33) .
The rest of the proof follows that of Proposition 2.
B A DDITIONAL E XPERIMENT D ETAILS B. B ASELINES
GCN : The original GCN cannot work on very large graphs (e.g., Reddit).
So we modiﬁed it into a batched version by simply removing the sampling in our FastGCN (i.e., using all the nodes instead of sampling a few in each batch).
For relatively small graphs (Cora and Pubmed), we also compared the results with the original GCN.
GraphSAGE : For training time comparison, we use GraphSAGE-GCN that employs GCN as the aggregator.
It is also the fastest version among all choices of the aggregators.
For accuracy comparison, we also compared with GraphSAGE-mean.
We used the codes from https: //github.com/williamleif/GraphSAGE .
Following the setting of Hamilton et al.
, we use two layers with neighborhood sample sizes S 1 = 25 and S 2 = 10 .
For fair comparison with our method, the batch size is set to be the same as FastGCN, and the hidden dimension is 128.
B. E XPERIMENT S ETUP
Datasets: The Cora and Pubmed data sets are from https://github.com/tkipf/gcn .
As we explained in the paper, we kept the validation index and test index unchanged but changed the training index to use all the remaining nodes in the graph.
The Reddit data is from http: //snap.stanford.edu/graphsage/ .
Experiment Setting: We preformed hyperparameter selection for the learning rate and model dimension.
We swept learning rate in the set { 0.01, 0.001, 0.0001 } .
The hidden dimension of FastGCN for Reddit is set as 128, and for the other two data sets, it is 16.
The batch size is
Hardware: Running time is compared on a single machine with 4-core 2.5 GHz Intel Core i7, and 16G RAM.