C HORDAL -GCN: E XPLOITING S PARSITY IN T RAINING L ARGE -S CALE G RAPH C ONVOLUTIONAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the impressive success of graph convolutional networks (GCNs) on numerous applications, training on large-scale sparse networks remains challenging.
Current algorithms require large memory space for storing GCN outputs as well as all the intermediate embeddings.
Besides, most of these algorithms involves either random sampling or an approximation of the adjacency matrix, which might unfortunately lose important structure information.
In this paper, we propose
Chordal-GCN for semi-supervised node classiﬁcation.
The proposed model utilizes the exact graph structure ( i.e. , without sampling or approximation), while requires limited memory resources compared with the original GCN.
Moreover, it leverages the sparsity pattern as well as the clustering structure of the graph.
The proposed model ﬁrst decomposes a large-scale sparse network into several small dense subgraphs (called cliques ), and constructs a clique tree.
By traversing the tree, GCN training is performed clique by clique, and connections between cliques are exploited via the tree hierarchy.
Furthermore, we implement Chordal-GCN on large-scale datasets and demonstrate superior performance.
I NTRODUCTION
Graph convolutional network (GCN) (Kipf & Welling, 2017) is a generalization of convolutional neural networks (CNNs) (LeCun & Bengio, 1998) to the graph structure.
For a given node, the graph convolution operation aggregates the embeddings (features) of its neighbors, followed by a non-linear transformation.
By stacking multiple graph convolutional layers, one can learn node representations by utilizing features of its distant neighborhood.
The original GCN model, as well as its numerous variations, has shown great success in a variety of applications, including semisupervised node classiﬁcation (Kipf & Welling, 2017), inductive node embedding (Hamilton et al., 2017), link prediction (van den Berg et al., 2017), and knowledge graphs (Schlichtkrull et al., 2018).
Despite the success of GCNs, training GCNs on large-scale graphs remains challenging due to the memory issue: we need to store all the parameters and outputs of GCN.
Thus, the memory space scales linearly in the size of graph while quadratically in the feature dimension (Chiang et al., 2019;
Zou et al., 2019).
This prevents applications of GCN on many real-world networks, where the graphs usually contain millions or even billions of nodes.
Methods aimed at large-scale training have been proposed and can be roughly divided into two categories: (1) sampling-based methods and (2) clustering-based methods.
For sampling-based methods, only a few neighbors for every node will be sampled in every GCN layer, and thus the size of intermediate embeddings for every layer will be reduced for each mini-batch.
Works in this track include Hamilton et al.
; Chen et al.
(2018a;b); Zou et al.
However, ignorance of some neighbors might lose important structure information, which is the main drawback of all the sampling methods.
Another direction of research notices the sparsity of real-world networks and exploits the clustering structure of the graph.
For example, Cluster-GCN (Chiang et al., 2019) separates the graph into several clusters, and in every iteration of training, only one or a few clusters are picked to calculate the stochastic gradient for the mini-batch.
However, Cluster-GCN ignores all the inter-cluster links, which are not negligible in many real-world networks.
For example, Figureshows the sparsity pattern of three citation networks.
We ﬁrst rearrange the vertices via an approximate minimum degree (AMD) ordering algorithm (Amestoy et al., 1996), and then observe a nice arrow pattern in the adjacency matrices.
This indicates the existence of some highly-cited papers observed
(in the right bottom corner), which have impacts on multiple communities.
In Cluster-GCN, these highly-cited papers are randomly put into one community and the adjacency matrix is approximated by a block-diagonal matrix; i.e. , Cluster-GCN removes the ﬂetching part of the arrow patterns in Figure 1.
This approximation ruins the beautiful arrow pattern, and thus ignores the multi-community inﬂuence of some seminal papers.
The above difﬁculties can be easily tackled by the Chordal-GCN , a novel clustering-based method for the semi-supervised node classiﬁcation task.
Recall that Cluster-GCN ignores all the intercluster links and trains each cluster separately; comparatively, in Chordal-GCN, we keep all the links, train every cluster separately, and at the same time capture the connections between clusters by an additional loss term.
This partially separable training process can be achieved with the help of chordal sparsity theory (Vandenberghe & Andersen, 2015): we ﬁrst decompose a large-scale sparse graph into several small dense subgraphs, and we construct a tree of which the nodes are the subgraphs.
Note that two subgraphs are adjacent in the tree if they share some vertices.
In the training process, we traverse the tree from leaf to root; and when training on a certain subgraph, we minimize the usual GCN loss, plus an additional term called consistency loss .
With the consistency loss, messages in the children subgraphs can be passed to their parent, and thus the relationship between subgraphs is leveraged via the hierarchy of the tree.
Therefore, Chordal-GCN exploits the sparsity pattern as well as the clustering structure of the graph without any approximation or random sampling, while requires similar memory space to Cluster-GCN.
Our contribution is summarized as follows:
• We propose Chordal-GCN for semi-supervised node classiﬁcation on large-scale sparse networks.
The proposed model fully exploits the exact graph structure, while requires limited memory usage on large-scale graphs (much smaller than the original GCN).
• Chordal-GCN is able to train a large-scale graph in a partially separable manner; i
e. , in every iteration, the training is performed on a subgraph, and the connections between subgraphs are handled by a consistency loss.
• We analyze the memory and time complexity of Chordal-GCN and compare them with other state-of-the-art GCN models.
Also, we evaluate the performance of Chordal-GCN on benchmark datasets and demonstrate superior performance in large-scale datasets.
B ACKGROUND AND RELATED WORK 2. S EMI - SUPERVISED NODE CLASSIFICATION WITH GCN S
Suppose G =
( V, E ) is an undirected graph with | V | = n and the n ×
n symmetric matrix
A is the adjacency matrix.
Every node k is associated with a feature vector x k ∈ R d , and all the feature vectors are stored in the rows of X ∈ R n × d .
An L -layer graph convolutional network (GCN) (Kipf & Welling, 2017) has the layer-wise propagation rule:
[Equation]
e a d c b
[Figure]
) e a d c b [Figure] [Figure] where W ( l ) ∈ R d l × d l +1 is a weight matrix.
The matrix ¯ A = ˜ D − 1 / 2 ˜
A ˜ D − 1 / 2 is the normalized adjacency matrix, where
˜ A = I + A and ˜ D ∈ R n × d is diagonal with ˜ D ii
=
(cid:80)
j ˜
A ij .
The matrix
H ( l ) ∈ R n × d l are the activations in the l -th layer ( H (0) =
X and d 0 = d ).
The function σ l is the activation function for layer l .
The output of an L -layer GCN is an n × p matrix H ( L ) , of which the k -th row is the predicted label for node k .
Later in the paper, we will train GCN using only a subgraph G ( γ ) , so the notation y (cid:104) pred (cid:105) k
( W ; X γ, : , ¯ A γγ ) is used to indicate the predicted label of node k , using the coefﬁcient matrices W =
( W (0) , . . .
, W ( L − 1) ) , the submatrix ¯ A
γγ , and the corresponding feature vectors X γ, : .
Semi-supervised node classiﬁcation is a popular application of GCN.
When applying GCN for this task, we minimize the loss function
[Equation]
[Table]
k
[Equation]
(cid:104) pred (cid:105)
[Equation]
(1) to learn the weight matrices W =
( W (0) , . . .
, W ( L − 1) ) .
In the formula, y (cid:104) true (cid:105) k is the given true label for node k , and V l ⊆
V is the subset of vertices with given true labels.
The function (cid:96) 1 ( · , · ) is a loss function, and usually it is the cross entropy loss.
Much work has been done to apply GCN on large-scale datasets.
For the sampling-based methods, different rules for random selection have been proposed: uniform sampling as in Hamilton et al.
(2017), or importance sampling in Chen et al.
(2018a).
On the other hand, clustering-based methods become more interesting in the recent literature.
Besides Cluster-GCN (Chiang et al., 2019), clustering structure can also be utilized implicitly as a low-rank approximation of the graph Laplacian.
This idea is exploited in LanczosNet (Liao et al., 2019) as well as the Lov ´ asz convolutional network (Yadav et al., 2019).
2. P RELIMINARIES ON CHORDAL GRAPHS
Chordal sparsity has been a classical topic in graph theory and found useful applications in various ﬁelds, including database theory (Beeri et al., 1983), probabilistic networks (Pearl, 1988; Cowell et al., 1999; Darwiche, 2009), linear algebra (Rose, 1970), combinatorial optimization (Golumbic, 2004; Gavril, 1972), semideﬁnite optimization (Andersen et al., 2010; 2013).
In this paper, we will use chordal sparsity theory to exploit the clustering structure.
So now we show that a chordal graph can be separated into cliques, and that these cliques form a clique tree with desirable properties.
Ordered undirected graph.
Given an undirected graph G =
( V, E ) with | V | = n , we can order the vertices from 1 to n , and then we refer to a vertex in V by its order i ∈ [ n ] = { 1 , 2 , . . . , n } .
In addition, an index set γ ⊂ [ n ] represents a subset of vertices in V .
Cliques.
A complete graph is a graph in which every pair of distinct vertices is adjacent.
A clique γ ⊆ V of a graph G is a subset of vertices that induces a maximal complete subgraph.
The graph induced by γ is denoted G ( γ ) =
( γ, E ( γ )) and has edge set E ( γ ) =
{{ v, w } ∈ E | v, w ∈ γ } .
6 , 8 ,5 , 6 ,9 , 9 ,1 ,10 ,8 ,14 , 15 ,11 ,16 , 13 , 14 , 15 , 16 ,separator α i residual
ν
i clique γ
i
Chordal graph. A path between v 0 and v k is a sequence of distinct vertices ( v 0 , v 1 , v 2 , . . , v k ) with { v i , v i +1 } ∈ E for i = 0 , 1 , . . . , k − 1 .
For example, the graph in Figure 2a has a path ( a, e, c, d ) .
A chord in a path of an undirected graph is an edge between non-consecutive vertices on the path.
For example, in Figure 2a, the edge { d, e } is a chord in the path ( a, e, c, d ) , but in
Figure 2b, the path ( b, a, c, d ) is chordless.
An undirected graph is chordal if every cycle of length greater than three has a chord.
Figure 2b and 2c show two examples of chordal graphs, while the graph in Figure 2a is non-choral.
Postordered clique tree.
Given a (connected) chordal graph G =
( V, E ) , we can always construct a clique tree T c : the nodes of T c are the cliques γ ⊆ V and two cliques are adjacent in T c if they share some vertices in V .
In addition, we can arbitrarily pick a clique as the root of the tree, and then every non-rooted clique can be partitioned into clique separators and clique residuals .
The clique separator is the intersection of the clique with its parent clique in T c .
The clique residual is also called the (maximal) supernode , and by deﬁnition, all the supernodes form a partition of V .
More interestingly, we can always ﬁnd an ordering of vertices V so that the clique tree T c satisﬁes the following properties.
The higher neighborhood of every vertex is complete: j, k ∈ adj + ( i ) implies { j, k } ∈ E , where the higher neighborhood of vertex i is deﬁned as adj + ( i ) =
{ j | { i, j } ∈ E, j > i } .
This means that every clique and supernode in T c can be written as γ i and ν i , respectively, where i ∈ V has the lowest order in the clique (and supernode).
We denote by i the representative of the supernode ν i , by α i =
γ
i \
ν
i the corresponding clique separator, and by V c the set of all representatives.
The index set α i is empty if i is the root of T c . 2.
The elements of each supernode are numbered consecutively:
ν i
= {
i, i + 1 , . . .
, i + n i } with n i = | ν i − 1 | .
All the representatives i ∈ V c are topologically ordered: i < p ( i ) if i ∈ V c and p ( i ) is the representative of the parent of γ i in T c .
An ordering satisfying the above three properties is called topological postordering , and a clique tree T c with certain topological postordering is referred to as a postordered clique tree .
Figureshows a supernode partition of the chordal graph in Figure 2c with a topological postordering.
(This example is adapted from the survey paper Vandenberghe & Andersen (2015).)
In this example, γ
[Equation]
= { 5 , 6 , 7 , 8 , 10 } , ν {+
+} = { 5 , 6 , 7 } , α 5 = { 8 , 10 } .
Given a chordal sparsity graph G =
( V, E ) , there exist efﬁcient algorithms (in O ( | E | ) ) to ﬁnd a topological postordering, and to generate a postordered clique tree T c together with the parent function p ( i ) and all the index sets ( i.e. , γ i , ν i , and α i for all i ∈ V c ) (Lewis et al., 1989; Pothen &
Sun, 1990).
The whole process is called chordal decomposition (Vandenberghe & Andersen, 2015).
C HORDAL -GCN
Now we are ready to utilize the chordal decomposition technique in training large-scale sparse networks.
We note that most real-world networks are indeed sparse, but not chordal.
So we ﬁrst introduce some data preprocessing steps, followed by the detailed description of Chordal-GCN.
We also analyze the time and memory complexity of our model, and compare it with other GCN models.
3. P REPROCESSING
Reordering and chordal extension.
Chordal extension describes the idea of transforming a nonchordal graph G =
( V, E ) into a chordal graph G (cid:48)
=
( V, E (cid:48) ) by adding edges, i.e. , E ⊆ E (cid:48) .
Unfortunately, to ﬁnd the minimum ﬁll-in is claimed to be an NP-complete problem (Rose et al., 1976; Yannakakis, 1981), and is not always desirable due to algorithm complexity.
However, given an ordering of vertices, it is practical to ﬁnd a minimal chordal extension (Ohtsuki, 1976; Rose, 1974).
(It is called minimal if the removal of any added edge results in a non-chordal graph.)
That is to say, every minimal chordal extension is associated with certain vertex ordering.
Thus, the reordering of vertices is important, especially when we prefer a sparse extended graph.
So in this work, we use an approximate minimum degree (AMD) ordering algorithm (Amestoy et al., 1996) to ﬁrst reorder the vertices, and then ﬁnd a minimal chordal extension.
Clique merging.
Often, the resulting extension contains many small cliques and it would be more efﬁcient to merge some neighboring cliques, as reported in Ashcraft & Grimes (1989).
In this work, traversing the tree from leaf to root, we greedily merge clique γ i with its parent γ p ( i ) if | ν i | ≤ τ and | ν p ( i ) | ≤ τ where τ is a pre-deﬁned threshold.
We denote the resulting graph as G (cid:48)(cid:48)
=
( V, E (cid:48)(cid:48) ) .
Chordal decomposition.
Given the graph G (cid:48)(cid:48) , we can now construct the postordered clique tree T c (together with all the index sets ( γ i , ν i , and α i ), and the parent function p ( i ) ).
With a little abuse of notation, we also use T c to indicate the postordered clique tree of G .
In other words, the preprocessing steps reorder and partition the vertex set V , which helps us identify the clique γ i and the corresponding induced subgraph G ( γ i ) .
Although chordal extension and clique merging add edges into the graph, it will not affect the training process of GCN because we will use the subgraph
G ( γ i ) , not G (cid:48)(cid:48) ( γ i ) .
Lastly, we emphasize that, although G ( γ i ) no longer represents a complete subgraph, we still refer to γ i as clique to distinguish from clusters or communities : in the chordal decomposition, one node k ∈ V is allowed to exist in multiple cliques while in community detection or graph clustering, one node appears in only one cluster.
3. C HORDAL -GCN: META ALGORITHM
With the postordered clique tree T c in hand, we are able to train a large-scale sparse graph in a partially separable manner, i.e. , clique by clique.
Suppose we traverse the tree T c in the topological order, i.e. , from leaf to root.
At clique γ i , the loss function consists of two parts.
The ﬁrst part is the usual GCN loss deﬁned in (1), but using only the induced subgraph G ( γ i ) instead of the entire G :
[Equation]
where X γ i , : takes the rows of X indexed by γ i and ¯ A γ i γ i is the submatrix of ¯ A with rows and columns indexed by γ i .
The second part of the loss is the consistency loss .
When we arrive at clique γ i , we have traversed all its children in the tree T c .
This means that some nodes in γ i have already been visited; namely, those clique separators of γ i ’s children in T c .
For clarity, we denote by ch( i ) the set of children
L i ( W
[Equation]
= (cid:88)
k ∈
V l ∩ γ i (cid:96) 1 (cid:0)
y k , y (cid:104) pred (cid:105) k
( W ; X γ i , :
[Equation]
¯ A γ i γ i ) (cid:1) + (cid:88) j ∈ ch( i ) (cid:88) k ∈ α j (cid:96) 2 (cid:0) y (cid:104) pred (cid:105) k
( W ; X γ i , : , ¯ A γ i γ i ) , y (cid:104) pred (cid:105) k
( W [ t ] ; X γ j , : , ¯ A γ j γ j ) (cid:1) .
representatives of γ i ; i.e. , j ∈ ch( i ) if j is a representative ( i.e. , j ∈ V c ) and γ j is a child of γ i in T c .
Then all the nodes in α j for j ∈ ch( i ) have already been visited when we arrive at γ i .
Thereby, for k ∈ α j and j ∈ ch( i ) , we hope that the predicted label y (cid:104) pred (cid:105) k
( · ; X γ i , : , ¯ A γ i γ i ) using the current clique γ i is consistent with the previous prediction y (cid:104) pred (cid:105) k
( ˜ W ; X γ j , ¯ A γ j γ j ) using clique γ j and the most recent parameter ˜ W .
Hence, the consistency loss for training clique γ i can be formulated as
[Equation]
where (cid:96) 2 ( · , · ) is a loss function, for example, the KL divergence.
Note that ˜ W is not treated as the variable of L (cid:104) cons (cid:105) ; i.e. , the term y (cid:104) pred [Equation] k ( ˜ W ; X
[Equation]
j , : , ¯ A γ j γ j ) is considered constant when we take the gradient of L (cid:104) cons (cid:105) .
Therefore, when training the clique γ i , we compute the total loss
[Equation]
and perform a gradient descent step W + = W − η ∇L i ( W ) where η is the learning rate.
The whole algorithm is summarized in Algorithm 1.
Note that the gradient descent step in line 7 of Algorithmcan be replaced with any other accelerated version, for example, Adam (Kingma & Ba, 2015).
3.2. C ONNECTION TO THE ORIGINAL GCN
We note that Algorithm 1 is essentially the mini-batch gradient descent method applied to the loss
[Equation]
with γ i taken as the mini-batch.
Clearly, it is equal to the original GCN loss (1) if we have only one clique ( i.e. , V c = { 1 } and thus γ 1 = V ).
Also, when ¯ A is block diagonal, Chordal-GCN is equivalent to Cluster-GCN because the consistent loss (3) vanishes.
However, in the general case, without any assumption on the graph structure or the GCN structure, it is impossible to show the equivalence between Chordal-GCN and GCN.
From another view of point, the non-convex functions (1) and (5) don’t share any global minimizer.
For example, consider the chordal graph with two overlapping dense principal submatrices.
We can easily choose the set of true labels y (cid:104) true (cid:105) such that (1) and (5) don’t have the same minimizer.
3.2. T IME AND MEMORY EFFICIENCY
Following Chiang et al.
(2019), we report the time and memory complexity in Table 1.
Although our method has the same time complexity as Cluster-GCN in big-O notation, Chordal-GCN needs
[Table]
an extra forward propagation on the children cliques in order to compute the predicted labels
[Table] k ( W [ t ] ; X γ j , : , ¯ A γ j γ j ) in (2).
The extra forward propagation step would be the only price paid for single node being in multiple subgraphs.
As we will see in the numerical experiments, this additional cost is minimal and worthwhile, especially in citation networks with an arrow pattern.
In addition, the memory bottleneck for Chordal-GCN and Cluster-GCN depend on the maximum cluster size and maximum clique size, respectively.
Since these two numbers depend totally on the hyperparameters, i.e. , number of clusters in METIS algorithm and the merging threshold τ , it is difﬁcult to compare analytically.
But both are much smaller than the memory usage of the original
GCN, which is desirable.
E XPERIMENTS 4. E XPERIMENT SETTINGS
We evaluate our proposed model for the semi-supervised node classiﬁcation task on four public datasets: cora, citeseer, pubmed, and reddit.
We compare Chordal-GCN with the following stateof-the-art GCN training algorithms for comparison: GCN (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017), FastGCN (both uniform and importance sampling)
(Chen et al., 2018a),
VR-GCN (Chen et al., 2018b), and Cluster-GCN (Chiang et al., 2019).
For all the methods we use the same GCN structure ( i.e. , L = 2 ), the same label rate, and Adam optimizer (Kingma & Ba, 2015) with learning rate 0.001, dropout rate as 20%, weight decay as zero.
For the baselines, we use the implementations provided by the authors, and follow the default parameter settings in these models.
Other implementation details are included in appendix A.
For all the methods and datasets, we conduct training for 10 times and take the mean of the evaluation results.
Also, we stop training when the validation accuracy does not increase a threshold ( 0 . 01 ) for 10 epochs, and choose the model with the highest validation accuracy as convergent point.
We use the following metrics to evaluate the performance of all methods.
• Accuracy: The micro F1 score of the test data at the convergent point.
• Memory usage: The maximum GPU memory occupied by tensors.
• Epoch time and number of epochs: Time to run an epoch and total number of epochs before convergence.
Note that in Chordal-GCN, an epoch is a traverse of the clique tree, rather than one clique.
4. N UMERICAL RESULTS
We summarize all the results in Table 2.
In the three citation networks (cora, citeseer, and pubmed),
Chordal-GCN achieves the highest accuracy among all baseline models.
This desirable result is due to the exploitation of the exact sparsity pattern and clustering structure.
In terms of memory space, Chordal-GCN is superior to the original GCN, just as expected.
All the sampling-based methods require limited memory space because they reduce the number of intermediate node embeddings.
More interestingly, Chordal-GCN uses less memory space than Cluster-
[Table]
GCN (except in reddit).
This is because the Cluster-GCN implementation uses a few clusters in one epoch
[Table]
and thus the memory bottleneck depends on the sum of the selected clusters, rather than the maximum cluster.
The abnormal memory cost of Chordal-GCN in reddit dataset is attributed to our clique merging heuristic.
The maximal clique in reddit has size 220 , 069 , almost 95% of the vertices.
Splitting the graph into more cliques will help reduce the memory requirement, while increases the epoch time: we train smaller but more cliques in one epoch.
Lastly
[Table]
we emphasize that the comparison of epoch time can never be fair: one epoch in ChordalGCN means one traverse of the clique tree while in other methods one epoch is one mini-batch.
This explains the long epoch time of Chordal-GCN.
C ONCLUSION
We propose Chordal-GCN for semi-supervised node classiﬁcation on large-scale sparse networks.
The proposed model exploits the sparsity pattern and clustering structure of the graph
[Table]
and utilizes the exact graph structure ( i.e. , without sampling or approximation).
Moreover, the memory usage of Chordal-GCN is limited because the training is performed in a partially separable manner; i.e. , clique by clique.
Experiment results demonstrate that Chordal-GCN achieves the best test accuracy with much smaller memory cost on benchmark datasets.
A I MPLEMENTATION DETAILS
Hardware and software.
We use Tesla V100 GPU, Intel Xeon CPU E5-2698 (2.20GHz) with 500GB.
We use PyTorch 1.2.0 in our implementation and Tensorﬂow 1.13 to test the baselines.
Datasets.
The cora, citeseer
[Table]
and pubmed datasets are from https://github.com/tkipf/ gcn .
The reddit data is from https://snap.stanford.edu/graphsage .
We report the data statistics in Table
[Table]
[Table]
[Table]
Preprocessing.
In Chordal-GCN, we use the AMD ordering (Amestoy et al., 1996) implemented in the python package CVXOPT (Andersen et al., 2015).
The minimal chordal extension and the chordal decomposition are performed using the python package chompack (Andersen & Vandenberghe, 2015).
We also use the clique merging heuristic described in section 3.1.
The threshold τ is different in different datasets, and mainly depends on the size of the graph.
We report the parameters in Table 4.
GCN structure.
We implement our model with PyTorch.
We use a two-layer GCN with ﬁxed parameter size 128 .
To calculate the memory usage, we use tf.contrib.memory BytesInUse() for TensorFlow and torch.cuda.memory allocated() for PyTorch.
B A DDITIONAL EXPERIMENTS B. A BLATION STUDY
We evaluate the contribution of the consistency loss L (cid:104) cons (cid:105) (3) here.
We train Chordal-GCN with and without L (cid:104) cons (cid:105) on the cora, citeseer, and pubmed datasets, and plot the validation accuracy versus the number of epochs.
Figure 4 shows that the consistency loss L (cid:104) cons (cid:105) correctly captures the connection between cliques, and that improve the performance of Chordal-GCN.
This explains why our model performs consistently better than Cluster-GCN: Cluster-GCN ignores those inter-cluster links and thus loses the connections between clusters.
In the pubmed dataset, Chordal-GCN without consistency loss is even overﬁtting: the validation accuracy decreases after several epochs
[Figure]
B. T RAINING DEEP GCN S
In this section
[Figure]
we study the performance of deeper Chordal-GCN [Figure] and explore how time and memory cost scales with GCN depth.
As shown in Figure [Figure] both memory and time cost scale linearly with the number of layers.
This result is consistent with theoretical analysis
[Figure]
and shows that ChordalGCN is applicable to training deeper GCNs
[Figure]
[Figure]
layers m e m o r y c o s t ( M B )           number of GCN layers ti m e c o m p l e x it y ( m s )