Original: P IPE GCN: E FFICIENT F ULL -G RAPH T RAINING 
Revised: P IPE GCN: E FFICIENT F ULL -G RAPH T RAINING OF G RAPH C ONVOLUTIONAL N ETWORKS WITH P IPELINED F EATURE C OMMUNICATION

Original: OF

Original: G RAPH

Original: C ONVOLUTIONAL

Original: N ETWORKS

Original: WITH

Original: P IPELINED

Original: F EATURE C OMMUNICATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph Convolutional Networks (GCNs) is the state-of-the-art method for learning graph-structured data, and training large-scale GCNs requires distributed training across multiple accelerators such that each accelerator is able to hold a partitioned subgraph.
Revised: Graph Convolutional Networks (GCNs) is the state-of-the-art method for learning graph-structured data, and training large-scale GCNs requires distributed training across multiple accelerators such that each accelerator is able to hold a partitioned subgraph.

Original: However, distributed GCN training incurs prohibitive overhead of communicating node features and feature gradients among partitions for every GCN layer in each training iteration, limiting the achievable training efﬁciency and model scalability.
Revised: However, distributed GCN training incurs prohibitive overhead of communicating node features and feature gradients among partitions for every GCN layer during each training iteration, limiting the achievable training efﬁciency and model scalability.

Original: To this end, we propose PipeGCN, a simple-yet-effective scheme that hides the communication overhead by pipelining inter-partition communication with intra-partition computation.
Revised: To this end, we propose PipeGCN, a simple yet effective scheme that hides the communication overhead by pipelining inter-partition communication with intra-partition computation.

Original: It is non-trivial to pipeline for efﬁcient GCN training, as communicated node features/gradients will become stale and thus can harm the convergence, negating the pipeline beneﬁt.
Revised: It is non-trivial to pipeline for efﬁcient GCN training, as communicated node features/gradients will become stale and thus can harm the convergence, negating the pipeline beneﬁt.

Original: Notably, little is known regarding the convergence rate of GCN training with both stale features and stale feature gradients.
Revised: Notably, little is known regarding the convergence rate of GCN training with both stale features and stale feature gradients.

Original: This work not only provides a theoretical convergence guarantee but also ﬁnds the convergence rate of PipeGCN to be close to that of the vanilla distributed GCN training without  staleness.
Revised: This work not only provides a theoretical convergence analysis but also ﬁnds the convergence rate of PipeGCN to be close to that of the vanilla distributed GCN training without any staleness.

Original: Furthermore, we develop a smoothing method to further improve PipeGCN’s convergence.
Revised: Furthermore, we develop a smoothing method to further improve PipeGCN’s convergence.

Original: Extensive experiments show that PipeGCN can largely boost  training throughput (up to 2.2 × ) while achieving the same accuracy as its vanilla counterpart andoutperforming existing full-graph training methods.
Revised: Extensive experiments show that PipeGCN can largely boost the training throughput (1.7 ×∼ 28.5 × ) while achieving the same accuracy as its vanilla counterpart and existing full-graph training methods.

Original: All code will be released publicly upon acceptance.
Revised: The code is available at https://github.com/RICE-EIC/PipeGCN.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph convolutional networks (GCNs) (Kipf & Welling, 2016) have gained great popularity recently as they demonstrated the state-of-the-art (SOTA) performance invarious graph-based learning tasks, including node classiﬁcation (Kipf & Welling, 2016), link prediction (Zhang & Chen, 2018), graph classiﬁcation (Xu et al., 2018), and recommendation systems (Ying et al., 2018).
Revised: Graph Convolutional Networks (GCNs) (Kipf & Welling, 2016) have gained great popularity recently as they demonstrated the state-of-the-art (SOTA) performance in learning graph-structured data (Zhang & Chen, 2018; Xu et al., 2018; Ying et al., 2018).

Original: The promising performanceof GCNs is due to their diverse neighborhood connectivity, which provides a greater applicability to graph-based data than convolutional neural networks (CNNs) that adopt a ﬁxed regular neighborhood structure.
Revised: Their promising performance is resulting from their ability to capture diverse neighborhood connectivity.

Original: In particular, a GCN aggregates all features from the neighbor node set for a given node, the feature of which is then updated via a multi-layer perceptron.
Revised: In particular, a GCN aggregates all features from the neighbor node set for a given node, the feature of which is then updated via a multi-layer perceptron.

Original: Such a two-step process (i.e., aggregate and  update ) empowers GCNs to better learn graph structures.
Revised: Such a two-step process ( neighbor aggregation and node update ) empowers GCNs to better learn graph structures.

Original: However, training GCNs at scale has been a challenging problem, as a prohibitive amount of compute and memory resources are required to train a real-world large-scale graph, let alone exploring deeper and more advanced models.
Revised: Despite their promising performance, training GCNs at scale is still a challenging problem, as a prohibitive amount of compute and memory resources are required to train a real-world large-scale graph, let alone exploring deeper and more advanced models.

Original: To tackle this challenge, various sampling-based methods have been proposed to reduce the resource requirement at a cost of incurring feature approximation errors.
Revised: To overcome this challenge, various sampling-based methods have been proposed to reduce the resource requirement at a cost of incurring feature approximation errors.

Original: A straightforward instance is to create mini-batches by sampling neighbors (e.g., GraphSAGE (Hamilton et al., 2017) and VR-GCN (Chen et al., 2018)) or to extract subgraphs as training samples (e.g., Cluster-GCN (Chiang et al., 2019) and GraphSAINT (Zeng et al., 2020)).
Revised: A straightforward instance is to create mini-batches by sampling neighbors (e.g., GraphSAGE (Hamilton et al., 2017) and VR-GCN (Chen et al., 2018)) or to extract subgraphs as training samples (e.g., Cluster-GCN (Chiang et al., 2019) and GraphSAINT (Zeng et al., 2020)).

Original: In addition to sampling-based methods, distributed GCN training has emerged as a promising alternative, as it enables large full graph training of GCNs across multiple accelerators such as GPUs.
Revised: In addition to sampling-based methods, distributed GCN training has emerged as a promising alternative, as it enables large full-graph training of GCNs across multiple accelerators such as GPUs.

Original: The essence of this approach is to separate a giant graph into several small partitions, each of which is able to ﬁt into a single GPU, and then train these  subgraphs locally on GPUs but with indispensable communication.
Revised: This approach ﬁrst partitions a giant graph into multiple small subgraps, each of which is able to ﬁt into a single GPU, and then train these partitioned subgraphs locally on GPUs together with indispensable communication across partitions.

Original: Following this direction, several recent works (Ma et al., 2019; Jia et al., 2020; Tripathy et al., 2020; Thorpe et al., 2021) have been proposed and veriﬁed the great potential of distributed GCN training.
Revised: Following this direction, several recent works (Ma et al., 2019; Jia et al., 2020; Tripathy et al., 2020; Thorpe et al., 2021; Wan et al., 2022) have been proposed and veriﬁed the great potential of distributed GCN training.

Original: P 3 (Gandhi & Iyer, 2021) follows another direction that splits the data along the feature dimension and leverages intra-layer model parallelism for training, which shows superior performance on small models.
Revised: P 3 (Gandhi & Iyer, 2021) follows another direction that splits the data along the feature dimension and leverages intra-layer model parallelism for training, which shows superior performance on small models.

Original: In this work, we propose a new method of distributed GCN training, PipeGCN, which targets achieving a full-graph accuracy with boosted training efﬁciency.
Revised: In this work, we propose a new method for distributed GCN training, PipeGCN, which targets achieving a full-graph accuracy with boosted training efﬁciency.

Original: Our main contributions are:
Revised: Our main contributions are following:

Original: • We ﬁrst analyze two efﬁciency bottlenecks in distributed GCN training:  signiﬁcant communication overhead and frequent synchronization , and then propose a simple-yet-effective technique called PipeGCN to address the above two bottlenecks by pipelining inter-partition communication with intra-partition computation to hide the communication overhead.
Revised: • We ﬁrst analyze two efﬁciency bottlenecks in distributed GCN training: the required signiﬁcant communication overhead and frequent synchronization , and then propose a simple yet effective technique called PipeGCN to address both aforementioned bottlenecks by pipelining inter-partition communication with intra-partition computation to hide the communication overhead.

Original: • We address the challenge raised by PipeGCN,
Revised: • We address the challenge raised by PipeGCN,

Original: i
Revised: i

Original: e.,  staleness in communicated features and feature gradients, by providing a theoretical convergence guarantee of PipeGCN which ﬁnds the convergence rate to be O
Revised: e., the resulting staleness in communicated features and feature gradients ( neither weights nor weight gradients ), by providing a theoretical convergence analysis and showing that PipeGCN’s convergence rate is O

Original: ( T − 23 ) ,
Revised: ( T − 23 ) ,

Original: i
Revised: i

Original: e., close to vanilla distributed GCN training without staleness.
Revised: e., close to vanilla distributed GCN training without staleness.

Original: To the best of our knowledge, this is the ﬁrst work providing theoreticalproof for the convergence  of GCN training with both stale feature and stale feature gradients.
Revised: To the best of our knowledge, we are the ﬁrst to provide a theoretical convergence proof of GCN training with both stale feature and stale feature gradients.

Original: • We further propose a low-overhead smoothing method to further improve PipeGCN’s convergence by reducing the error incurred by the staleness.
Revised: • We further propose a low-overhead smoothing method to further improve PipeGCN’s convergence by reducing the error incurred by the staleness.

Original: • Extensive empirical and ablation studies consistently validate the advantages of PipeGCN over both vanilla distributed GCN training (boosting training throughput by up to 2.2 × while achieving the same or a better accuracy) as well as SOTA full-graph training methods.
Revised: • Extensive empirical and ablation studies consistently validate the advantages of PipeGCN over both vanilla distributed GCN training and those SOTA full-graph training methods (e.g., boosting the training throughput by 1.7 ×∼ 28.5 × while achieving the same or a better accuracy ).

Original:  B ACKGROUND AND R ELATED W ORKS
Revised:  B ACKGROUND AND R ELATED W ORKS

Original: Graph Convolutional Networks.
Revised: Graph Convolutional Networks.

Original: GCNs (Kipf & Welling, 2016) exhibit a powerful learning ability for graph-structured data.

Original: They represent each node in a graph as a feature (embedding) vector and learn the feature vector via a two-step process ( neighbor aggregate and then  update ) for each layer, which can be mathematically described as:
Revised: GCNs represent each node in a graph as a feature (embedding) vector and learn the feature vector via a two-step process ( neighbor aggregation and then node update ) for each layer, which can be mathematically described as:

Original: [Equation]
Revised: [Equation]

Original: where N ( v ) is the neighbor set of node v in the graph, h ( (cid:96) ) u represents the learned embedding vector of node u at the (cid:96) -th layer, z ( (cid:96) ) v is an intermediate aggregated feature calculated by an aggregation function ζ ( (cid:96) ) , andﬁnally φ ( (cid:96) ) is the function for updating the feature of node v .
Revised: where N ( v ) is the neighbor set of node v in the graph, h ( (cid:96) ) v represents the learned embedding vector of node v at the (cid:96) -th layer, z ( (cid:96) ) v is an intermediate aggregated feature calculated by an aggregation function ζ ( (cid:96) ) , and φ ( (cid:96) ) is the function for updating the feature of node v .

Original: The original GCN (Kipf & Welling, 2016) uses a mean aggregator for ζ ( (cid:96) ) and the update function φ ( (cid:96) ) is a singlelayer perceptron σ ( W ( (cid:96) )
Revised: The original GCN (Kipf & Welling, 2016) uses a weighted average aggregator for ζ ( (cid:96) ) and the update function φ ( (cid:96) ) is a single-layer perceptron σ ( W ( (cid:96) )

Original: z ( (cid:96) ) v ) where σ ( · ) is a non-linear activation function and W ( (cid:96) ) is the weight matrix.
Revised: z ( (cid:96) ) v ) where σ ( · ) is a non-linear activation function and W ( (cid:96) ) is a weight matrix.

Original: Another famous GCN instance is GraphSAGE (Hamilton et al., 2017) in which φ ( (cid:96) ) is σ (cid:16) W ( (cid:96) )
Revised: Another famous GCN instance is GraphSAGE (Hamilton et al., 2017) in which φ ( (cid:96) ) is σ (cid:16) W ( (cid:96) )

Original: · CONCAT (cid:16) z ( (cid:96) ) v , h ( (cid:96) − 1) v (cid:17)(cid:17)
Revised: · CONCAT (cid:16) [Equation] ( (cid:96) [Equation]

Revised: [Equation]

Revised: ( (cid:96)

Original: [Equation]
Revised: [Equation]

Revised: (cid:17)(cid:17) .

Original: Distributed Training for GCNs.
Revised: Distributed Training for GCNs.

Original: Real-world graphs may containhundreds of millions of nodes and billions of edges (Hu et al., 2020), for which atraining feasible  approach is to partition it into small subgraphs (to ﬁt  each GPU’s resource), and train them in parallel, during which necessary communication is performed to exchange boundary node features and gradients to satisfy GCNs’ neighbor aggregation (Equ. 1).
Revised: A real-world graph can contain millions of nodes and billions of edges (Hu et al., 2020), for which a feasible training approach is to partition it into small subgraphs (to ﬁt into each GPU’s resource), and train them in parallel, during which necessary communication is performed to exchange boundary node features and gradients to satisfy GCNs’s neighbor aggregation (Equ. 1).

Original: Such  approach is called vanilla partition-parallel training and  illustrated in Fig. 1 (a).
Revised: Such an approach is called vanilla partition-parallel training and is illustrated in Fig.(a).

Original: Following this approach, pioneer works have been proposed recently.
Revised: Following this approach, several works have been proposed recently.

Original: ROC (Jia et al., 2020), NeuGraph (Ma et al., 2019) and AliGraph (Zhu et al., 2019) perform the partition-parallel training but rely on CPU storage for all partitions and repeated swapping of a partial partition to GPUs.
Revised: NeuGraph (Ma et al., 2019), AliGraph (Zhu et al., 2019), and ROC (Jia et al., 2020) perform such partition-parallel training but rely on CPUs for storage for all partitions and repeated swapping of a partial partition to GPUs.

Original: Inevitably, prohibitive CPU-GPU swaps are incurred, plaguing the achievable training efﬁciency.
Revised: Inevitably, prohibitive CPU-GPU swaps are incurred, plaguing the achievable training efﬁciency.

Original: CAGNET (Tripathy et al., 2020) is different in that itfurther splits  node feature vectors into tinysub-vectors, which however demands broadcast of those sub-vectors per node and computing them sequentially, thus requires redundant communication and frequent synchronization.
Revised: CAGNET (Tripathy et al., 2020) is different in that it splits each node feature vector into tiny sub-vectors which are then broadcasted and computed sequentially, thus requiring redundant communication and frequent synchronization.

Original: More recently, P 3 (Gandhi & Iyer, 2021) proposes to split  featuredimension andpartition the ﬁrst layer for parallel training with mitigated communication overhead, but it is based on a strong assumption that the hidden dimensions of a GCN should be considerably smaller than that ofinput features. A concurrent work Dorylus (Thorpe et al., 2021) proposes to build a ﬁne-grain pipeline along each compute operation in GCN training and supports asynchronous usage of stale features. However, staleness of
Revised: Furthermore, P 3 (Gandhi & Iyer, 2021) proposes to split both the feature and the GCN layer for mitigating the communication overhead, but it makes a strong assumption that the hidden dimensions of a GCN should be considerably smaller than that of

Original: [Figure]
Revised: [Figure]

Original: 4 Partition
Revised: [Figure]

Original: Graph
Revised: [Figure]

Original: Inner Node Boundary Node

Original: Communicate Boundary Feature & Grad
Revised: Communicate Boundary Feature & Grad

Original: Pipeline
Revised: Pipeline

Original: Communicate Compute ...
Revised: Communicate Compute ...

Original: Iteration #2 Iteration # 

Revised: Iteration #2 Iteration #

Original: Part...
Revised: Part...

Original: Part...
Revised: Part...

Original: Communicate Compute ...
Revised: Communicate Compute ...

Original: Iteration # 56...
Revised: Iteration # 56...

Original: Communicate Compute
Revised: Communicate Compute

Original: Compute Communicate (b
Revised: Compute Communicate (

Original: [Figure]
Revised: [Figure]

Original: Timelineof vanilla partition
Revised: Timeline

Original: [Figure]
Revised: [Figure]

Original: parallel training (c) PipeGCN
Revised: parallel training (c [Figure]

Original: Part
Revised: Part

Original: Iteration #
Revised: Iteration #

Original: Timeline of (a) 3535...
Revised: Timeline of (a) 3535...

Revised: Timeline of PipeGCN

Revised: Part

Revised: Part

Revised: Part 3 Partition      input features, which restricts the model size.

Revised: A concurrent work Dorylus (Thorpe et al., 2021) adopts a ﬁne-grained pipeline along each compute operation in GCN training and supports asynchronous usage of stale features.

Original: Timeline ofPipeGCN feature gradients is neither analyzed nor considered for convergence proof, let alone error reduction methods for the incurred staleness. 
Revised: Nevertheless, the resulting staleness of feature gradients is neither analyzed nor considered for convergence proof, let alone error reduction methods for the incurred staleness. Method [Table] [Table]

Revised: PipeGCN

Revised: Target

Original: [Table]
Revised: [Table]

Revised: Small Feature

Revised: Large Feature

Revised: Staleness

Revised: Weight Gradients

Revised: Features and

Revised: Feature Gradients

Original: Asynchronous Distributed Training.
Revised: Asynchronous Distributed Training.

Original: Many prior works have been proposed for asynchronous distributed training of DNNs, such as
Revised: Many prior works have been proposed for asynchronous distributed training of DNNs.

Original:  Hogwild!
Revised: Most works (e.g., Hogwild!

Original: (Niu et al., 2011), SSP (Ho et al., 2013), and MXNet (Li et al., 2014). Most are based on the parameter serverarchitecture with multiple workers running asynchronously to hide communication overhead of  each other, at a cost of using stale weight gradients from previous iterations.
Revised: (Niu et al., 2011), SSP (Ho et al., 2013), and MXNet (Li et al., 2014)) rely on a parameter server with multiple workers running asynchronously to hide communication overhead of weights/(weight gradients) among each other, at a cost of using stale weight gradients from previous iterations.

Original: Similarly, other works like Pipe-SGD (Li et al., 2018) pipeline  communication with local computation of each worker, trading staleness of weight gradients for a better training efﬁciency (see details in Tab. 1).
Revised: Other works like Pipe-SGD (Li et al., 2018b) pipeline such communication with local computation of each worker. Another direction is to partition a large model along its layers across multiple GPUs and then stream in small data batches through the layer pipeline, e.g., PipeDream (Harlap et al., 2018) and PipeMare (Yang et al., 2021).

Original: Nonetheless,  these works are for large models with small data, where communication overhead of model weights/gradients are substantial but data feature communications are marginal, if not none.
Revised: Nonetheless, all these works aim at large models with small data, where communication overhead of model weights/weight gradients are substantial but data feature communications are marginal (if not none), thus not well suited for GCNs.

Original: Besides, most asynchronous DNN training focus on convergence with stale weight gradients of models, rather than stale features/feature gradients .
Revised: More importantly, they focus on convergence with stale weight gradients of models, rather than stale features/feature gradients incurred in GCN training.

Original: Another direction is to partition a large model along its layers across multiple GPUs and then stream in the data batch through the layer pipeline, e.g., PipeDream (Harlap et al., 2018) and PipeMare (Yang et al., 2021). However, it is also designed for large models with small data and thus not well suited for GCNs.
Revised: Tab. 1 summarizes the differences.

Original: In a nutshell, little effort has been made to studypipelined training or asynchronous  distributed training of GCNs, where feature communication is the major overhead , let alone  corresponding theoretical convergence proofs.
Revised: In a nutshell, little effort has been made to study asynchronous or pipelined distributed training of GCNs, where feature communication plays the major role, let alone the corresponding theoretical convergence proofs.

Original: GCNs with Stale Features/Feature Gradients.
Revised: GCNs with Stale Features/Feature Gradients.

Original: Several recent works have been proposed to adopt either stale features (Chen et al., 2018;
Revised: Several recent works have been proposed to adopt either stale features (Chen et al., 2018;

Original: Cong et al., 2020) or feature gradients (Cong et al., 2021) in GCN training.
Revised: Cong et al., 2020) or feature gradients (Cong et al., 2021) in single-GPU training of GCNs.

Original: Nevertheless, their convergence analysis considers only one of two kinds of staleness and derives a convergence rate of O ( T − 1 2 ) for pure sampling-based methods.
Revised: Nevertheless, their convergence analysis considers only one of two kinds of staleness and derives a convergence rate of O ( T − 12 ) for pure sampling-based methods.

Original: This is, however, limited in distributed GCN training as its convergence is simultaneously affected by both kinds of staleness .
Revised: This is, however, limited in distributed GCN training as its convergence is simultaneously affected by both kinds of staleness .

Original: PipeGCN proves such convergence with both stale features and feature gradients and offers a better rate of O ( T − 23 ) .
Revised: PipeGCN proves such convergence with both stale features and feature gradients and offers a better rate of O ( T − 23 ) .

Original: Furthermore, none of previous works has studied the errors incurred by staleness which harms the convergence speed, while PipeGCN develops a low-overhead smoothing method to reduce such errors.
Revised: Furthermore, none of previous works has studied the errors incurred by staleness which harms the convergence speed, while PipeGCN develops a low-overhead smoothing method to reduce such errors.

Original:  T HE P ROPOSED P IPE GCN F RAMEWORK
Revised:  T HE P ROPOSED P IPE GCN F RAMEWORK

Original: Overview.
Revised: Overview.

Original: To enable efﬁcient distributed GCN training, we ﬁrst identify the two bottlenecks associated with vanilla partition-parallel training: substantial communication overhead and frequently synchronized communication with computation, and then address them directly by proposing a novel strategy, PipeGCN, to pipeline the communication and computation stages across two adjacent iterations in each partition of distributed GCN training, thus breaking the synchrony and  hiding the communication overhead, as shown in Fig. 1 (c).
Revised: To enable efﬁcient distributed GCN training, we ﬁrst identify the two bottlenecks associated with vanilla partition-parallel training: substantial communication overhead and frequently synchronized communication (see Fig. 1(b)), and then address them directly by proposing a novel strategy, PipeGCN, which pipelines the communication and computation stages across two adjacent iterations in each partition of distributed GCN training for breaking the synchrony and then hiding the communication latency (see Fig. 1(c)).

Original: It is non-trivial to achieve efﬁcient GCN training with such a pipeline method, as staleness is incurred in communicated features/feature gradients and more importantly little effort has been made to study the convergence guarantee of GCN training using stale feature gradients .
Revised: It is non-trivial to achieve efﬁcient GCN training with such a pipeline method, as staleness is incurred in communicated features/feature gradients and [Figure] + [Figure]

Revised: Iteration

Revised: [Figure]

Original: This work takes an initial effort to prove both the theoretical and
Revised: Send + [Figure] from other subgraphs Current Iteration Previous Iteration

Revised: Communicate

Revised: Communicate

Revised: Communicate

Revised: L

Original: [Figure]
Revised: [Figure]

Revised: L2 Forward

Revised: L

Revised: [Figure]

Revised: L

Revised: [Figure]

Revised: [Figure]

Original: {+L
+}
Revised: {+L
+}

Original: empirical convergence of such a pipeline GCN training method, and for the ﬁrst time ﬁnds its convergence rate to be close to that of vanilla training without staleness.
Revised: For [Figure] L [Figure]

Revised: L2 Back

Revised: [Figure]

Revised: L

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: more importantly little effort has been made to study the convergence guarantee of GCN training using stale feature gradients.

Revised: This work takes an initial effort to prove both the theoretical and empirical convergence of such a pipelined GCN training method, and for the ﬁrst time shows its convergence rate to be close to that of vanilla GCN training without staleness.

Original: Furthermore, we propose a low-overhead smoothing method to reduce the errors due to stale features/feature gradients for further improving the convergence.
Revised: Furthermore, we propose a low-overhead smoothing method to reduce the errors due to stale features/feature gradients for further improving the convergence.

Original: 3. B OTTLENECKS IN V ANILLA P ARTITION -P ARALLEL T RAINING
Revised: 3. B OTTLENECKS IN V ANILLA P ARTITION -P ARALLEL T RAINING

Revised: [Table]

Original: Signiﬁcant communication overhead.
Revised: Signiﬁcant communication overhead.

Original: Fig.
Revised: Fig.

Original: 1 (a) illustrates vanilla partition-parallel training , where each partition holds inner nodes that come from the original graph and boundary nodes that come from other subgraphs.
Revised: 1(a) illustrates vanilla partition-parallel training , where each partition holds inner nodes that come from the original graph and boundary nodes that come from other subgraphs.

Original: These boundary nodes are demanded by the neighbor aggregate of GCNs across neighbor partitions. E.g., in Fig.
Revised: These boundary nodes are demanded by the neighbor aggregation of GCNs across neighbor partitions, e.g., in Fig.

Original: 1 (a), node-5 needs nodes-[3,4,6] residing on other partitions for calculating Equ. 1.
Revised: 1(a) node-5 needs nodes-[3,4,6] from other partitions for calculating Equ. 1.

Original: Therefore, it is the features/gradients of boundary nodes that dominate the communication overhead in distributed GCN training .
Revised: Therefore, it is the features/gradients of boundary nodes that dominate the communication overhead in distributed GCN training .

Original: Note that the amount of boundary nodes can be excessive and far exceeds the inner nodes (up to 5 . 5 × in our evaluations), as the boundary nodes are replicated across partitions and scale with more partitions.
Revised: Note that the amount of boundary nodes can be excessive and far exceeds the inner nodes, as the boundary nodes are replicated across partitions and scale with the number of partitions.

Original: Besides the sheer size, communication of boundary nodes occurs for  each layer and for both forward and backward passes , making communication overhead substantial.
Revised: Besides the sheer size, communication of boundary nodes occurs for (1) each layer and (2) both forward and backward passes, making communication overhead substantial.

Original: We evaluate such overhead  in Tab. 5 and ﬁnd communication to be dominant, which is consistent with CAGNET (Tripathy et al., 2020).[Table] [Table]
Revised: We evaluate such overhead 1 in Tab.and ﬁnd communication to be dominant, which is consistent with CAGNET (Tripathy et al., 2020).

Original: Frequently synchronized communication.
Revised: Frequently synchronized communication.

Original: Note that this communication of boundary nodes must be ﬁnished before calculating Equ. 1 and Equ. 2, which  forcesa synchronization between communication and computation and results in a full sequential execution , as shown in Fig. 1 (b).
Revised: The aforementioned communication of boundary nodes must be ﬁnished before calculating Equ. 1 and Equ. 2, which inevitably forces synchronization between communication and computation and requires a fully sequential execution (see Fig. 1(b)).

Original: Thus, for most of training time, each partition is waiting for tedious communication to ﬁnish beforedoing the actual compute work, and this repeats frequently for eachGCN layer and for both forward and backward passes.
Revised: Thus, for most of training time, each partition is waiting for dominant features/gradients communication to ﬁnish before the actual compute, repeated for each layer and for both forward and backward passes.

Original: 3. T HE P ROPOSED P IPE GCN M ETHOD
Revised: 3. T HE P ROPOSED P IPE GCN M ETHOD

Original: Fig.
Revised: Fig.

Original: 1 (c) illustrates the high-level overview of PipeGCN, which pipelines the communicate and compute stages spanning two iterations for each GCN layer .
Revised: 1(c) illustrates the high-level overview of PipeGCN, which pipelines the communicate and compute stages spanning two iterations for each GCN layer.

Original: Fig.
Revised: Fig.

Original: 2 further provides the detailed end-to-end ﬂow, where PipeGCN removes the heavy communication overhead in the vanilla approach by breaking the synchronization between communicate and compute and hiding it with compute of each GCN layer.
Revised: 2 further provides the detailed end-to-end ﬂow, where PipeGCN removes the heavy communication overhead in the vanilla approach by breaking the synchronization between communicate and compute and hiding communicate with compute of each GCN layer.

Original: This is achieved by deferring the communicate to next iteration’s compute (instead of serving the current iteration) such that compute and communicate can run inparallel.
Revised: This is achieved by deferring the communicate to next iteration’s compute (instead of serving the current iteration) such that compute and communicate can run in

Original: Inevitably, staleness is introduced in the deferred communication and results in a mixture usage of fresh inner features/gradients and staled boundary features/gradients.
Revised: [Figure] [Figure] [Figure]

Original: Analytically, PipeGCN is achieved by modifying Equ. 1.
Revised: [Figure] [Figure] [Figure] [Figure]

Original: For instance, when using a mean aggregator, Equ. 1 and its corresponding backward formulation become:
Revised: [Figure] [Figure]

Original:  − 1) W ( (cid:96) )t − 1 ) (cid:46)
Revised: P i H ( (cid:96) − 1) W ( (cid:96) )

Original:  end
Revised: t − 1 ) (cid:46) update inner nodes feature end

Original: J ( L ) V
Revised: J ( L ) V

Original: i ← d Loss ( H ( L ) V i ,Y i ) 
Revised: ← ∂Loss ( H ( L ) V i ,Y i ) ∂H

Original: d H ( L ) Vi for (cid:96) :
Revised: ( L ) V for (cid:96) :

Original: 1 do (cid:46) 
Revised: = do (cid:46) backward pass

Original:  ( (cid:96) )
Revised: G ( (cid:96) )

Original: i (cid:104) Pi
Revised: ← (cid:104) P

Original:  ( (cid:96) − 1) (cid:105) (cid:62) (cid:16) J ( (cid:96) )
Revised: H ( (cid:96) − 1) (cid:105) (cid:62) (cid:16) J ( (cid:96) )

Original: V i ◦ σ (cid:48)
Revised: V i ◦ σ (cid:48)

Original: ( P i  ( (cid:96) −1)
Revised: ( P i H ( (cid:96) −

Revised: [Table]

Original:  ( (cid:96))
Revised: ) W ( (cid:96)

Original: t −
Revised: [Table]

Original:  (cid:17) (cid:46)  if (cid:96) > 1 then
Revised: − 1 ) (cid:17) (cid:46) calculate weight gradient if (cid:96) > 1 then

Original: J ( (cid:96) − 1) ← P (cid:62) i (cid:16) J ( (cid:96) ) 
Revised: J ( (cid:96) − [Table] ) ← P (cid:62) i (cid:16) J ( (cid:96) ) V

Revised: [Table]

Original: V i σ (cid:48)
Revised: ◦ σ (cid:48)

Original: ( P i H ( (cid:96) −1)
Revised: ( P i H ( (cid:96) −

Original:  ) t −
Revised: [Table] ) W ( [Table]

Original: ) t − 1 ] (cid:62)
Revised: [Table] − [Table] ) (cid:17) [ W ( (cid:96) )

Original:  if  then wait until thread (  completes for 
Revised: [Table] − 1 ] (cid:62) (cid:46) calculate feature gradient if t > 1 then wait until thread ( (cid:96) ) b completes for j :

Original:  →  do
Revised: 1 → n do

Original: J ( (cid:96) − 1)S i,j ← J  − 1) S i,j
Revised: J ( (cid:96) − 1) ← J ( (cid:96) − 1) ,j + C ( (cid:96)

Original: ) j (cid:46) accumulate  end end with thread ( (cid:96) )
Revised: (cid:46) accumulate feature gradient end end with thread ( (cid:96) )

Original: b (cid:46) communicate boundary featuregradient in parallel
Revised: b (cid:46) communicate boundary feature in parallel

Original: Send [ J ( (cid:96) − 1) S 1 ,i , · · · , J ( (cid:96) − 1) S n,i ] to partition [1 , · · · , n ] and Receive [ C ( (cid:96) ) 1 , · · · , C ( (cid:96) ) n ] end end
Revised: Send [ J ( (cid:96) − 1) S 1 ,i , · · · , J ( (cid:96) − 1) S n,i ] to partition 1 , · · · , n ] and Receive [ C ( (cid:96) ) 1 , · · · , C ( (cid:96) ) n ] end end

Original: G ← AllReduce ( G [Equation] ) (cid:46) synchronize model gradient
Revised: G ← AllReduce ( G i ) (cid:46) synchronize model gradient

Original: W t ← W t − 1 − ηG (cid:46) update model 34 end 35 return W T 
Revised: W t ← W t − 1 − η · G (cid:46) update model 34 end 35 return W T parallel.

Revised: Inevitably, staleness is introduced in the deferred communication and results in a mixture usage of fresh inner features/gradients and staled boundary features/gradients .

Revised: Analytically, PipeGCN is achieved by modifying Equ. 1.

Revised: For instance, when using a mean aggregator,

Revised: Equ. 1 and its corresponding backward formulation in PipeGCN become:

Original: [Equation]
Revised: [Equation]

Original: where B  is the set of boundary neighbors of v , d v is the degree of node v , and δ ( t,(cid:96) )
Revised: where B ( v ) is node v ’s boundary node set, d v denotes node v ’s degree, and δ ( t,(cid:96) )

Original: x is the gradient approximation of variable x at layer (cid:96) and iteration t .
Revised: h u and δ ( t,(cid:96) ) z v represent the gradient approximation of h u and z v at layer (cid:96) and iteration t , respectively.

Original: Lastly, PipeGCN’s details are shown in Alg. 
Revised: Lastly, the implementation of PipeGCN are outlined in Alg. 1.

Original: 3. P IPE GCN’ S C ONVERGENCE G UARANTEE
Revised: 3. P IPE GCN’ S C ONVERGENCE G UARANTEE

Original: Staleness of communicated boundary features/gradients is the major challenge of PipeGCN, due to itsunknown impact to the convergence. Here we provide convergence analysis under three assumptions: Assumption 3.1. The loss function Loss ( · , · ) is C loss -Lipschitz continuous and L loss -smooth w.r.t.
Revised: As PipeGCN adopts a mixture usage of fresh inner features/gradients and staled boundary features/gradients, its convergence rate is still unknown.

Original: to the input node embedding vector, i.e., | Loss ( h ( L ) , y ) − Loss

Original: ( h (cid:48) ( L ) , y ) | ≤ C loss (cid:107) h ( L ) − h (cid:48) ( L ) (cid:107) 2 and 
Revised: We have proved the convergence of PipeGCN and present the convergence property in the following theorem.

Original: [Table]

Original: 2 where h is the predicted label and y is the correct label vector.

Original: Assumption 3.2.

Original: The activation function σ ( · ) is C σ -Lipschitz continuous and L σ -smooth, i.e.,

Original: [Table]

Original: σ ( z (

Original: [Table] z [Table] Assumption 3.3.
Revised: Theorem 3.1 (Convergence of PipeGCN, informal version) .

Original: For any (cid:96) ∈ [ L ] , the norm of weight matrices, the propagation matrix, and the input feature matrix are bounded: (cid:107) W ( (cid:96) ) [Table] F ≤ B W , (cid:107) P (cid:107) F ≤ B P , (cid:107) X (cid:107) F ≤ B X .
Revised: There exists a constant E such that for any arbitrarily small constant ε > 0 , we can choose a learning rate η = √ εE and number of training iterations T

Original: (This generic assumption is also used in (Chen et al., 2018; Liao et al., 2020; Garg et al., 2020; Cong et al., 2021).) Then we provide the convergence rate of PipeGCN in the following theorem:
Revised: =

Original: Theorem 3.1.

Original: Under Assumptions 3.1, 3.2, and 3.3, we can derive the following by choosing a learning rate η = √ εE and number of training iterations T =

Original: ( L ( θ (1) )
Revised: ( L ( θ (1) )

Original: − L ( θ ∗ ))
Revised: − L ( θ ∗ ))

Original: Eε − 32 : T
Revised: Eε − 32 such that:

Original: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: whereε > 0 is an arbitrarily small constant, L ( · ) is the loss function, θ ( t ) is the parameter vector at iteration t, and [Equation]
Revised: where L ( · ) is the loss function, θ ( t ) and θ ∗ represent the parameter vector at iteration t and the optimal parameter respectively.

Original: Therefore the convergence rate of PipeGCN is O
Revised: Therefore, the convergence rate of PipeGCN is O

Original: ( T − 23 ) , which is better than sampling-based method ( O ( T − 12 ) )
Revised: ( T − 23 ) , which is better than sampling-based method ( O ( T − 12 ) )

Original: (Chen et al., 2018; Cong et al., 2021) and close to full-graph training ( O ( T − 1 ) ) .
Revised: (Chen et al., 2018; Cong et al., 2021) and close to full-graph training ( O ( T − 1 ) ) .

Original: The  detailed proof can be found in Appendix A.
Revised: The formal version of the theorem and our detailed proof can be founded in Appendix A.

Original: 3. T HE P ROPOSED S MOOTHING M ETHOD
Revised: 3. T HE P ROPOSED S MOOTHING M ETHOD

Original: The proposed smoothing method aims at reducing errors incurred by stale features/(feature gradients) at a minimal overhead.
Revised: To further improve the convergence of PipeGCN, we propose a smoothing method to reduce errors incurred by stale features/feature gradients at a minimal overhead.

Original: Here we take the smoothing of feature gradient as an example, but the same formulation also applies tostale features.
Revised: Here we present the smoothing of feature gradients, and the same formulation also applies to features.

Original: To improve the approximate gradients for each feature, ﬂuctuations in feature gradients between adjacent iterations should be reduced.
Revised: To improve the approximate gradients for each feature, ﬂuctuations in feature gradients between adjacent iterations should be reduced.

Original: Therefore, we apply a light-weight moving average for the feature gradients of each boundary node v as follow:
Revised: Therefore, we apply a light-weight moving average to the feature gradients of each boundary node v as follow:

Original: [Equation]
Revised: [Equation]

Original: where ˆ δ ( t,(cid:96) )
Revised: where ˆ δ ( t,(cid:96) )

Original: z v is the smoothed feature gradient at layer (cid:96) and iteration t , and γ is the decay rate.
Revised: z v is the smoothed feature gradient at layer (cid:96) and iteration t , and γ is the decay rate.

Original: To integrate the smoothed feature gradient  into the backward pass,we rewrite Equ. 4  as:
Revised: When integrating this smoothed feature gradient method into the backward pass, Equ. 4 can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: [Table] Such smoothing of stale features and gradients canthus be independently applied to PipeGCN.
Revised: Note that the smoothing of stale features and gradients can be independently applied to PipeGCN.

Original:  E XPERIMENT R ESULTS
Revised:  E XPERIMENT R ESULTS

Original: We evaluate PipeGCN on four large-scale datasets, Reddit (Hamilton et al., 2017), ogbn-products (Hu et al., 2020), Yelp (Zeng et al., 2020), and ogbn-papers100M (Hu et al., 2020).
Revised: We evaluate PipeGCN on four large-scale datasets, Reddit (Hamilton et al., 2017), ogbn-products (Hu et al., 2020), Yelp (Zeng et al., 2020), and ogbn-papers100M (Hu et al., 2020).

Original: More details are provided in Tab. 3.
Revised: More details are provided in Tab. 3.

Original: To ensure robustness and reproducibility, we ﬁx (i.e., do not tune) the hyperparameters and settings for PipeGCN and its variants throughout all experiments.
Revised: To ensure robustness and reproducibility, we ﬁx (i.e., do not tune) the hyperparameters and settings for PipeGCN and its variants throughout all experiments .

Original: To implement partition parallelism (for both vanilla distributed GCN training and PipeGCN), the widely used
Revised: To implement partition parallelism (for both vanilla distributed GCN training and PipeGCN), the widely used

Original: METIS (Karypis & Kumar, 1998) partition algorithm is adopted for graph partition with its objective set to minimize the communication volume.
Revised: METIS (Karypis & Kumar, 1998) partition algorithm is adopted for graph partition with its objective set to minimize the communication volume.

Original: We implement PipeGCN in PyTorch (Paszke et al., 2019) and DGL (Wang et al., 2019).
Revised: We implement PipeGCN in PyTorch (Paszke et al., 2019) and DGL (Wang et al., 2019).

Original: Experiments are conducted on a machine with 10 RTX-2080Ti (11GB),
Revised: Experiments are conducted on a machine with 10 RTX-2080Ti (11GB),

Original: Xeon 6230R@2.10GHz (187GB), and PCIe3x16 connecting CPU-GPU and GPU-GPU.
Revised: Xeon 6230R@2.10GHz (187GB), and PCIe3x16 connecting CPU-GPU and GPU-GPU.

Original: Only for ogbn-papers100M, we use 4 computational nodes (each contains 8 MI60 GPUs, an AMD EPYC
Revised: Only for ogbn-papers100M, we use 4 compute nodes (each contains 8 MI60 GPUs, an AMD EPYC

Original: CPU, and 48 lane PCI 3.0 connecting CPU-GPU and GPU-GPU) networked with 10Gbps Ethernet.
Revised: CPU, and 48 lane PCI 3.0 connecting CPU-GPU and GPU-GPU) networked with 10Gbps Ethernet.

Original: To support full-graph GCN training with the model sizes in Tab. 3, the minimum required partition numbers are 2, 3, 5, 4 for Reddit, ogbn-products, Yelp, and ogbn-papers100M, respectively.[Table] [Figure] [Table]
Revised: To support full-graph GCN training with the model sizes in Tab. 3, the minimum required partition numbers are 2, 3, 5, 32 for Reddit, ogbn-products, Yelp, and ogbn-papers100M, respectively.

Original: [Figure] For convenience, we here name all methods: vanilla partition-parallel training of GCNs ( GCN ),
Revised: For convenience, we here name all methods: vanilla partition-parallel training of GCNs ( GCN ),

Original: PipeGCN with feature gradient smoothing ( PipeGCN-G ), PipeGCN with feature smoothing ( PipeGCN-F ), and PipeGCN with both smoothing ( PipeGCN-GF ).
Revised: PipeGCN with feature gradient smoothing ( PipeGCN-G ), PipeGCN with feature smoothing ( PipeGCN-F ), and PipeGCN with both smoothing ( PipeGCN-GF ).

Original: The default decay rate γ for all smoothing methods is set to 0.95 .
Revised: The default decay rate γ for all smoothing methods is set to 0.95.

Original: 4. I MPROVING T RAINING T HROUGHPUT OVER

Original: [Figure]
Revised: [Figure]

Original: G
Revised: 4. I MPROVING T RAINING T HROUGHPUT OVER F ULL -G RAPH T RAINING M ETHODS

Original: [Figure] T RAINING [Figure]

Original: Fig.
Revised: Fig.

Original: 3 compares the training throughput between PipeGCN and  SOTA full-graph training methods (ROC (Jia et al., 2020) and CAGNET (Tripathy et al., 2020)).
Revised: 3 compares the training throughput between PipeGCN and the SOTA full-graph training methods (ROC (Jia et al., 2020) and CAGNET (Tripathy et al., 2020)).

Original: We observe that both vanilla partition-parallel training (GCN) and PipeGCN greatly outperform ROC and CATNET across different number of partitions, because they avoid both the expensive CPU-GPU swaps (ROC) and the redundant node broadcast (CAGNET).
Revised: We observe that both vanilla partitionparallel training (GCN) and PipeGCN greatly outperform ROC and CAGNET across different number of partitions, because they avoid both the expensive CPU-GPU swaps (ROC) and the redundant node broadcast (CAGNET).

Original: Speciﬁcally, GCN is 3.1 ∼ 6.2 × faster than ROC and 2.1 ∼ 7.9 × faster than CAGNET ( c =2).
Revised: Speciﬁcally, GCN is 3.1 ×∼ 16.4 × faster than ROC and 2.1 ×∼ 10.2 × faster than CAGNET ( c =2).

Original: PipeGCN further improves upon GCN, achieving a throughput improvement of 5.6 ∼ 9.9 × over ROC and 3.9 ∼ 14.7 × over CAGNET ( c =2).
Revised: PipeGCN further improves upon GCN, achieving a throughput improvement of 5.6 ×∼ 28.5 × over ROC and 3.9 ×∼ 17.7 × over CAGNET ( c =2) 2 .

Original: The comparison on more datasets can be found in the Appendix B, which consistently show the advantages of PipeGCN.
Revised: Note that we are not able to compare PipeGCN with NeuGraph (Ma et al., 2019), AliGraph (Zhu et al., 2019), and P 3 (Gandhi & Iyer, 2021) as their code are not publicly available.

Original: Furthermore, we further provide the epoch time breakdown of ROC and CAGNET on Reddit in the Appendix F for understanding where PipeGCN gains signiﬁcant savings over the baseline algorithms.
Revised: Besides, Dorylus (Thorpe et al., 2021) is not comparable, as it is not for regular GPU servers.

Original: Considering the substantial performance gap between ROC/CAGNET and GCN, we focus on comparing GCN with PipeGCN for the reminder of the section.
Revised: Considering the substantial performance gap between ROC/CAGNET and GCN, we focus on comparing GCN with PipeGCN for the reminder of the section .

Original: Note that we are not able to compare with NeuGraph (Ma et al., 2019), AliGraph (Zhu et al., 2019), and

Original: P 3 (Gandhi & Iyer, 2021) as their codes are not open source.

Original: 4. I MPROVING T RAINING T HROUGHPUT WITHOUT C OMPROMISING A CCURACY
Revised: 4. I MPROVING T RAINING T HROUGHPUT WITHOUT C OMPROMISING A CCURACY

Original: We compare the training performance of both test score and training throughput between GCN and
Revised: We compare the training performance of both test score and training throughput between GCN and

Original: PipeGCN in Tab. 4.
Revised: PipeGCN in Tab. 4.

Original: We can see that PipeGCN without smoothing already achieves a comparable test score with the vanilla GCN training on both Reddit and Yelp, and incurrs only a negligible accuracy drop (-0.08% ∼ -0.23%) on ogbn-products, while boosting the training throughput by 1.72 × ∼ 2.16 × across all datasets and different number of partitions, thus validating the effectiveness of PipeGCN.
Revised: We can see that PipeGCN without smoothing already achieves a comparable test score with the vanilla GCN training on both Reddit and Yelp, and incurs only a negligible accuracy drop (-0.08% ∼ -0.23%) on ogbn-products, while boosting the training throughput by 1.72 ×∼ 2.16 × across all datasets and different number of partitions 3 , thus validating the effectiveness of PipeGCN.

Original: With the proposed smoothing method plugged in, PipeGCN-G/F/GF is able to compensate the dropped score of vanilla PipeGCN, achieving an equal or even better test score as/than the vanilla
Revised: With the proposed smoothing method plugged in, PipeGCN-G/F/GF is able to compensate the dropped score of vanilla PipeGCN, achieving an equal or even better test score as/than the vanilla

Original: GCN training (without staleness), e.g., 97.14% vs. 97.11% on Reddit, 79.36% vs. 79.14% on ogbn-products and 65.28% vs. 65.26% on Yelp.
Revised: GCN training (without staleness), e.g., 97.14% vs. 97.11% on Reddit, 79.36% vs. 79.14% on ogbn-products and 65.28% vs. 65.26% on Yelp.

Original: Meanwhile, PipeGCN-G/F/GF enjoys a similar throughput improvement as vanilla PipeGCN, thus validating the negligible overhead of the proposed smoothing method.
Revised: Meanwhile, PipeGCN-G/F/GF enjoys a similar throughput improvement as vanilla PipeGCN, thus validating the negligible overhead of the proposed smoothing method.

Original: Therefore, pipelined transfer of features and gradients greatly improves the training throughput while maintaining the full-graph accuracy .
Revised: Therefore, pipelined transfer of features and gradients greatly improves the training throughput while maintaining the full-graph accuracy .

Original: Note that our distributed GCN training methods consistently achieve higher test scores than SOTA sampling-based methods for GraphSAGE-based models reported in (Zeng et al., 2020) and (Hu et al., 2020), conﬁrming thatthe full-graph trainingtechnique is preferred to obtain better GCN models.
Revised: Note that our distributed GCN training methods consistently achieve higher test scores than SOTA sampling-based methods for GraphSAGE-based models reported in (Zeng et al., 2020) and (Hu et al., 2020), conﬁrming that full-graph training is preferred to obtain better GCN models.

Original: For example, the best sampling-based method achieves a 96.6% accuracy on Reddit (Zeng et al., 2020) while full-graph GCN training achieves 97.1%, and PipeGCN improves the accuracy by 0.28% over sampling-based GraphSAGE models on ogbn-products (Hu et al., 2020).
Revised: For example, the best sampling-based method achieves a 96.6% accuracy on Reddit (Zeng et al., 2020) while full-graph GCN training achieves 97.1%, and PipeGCN improves the accuracy by 0.28% over sampling-based GraphSAGE models on ogbn-products (Hu et al., 2020).

Original: Such an advantage of full-graph training is also validated by recent works (Jia et al., 2020; Tripathy et al., 2020). 4. M AINTAINING C ONVERGENCE S PEED
Revised: This advantage of full-graph training is also validated by recent works (Jia et al., 2020; Tripathy et al., 2020; Liu et al., 2022; Wan et al., 2022).

Original: To understand PipeGCN’s inﬂuence on the convergence speed, we compare the training curve among different methods in Fig.

Original: We observe that the convergence of PipeGCN without smoothing is still comparable with that of the vanilla GCN training, although PipeGCN converges slower at the

Original: [Table]
Revised: [Table]

Original: 1.82 ×

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: 1 × (2.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: 2.16 ×

Original: PipeGCN

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: 2.15 ×

Original: PipeGCN

Original: [Figure]
Revised: [Figure]

Original: 65.26 ± [Figure] 2.15 × PipeGCN 
Revised: variants (PipeGCN*), where PipeGCN and its variants achieve a similar convergence as the vanilla training (without staleness) but are twice as fast in wall-clock time (see Tab. 4).

Original: [Figure] [Figure] [Figure] [Figure]
Revised: 4. M AINTAINING C ONVERGENCE S PEED

Original: 2.11 ×

Original: [Figure] [Figure] [Figure] 1 × (2.25 epochs
Revised: To understand PipeGCN’s inﬂuence on the convergence speed, we compare the training curve among different methods in Fig.

Original: [Figure] [Figure] 1.72 × [Figure]
Revised: We observe that the convergence of PipeGCN without smoothing is still comparable with that of the vanilla GCN training, although PipeGCN converges slower at the early phase of training and then catches up at the later phase, due to the staleness of boundary features/gradients.

Original: [Figure]

Original: [Figure] 1.69 × [Figure] [Figure]
Revised: With the proposed smoothing methods, PipeGCN-G/F boosts the convergence substantially and matches the convergence speed of vanilla GCN training .

Original: 25 ± [Figure] 1.68 × [Figure]
Revised: There is no clear difference between PipeGCN-G and PipeGCN-F. Lastly, with combined smoothing of features and gradients,

Original: 65.26 ± 0. 1.67 × [Figure] variants (PipeGCN*), where PipeGCN and its variants achieve a similar convergence  asthe vanilla  training (without staleness) but are twice as fast in wall-clock time (see Tab. 4).
Revised: PipeGCN-GF can acheive the same or even slightly better convergence speed as vanilla GCN training (e.g., on Reddit) but can overﬁt gradually similar to the vanilla GCN training, which is further investigated in Sec. 4.4.

Original: early phase of training and then catches up at the later phase, due to the staleness of boundary features/gradients.

Original: With the proposed smoothing methods, PipeGCN-G/F boosts the convergence substantially and matches or even outperforms (esp.

Original: at late training phase) the convergence speed of vanilla GCN training .

Original: There is no clear difference between PipeGCN-G and PipeGCN-F. Lastly, with combined smoothing of features and gradients, PipeGCN-GF can acheive the same convergence speed as vanilla GCN training (e.g., on Reddit) but can overﬁt gradually similar to the vanilla GCN training, which is further investigated in Sec. 4.4.

Original: Therefore, PipeGCN maintains the convergence speed w.r.t the number of epochs while reduces the end-to-end training time by around 50% thanks to its boosted training throughput (see Tab. 4).
Revised: Therefore, PipeGCN maintains the convergence speed w.r.t the number of epochs while reduces the end-to-end training time by around 50% thanks to its boosted training throughput (see Tab. 4).

Original: 4. B ENEFIT OFP IPE GCN WITH S TALENESS S MOOTHING 
Revised: 4. B ENEFIT OF S TALENESS S MOOTHING M ETHOD

Original: Error Reduction and Convergence Speedup.
Revised: Error Reduction and Convergence Speedup.

Original: To understand why the proposed smoothing technique (see Sec. 3.4) speeds up convergence, we compare the error incurred by the stale communication between PipeGCN and PipeGCN-G/F.
Revised: To understand why the proposed smoothing technique (Sec. 3.4) speeds up convergence, we compare the error incurred by the stale communication between PipeGCN and PipeGCN-G/F.

Original: The error is calculated as the Frobenius-norm of the gap between the correct gradient/feature and the stale gradient/feature used in PipeGCN training.Fig. [Figure] [Figure]
Revised: The error is calculated as the Frobenius-norm of the gap between the correct gradient/feature and the stale gradient/feature used in PipeGCN training.

Original: [Figure] compares the error at each GCN layer.
Revised: Fig.compares the error at each GCN layer.

Original: We can see that the proposed smoothing technique (PipeGCNG/F) reduces the error of staleness substantially (from the base version of PipeGCN) and this beneﬁt consistently holds across different layers in terms of both feature and gradients errors, validating the effectiveness of our smoothing method and explains its improvement to the convergence speed.
Revised: We can see that the proposed smoothing technique (PipeGCNG/F) reduces the error of staleness substantially (from the base version of PipeGCN) and this beneﬁt consistently holds across different layers in terms of both feature and gradient errors, validating the effectiveness of our smoothing method and explaining its improvement to the convergence speed.

Original: Overﬁtting Reduction.
Revised: Overﬁtting Mitigation.

Original: To understand the effect of staleness smoothing on model overﬁtting, we also evaluate the test-accuracy convergence under different decay rates γ in Fig.
Revised: To understand the effect of staleness smoothing on model overﬁtting, we also evaluate the test-accuracy convergence under different decay rates γ in Fig.

Original: Here ogbn-products is adopted as the study case because the distribution of its test set largely differs from that of its training set.
Revised: Here ogbnproducts is adopted as the study case because the distribution of its test set largely differs from that of its training set.

Original: From Fig.
Revised: From Fig.

Original: 6, we observe that smoothing with a large γ ( 0 .
Revised: 6, we observe that smoothing with a large γ ( 0 .

Original: 95 ) offers a fast convergence, i.e., close to the vanilla GCN training, but overﬁts rapidly.
Revised: 95 ) offers a fast convergence, i.e., close to the vanilla GCN training, but overﬁts rapidly.

Revised: To understand this issue,

Revised: we

Original: To understand this issue, we further provide detailed comparisons of the errors incurred under different γ in Fig.
Revised: [Figure] [Figure] [Figure] further provide detailed comparisons of the errors incurred under different γ in Fig.

Original: We can see that a larger γ enjoys lower approximation errors and makes the gradients/features more stable, thus improving the convergence speed.
Revised: We can see that a larger γ enjoys lower approximation errors and makes the gradients/features more stable, thus improving the convergence speed.

Original: The increased stability on the training set, however, constrains the model from exploring a more general minimum point on the test set, thus leading to overﬁtting as the vanilla GCN training.
Revised: The increased stability on the training set, however, constrains the model from exploring a more general minimum point on the test set, thus leading to overﬁtting as the vanilla GCN training.

Original: In contrast, a small γ ( 0 ∼ 0 . 5 ) mitigates this overﬁtting and achieves a better accuracy (see Fig. 6).
Revised: In contrast, a small γ ( 0 ∼ 0 . 5 ) mitigates this overﬁtting and achieves a better accuracy (see Fig. 6).

Original: But a too-small γ (e.g., 0 ) gives a high error for both stale features and gradients (see Fig. 7), thus suffering from a slower convergence.
Revised: But a too-small γ (e.g., 0 ) gives a high error for both stale features and gradients (see Fig. 7), thus suffering from a slower convergence.

Original: Therefore, a trade-off between convergence speed and achievable optimality exists between different smoothing decay rates, and γ = 0 .
Revised: Therefore, a trade-off between convergence speed and achievable optimality exists between different smoothing decay rates, and γ = 0 .

Original: 5 combines the best of both worlds in this study.
Revised: 5 combines the best of both worlds in this study.

Original: 4.  T RAINING T IME I MPROVEMENT B REAKDOWN
Revised: 4. S CALING L ARGE G RAPH T RAINING OVER M ULTIPLE S ERVERS

Revised: Method

Revised: [Figure]

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: 8s)

Original: To further understand the training time improvement of PipeGCN, we breakdown the epoch time into three parts (intrapartition computation, inter-partition communication, and  reduce  of model gradient) and provide an example in Tab. 5. More results can be found in Appendix D.
Revised: To further test the capability of PipeGCN, we scale up the graph size to ogbn-papers100M and train GCN over multiple GPU servers with 32 GPUs. Tab.shows that even at such a large-scale setting where communication overhead dominates, PipeGCN still reduce communication time by 61%, leading to a total training time reduction of 38% compared to the vanilla GCN baseline 4 .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we propose a new method, PipeGCN, for efﬁcient full-graph GCN training.
Revised: In this work, we propose a new method, PipeGCN, for efﬁcient full-graph GCN training.

Original: PipeGCN pipelines communication with computation in distributed GCN training to hide the substantial communication overhead.
Revised: PipeGCN pipelines communication with computation in distributed GCN training to hide the prohibitive communication overhead.

Original: Furthermore, we take an initial effort to understand the  convergence of
Revised: More importantly, we are the ﬁrst to provide convergence analysis for

Original: GCN training with both stale features and feature gradients, and further propose a  smoothing method to speedup the convergence of vanilla PipeGCN.
Revised: GCN training with both stale features and feature gradients, and further propose a light-weight smoothing method for convergence speedup.

Original: Extensive experiments validate the advantages of
Revised: Extensive experiments validate the advantages of

Original: PipeGCN over both vanilla  training (without staleness) and SOTA full-graph training.
Revised: PipeGCN over both vanilla GCN training (without staleness) and state-of-the-art full-graph training.

Revised:  A CKNOWLEDGEMENT

Revised: The work is supported by the National Science Foundation (NSF) through the MLWiNS program (Award number: 2003137), the CC ∗ Compute program (Award number: 2019007), and the NeTS program (Award number: 1801865)

Original: A C ONVERGENCE P ROOF
Revised: A C ONVERGENCE P ROOF

Original: In this section, we prove the convergence of PipeGCN.
Revised: In this section, we prove the convergence of PipeGCN.

Original: The essential step is  to prove that the bound of the gradient error is controlled by learning rate η .
Revised: Speciﬁcally, we ﬁrst ﬁgure out that when the model is updated via gradient descent, the change of intermediate features and their gradients are bounded by a constant which is proportional to learning rate η under standard assumptions. Based on this, we further demonstrate that the error occurred by the staleness is proportional to η , which guarantees that the gradient error is bounded by ηE where E is deﬁned in Corollary A.10, and thus PipeGCN converges in O ( ε − 32 ) iterations.

Original: A. N OTATIONS 
Revised: A. N OTATIONS AND A SSUMPTIONS

Original: For a given graph G =
Revised: For a given graph G =

Original: ( V , E ) with an adjacency matrix A , feature matrix X , we deﬁne the propagation matrix P as P :=
Revised: ( V , E ) with an adjacency matrix A , feature matrix X , we deﬁne the propagation matrix P as P :=

Original: ˜ D − 1 / 2 ˜
Revised: (cid:101) D − 1 / 2 (cid:101)

Original: A ˜ D − 1 / 2 , where 
Revised: A (cid:101) D − 1 / 2 , where (cid:101)

Original: ˜ A =
Revised: A =

Original: A + I, ˜ D u,u = (cid:80) v ˜
Revised: A + I, (cid:101) D u,u = (cid:80) v (cid:101)

Original: A u,v .
Revised: A u,v .

Original: One GCN layer performs one step of feature propagation (Kipf & Welling, 2016) as formulated below
Revised: One GCN layer performs one step of feature propagation (Kipf & Welling, 2016) as formulated below

Original: [Equation]
Revised: [Equation]

Original: where H ( (cid:96) ) , W ( (cid:96) ) , and Z ( (cid:96) ) denote the embedding matrix, the trainable weight matrix, and the intermediate embedding matrix in the (cid:96) -th layer, respectively, and σ denotes the non-linear  function.
Revised: where H ( (cid:96) ) , W ( (cid:96) ) , and Z ( (cid:96) ) denote the embedding matrix, the trainable weight matrix, and the intermediate embedding matrix in the (cid:96) -th layer, respectively, and σ denotes a non-linear activation function.

Original: For an L -layer GCN, the loss function is denoted by L ( θ ) where θ = vec [ W (1) , W (2) , · · · , W ( L ) ] .
Revised: For an L -layer GCN, the loss function is denoted by L ( θ ) where θ = vec [ W (1) , W (2) , · · · , W ( L ) ] .

Original: We deﬁne the (cid:96) -th layer as a function f ( (cid:96) ) ( · , · ) .
Revised: We deﬁne the (cid:96) -th layer as a function f ( (cid:96) ) ( · , · ) .

Original: [Equation]
Revised: [Equation]

Original: and its gradient w.r.t.
Revised: Its gradient w.r.t.

Original: the input embedding matrix can be represented as
Revised: the input embedding matrix can be represented as

Original: [Equation]
Revised: [Equation]

Original: and its gradient w.r.t.
Revised: and its gradient w.r.t.

Original: the weight can be represented as
Revised: the weight can be represented as

Original: [Equation]
Revised: [Equation]

Original: where M ( (cid:96) )
Revised: where M ( (cid:96) )

Original: =
Revised: =

Original: J ( (cid:96) ) ◦ σ (cid:48)
Revised: J ( (cid:96) ) ◦ σ (cid:48)

Original: ( Z ( (cid:96) ) ) and ◦ denotes Hadamard product.
Revised: ( PH ( (cid:96) − 1) W ( (cid:96) ) ) and ◦ denotes Hadamard product.

Original: For partition-parallel training, we can split P into two parts P = P in + P bd where P in represents interpartition propagation and P bd denotes intra-partition propagation.
Revised: For partition-parallel training, we can split P into two parts P = P in + P bd where P in represents intrapartition propagation and P bd denotes inter-partition propagation.

Original: For PipeGCN, we can represent one GCN layer as below
Revised: For PipeGCN, we can represent one GCN layer as below

Original: [Equation]
Revised: [Equation]

Original: where t is the epoch number and (cid:102) W ( t,(cid:96) ) is the weight at epoch t layer (cid:96) .
Revised: where t is the epoch number and (cid:102) W ( t,(cid:96) ) is the weight at epoch t layer (cid:96) .

Original: We deﬁne the loss function for this setting as (cid:101) L ( (cid:101) θ ( t ) ) where (cid:101) θ ( t ) = vec [ (cid:102) W
Revised: We deﬁne the loss function for this setting as (cid:101) L ( (cid:101) θ ( t ) ) where (cid:101) θ ( t ) = vec [ (cid:102) W

Original: ( t, 1) , (cid:102) W ( t, 2) , · · · , (cid:102) W ( t,L ) ] .
Revised: ( t, 1) , (cid:102) W ( t, 2) , · · · , (cid:102) W ( t,L ) ] .

Original: We can also summarize the layer as a function (cid:101) f ( t,(cid:96) ) ( · , · )
Revised: We can also summarize the layer as a function (cid:101) f ( t,(cid:96) ) ( · , · )

Original: [Equation]
Revised: [Equation]

Original: Note that (cid:101) H ( t − 1 ,(cid:96) − 1) is not a part of the input of (cid:101) f
Revised: Note that (cid:101) H ( t − 1 ,(cid:96) − 1) is not a part of the input of (cid:101) f

Original: ( t,(cid:96) )
Revised: ( t,(cid:96) )

Original: ( · , · ) because it is a constant for the t -th epoch.
Revised: ( · , · ) because it is a constant for the t -th epoch.

Original: The corresponding backward propagation follows the following computation
Revised: The corresponding backward propagation follows the following computation

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: ) [ (cid:102) W  ( t,(cid:96) ) ] (cid:62) + P (cid:62) bd (cid:102) M
Revised: [Table] W [Table] Again, (cid:101) J ( t − 1 ,

Original: ( t − 1 ,(cid:96) ) [ (cid:102) W ( t − 1 ,(cid:96) ) ] (cid:62) ∇ W (cid:101) f ( t,(cid:96) ) ( (cid:101) J ( t,(cid:96) ) , (cid:101) H ( t,(cid:96) − 1) , (cid:102) W ( t,(cid:96) ) ) : = [ P in (cid:101) H ( t,(cid:96) − 1) + P bd (cid:101) H ( t − 1 ,(cid:96) − 1) ] (cid:62) (cid:102) M ( t,(cid:96) )
Revised: [Table]

Original: Again, (cid:101) J ( t − 1 ,(cid:96) ) is not a part of the input of ∇ H (cid:101) f ( t,(cid:96) ) ( · , · , · ) or ∇ W (cid:101) f ( t,(cid:96) ) ( · , · , · ) because it is a constant for epoch t .
Revised: ) is not a part of the input of ∇ H (cid:101) f [Table] [Table] ( · , · , · ) or ∇ W (cid:101) f ( t,(cid:96) ) ( · , · , · ) because it is a constant for epoch t .

Original: Finally, we deﬁne ∇ (cid:101) L ( (cid:101) θ ( t ) )
Revised: Finally, we deﬁne ∇ (cid:101) L ( (cid:101) θ ( t ) )

Original: =
Revised: =

Original: vec [ (cid:101) G ( t, 1) , (cid:101) G ( t, 2) , · · · , (cid:101) G ( t,L ) ] .
Revised: vec [ (cid:101) G ( t, 1) , (cid:101) G ( t, 2) , · · · , (cid:101) G ( t,L ) ] .

Revised: It should be highlighted that the ‘gradient’ ∇

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: It should be highlighted that the ‘gradient’ ∇ H (cid:101) f ( t,(cid:96) ) ( · , · , · ) , ∇ W (cid:101) f ( t,(cid:96) ) ( · , · , · ) and ∇ (cid:101) L ( (cid:101) θ ( t ) ) are not the correct gradient for the corresponding forward process due to the stale communication.
Revised: , ∇ [Table] t,(cid:96) ) ( · , · , · ) and ∇ (cid:101) L ( (cid:101) θ ( t ) ) are not the standard gradient for the corresponding forward process due to the stale communication.

Original: Properties of gradient cannot be directly applied to these variables.
Revised: Properties of gradient cannot be directly applied to these variables.

Revised: Before proceeding our proof, we make the following standard assumptions about the adopted GCN architecture and input graph.

Revised: Assumption A.1.

Revised: The loss function Loss ( · , · ) is C loss -Lipschitz continuous and L loss -smooth w.r.t.

Revised: to the input node embedding vector, i.e., | Loss ( h ( L ) , y ) − Loss

Revised: ( h (cid:48)

Original: The deﬁnitions of other constants C loss , L loss ,  C σ , L σ , B W{+) − h (cid:48) ( L ) (cid:107) 
 and (cid:107)∇ Loss ( h ( L )+} , B P , B X can be found in Assumption
Revised: ( L ) , y ) | ≤ C loss (cid:107) h ( L {+) − h (cid:48) ( L ) (cid:107) 
 and (cid:107)∇ Loss ( h ( L )+} , y ) − ∇ Loss ( h (cid:48) ( L ) , y ) (cid:107) 2 ≤ L loss (cid:107) h ( L ) − h (cid:48)

Revised: ( L ) (cid:107) 2 where h is the predicted label and y is the correct label vector.

Revised: Assumption A.2.

Revised: The activation function σ ( · ) is C σ -Lipschitz continuous and L σ -smooth, i.e.,

Revised: [Equation]

Revised: σ

Revised: [Equation]

Revised: z

Revised: [Equation]

Revised: [Equation]

Revised: − σ ( z

Revised: [Equation]

Revised: ( (cid:96) ) )

Revised: (cid:107) 2 ≤ C σ (cid:107) z

Revised: ( (cid:96) )

Revised: − z (cid:48) ( (cid:96) )

Revised: (cid:107) 2 and (cid:107) σ (cid:48)

Revised: ( z ( (cid:96) ) ) − σ (cid:48)

Revised: ( z (cid:48) ( (cid:96) ) )

Revised: (cid:107) 2 ≤ L σ (cid:107) z

Revised: ( (cid:96) )

Revised: − z (cid:48) ( (cid:96) )

Revised: (cid:107) 2 .

Revised: Assumption A.3.

Revised: For any (cid:96) ∈

Revised: [ L ] , the norm of weight matrices, the propagation matrix, and the input feature matrix are bounded: (cid:107) W ( (cid:96) )

Revised: (cid:107) F ≤ B W , (cid:107) P (cid:107) F ≤ B P , (cid:107) X (cid:107) F ≤ B X .

Revised: (This generic assumption is also used in (Chen et al., 2018;

Revised: Liao et al., 2020; Garg et al., 2020;

Revised: Cong et al., 2021).)

Original: A. B OUNDED M ATRICES AND C HANGES
Revised: A. B OUNDED M ATRICES AND C HANGES

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For any (cid:96) ∈
Revised: For any (cid:96) ∈

Original: [ L ] , the Frobenius norm of node embedding matrices, gradient passing from the (cid:96) -th layer node embeddings to the ( (cid:96) − 1) -th, gradient matrices are bounded, i.e., (cid:107) H ( (cid:96) )
Revised: [ L ] , the Frobenius norm of node embedding matrices, gradient passing from the (cid:96) -th layer node embeddings to the ( (cid:96) − 1) -th, gradient matrices are bounded, i.e., (cid:107) H ( (cid:96) )

Original: (cid:107) F , (cid:107) (cid:101)
Revised: (cid:107) F , (cid:107) (

Original: [Equation]

Original:  ( t [Equation] (cid:96) )
Revised: cid:101) H ( t,(cid:96) )

Original: (cid:107) [Equation]
Revised: (cid:107) F ≤ B H , (cid:107) J

Original: ( (cid:96) )
Revised: ( (cid:96) )

Original: (cid:107) [Equation]
Revised: (cid:107) F , (cid:107) (cid:101) J ( t,(cid:96) ) (cid:107) F ≤ B J , (cid:107) M ( (cid:96) )

Original: (cid:107)  (cid:102) [Equation] ( t
Revised: (cid:107) F , (cid:107) (cid:102) M

Original: [Equation]

Original: (cid:96) )
Revised: ( t,(cid:96) )

Original: (cid:107) [Equation] (cid:107) G ( (cid:96) )
Revised: (cid:107) F ≤ B M , (cid:107) G ( (cid:96) )

Original: (cid:107) F , (cid:107) (cid:101) G ( t,(cid:96) )
Revised: (cid:107) F , (cid:107) (cid:101) G ( t,(cid:96) )

Original: (cid:107) [Equation] where
Revised: (cid:107) F ≤ B G where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof of (cid:107) H ( (cid:96) )
Revised: The proof of (cid:107) H ( (cid:96) )

Original: (cid:107) F ≤ B H and (cid:107) J
Revised: (cid:107) F ≤ B H and (cid:107) J

Original: ( (cid:96) )
Revised: ( (cid:96) )

Original: (cid:107) F ≤ B J can be found in Proposition 2 in (Cong et al., 2021).
Revised: (cid:107) F ≤ B J can be found in Proposition 1 in (Cong et al., 2021) [Equation]

Original: By induction,
Revised: By induction,

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Because the gradient matrices are bounded, the weight change is bounded.
Revised: Because the gradient matrices are bounded, the weight change is bounded.

Original: Corollary A.1.
Revised: Corollary A.2.

Original: For any t , (cid:96) , (cid:107) (cid:102) W ( t,(cid:96) ) − (cid:102) W ( t − 1 ,(cid:96) )
Revised: For any t , (cid:96) , (cid:107) (cid:102) W ( t,(cid:96) ) − (cid:102) W ( t − 1 , [Equation]

Original: (cid:107) F ≤ B ∆ W = ηB G where η is the learning rate.
Revised: [Equation] F ≤ B ∆ W = ηB G where η is the learning rate.

Original: Now we can analyze the changes of intermediate variables.
Revised: Now we can analyze the changes of intermediate variables.

Original: Lemma A.2.
Revised: Lemma A.3.

Original: For any t, (cid:96) , we have (cid:107) (cid:101)Z ( t,(cid:96) ) −
Revised: For any t, (cid:96) , we have (cid:107) (cid:101)

Original: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: t −1 , [Equation] [Equation] F ≤ B ∆ Z ,
Revised: −

Original: [Equation]
Revised: [Equation]

Original: H
Revised: F ≤ B

Original: [Equation]
Revised: [Equation]

Original: t,(cid:96) ) − (cid:101) H ( t − 1 ,(cid:96) ) (cid:107) F ≤ B ∆ H , where B ∆ Z =
Revised: H , where B ∆ Z =

Original: L − 1 (cid:80) i =

Revised: L − 1 (cid:80) i =

Original: C i σ B i [-+
-] P B i W B H B ∆ W and B ∆ H =
Revised: C iσ B i [-+
-]+1 P B iW B H B ∆ W and B ∆ H =

Original: C σ B ∆ Z .
Revised: C σ B ∆ Z .

Original: Proof.
Revised: Proof.

Original: When (cid:96) = 0 , (cid:107) (cid:101)
Revised: When (cid:96) = 0 , (cid:107) (

Original:  H 
Revised: cid:101) H ( t, 0) − (cid:101) H ( t − 1 , 0) (cid:107) F = (cid:107) X − X (cid:107) F = 0 . Now we consider (cid:96) > 0 by induction. (cid:107)

Original: [Equation]
Revised: [Equation]

Original: t
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: 0) − (cid:101) H ( t − 1 , 0) (cid:107) F = (cid:107) X − X (cid:107) F = 0 . Now we consider (cid:96) > 0 by induction.
Revised: [Equation]

Original:  [Equation]
Revised: ) − [Equation]

Original: Then we analyze the bound of s (
Revised: Then we analyze the bound of (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: H

Original: [Equation]
Revised: [Equation]

Original: :=
Revised: t

Original: [Equation]
Revised: [Equation]

Original: [Table] According to A
Revised: −

Original: [Table] [Table]
Revised: [Equation]

Original: W
Revised: W

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] By induction,
Revised: −

Original: [Table]
Revised: [Equation]

Original: H
Revised: H

Original: [Table]
Revised: [Equation] [Equation] − [Equation]

Revised: (cid:102) W

Revised: [Equation]

Revised: t − 1 ,

Revised: [Equation]

Revised: ) (cid:107) F which is denoted by s ( t,(cid:96) ) .

Revised: [Equation]

Revised: [Table]

Original: C
Revised: W

Revised: [Table]

Revised: ≤ B H (cid:107) (cid:102) W

Revised: [Table]

Revised: ,(cid:96) ) −

Revised: [Table]

Revised: W

Revised: [Table]

Revised: [Table]

Revised: F + B W

Revised: [Table]

Revised: H

Revised: [Table]

Revised: −

Revised: [Table]

Revised: −

Revised: [Table]

Revised: H

Revised: [Table]

Revised: −

Revised: [Table]

Revised: −

Revised: [Table]

Revised: (cid:107) F

Revised: According to Corollary A.2, (cid:107) (cid:102) W ( t,(cid:96) )

Revised: − (cid:102) W ( t − 1 ,(cid:96) )

Revised: (cid:107) F ≤ B ∆ W .

Revised: By induction, (cid:107) (cid:101)

Revised: H ( t,(cid:96) − 1) −

Original: [Table]
Revised: [Equation]

Original: +
Revised: i = [Equation] C [Equation]

Revised: B

Original: [Table]
Revised: [Equation]

Original: B iW B
Revised: B iW B

Original: [Table]
Revised: [Equation]

Original: B
Revised: B

Original: [Table]
Revised: [Equation]

Original: W .
Revised: W .

Original: Combining these inequalities,
Revised: Combining these inequalities,

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Plugging it back, we have
Revised: Plugging it back, we have

Original: [Equation]
Revised: [Equation]

Original: Lemma A.3.
Revised: Lemma A.4.

Original: (cid:107) (cid:101) J ( t,(cid:96) )
Revised: [Equation]

Original: − (cid:101) J ( t − 1 ,(cid:96) ) (cid:107) F ≤ B ∆ J where
Revised: − [Equation]

Original: B
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: max
Revised: F ≤ B ∆ J where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For the last layer ( (cid:96) =
Revised: For the last layer ( (cid:96) =

Original: L ) , (cid:107) (cid:101) J ( t,L ) − (cid:101) J ( t − 1 ,L ) (cid:107) F ≤ L loss (cid:107) (cid:101) H ( t
Revised: L ) , (cid:107) (cid:101) J ( t,L ) − (cid:101) J ( t − 1 ,L ) (cid:107) F ≤ L loss (cid:107) (cid:101) H ( t,L ) − (cid:101) H

Original: [Equation]
Revised: [Equation]

Original: L ) − (cid:101) H ( t − 1 ,L ) (cid:107) F ≤ L loss B ∆ H .
Revised: t − [Equation] L ) (cid:107) F ≤ L loss B ∆ H .

Original: For the case of (cid:96) <
Revised: For the case of (cid:96) <

Original: L , we prove the lemma by using induction.
Revised: L , we prove the lemma by using induction

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: We denote

Original: [Table]
Revised: [Table]

Original: [Equation]

Revised: by s ( t

Original: [Table]
Revised: [Table]

Revised: ) and analyze its bound

Original: [Equation] [Table] 
Revised: [Table] [Equation]

Original: According to Corollary A.
Revised: According to Corollary A.2,

Original: [Table] [Equation] [Table]
Revised: [Equation] F ≤ B ∆ W .

Original: For the second term,
Revised: For the second term,

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: According to the smoothness of σ and Lemma A.
Revised: According to the smoothness of σ and Lemma A.3,

Original: [Equation]
Revised: [Equation]

Original: − σ
Revised: σ

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: ≤ L σ B
Revised: t [Equation] − σ [Table]

Original: [Equation]
Revised: [Equation]

Original: Z .
Revised: Z [Table] [Equation] [Equation]

Revised: [Equation]

Revised: F ≤ L σ B ∆ Z .

Original: By induction,
Revised: By induction

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: As a result
Revised: As a result

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: From Equation
Revised: From Equation

Original: [Equation]
Revised: [Equation]

Original: we can also conclude that
Revised: we can also conclude that

Original: Corollary A.2.
Revised: Corollary A.5.

Original: (cid:107) (cid:102) M ( t,(cid:96) ) − (cid:102)M
Revised: [Equation] (cid:102) M ( t,(cid:96) ) − (cid:102)

Revised: M

Original: ( t − 1 ,(cid:96) )
Revised: ( t − 1 ,(cid:96) )

Original: (cid:107) F ≤ B ∆ M with B ∆ M = B J L σ B ∆ Z + C σ B ∆ J .
Revised: (cid:107) F ≤ B ∆ M with B ∆ M = B J L σ B ∆ Z + C σ B ∆ J .

Original: A. B OUNDED F EATURE E RROR AND G RADIENT E RROR
Revised: A. B OUNDED F EATURE E RROR AND G RADIENT E RROR

Revised: In this subsection

Revised: [Equation]

Original: In this subsection, we compare the difference between generic GCN and PipeGCN with the same parameter set, i.e., θ = (cid:101) θ ( t ) . 
Revised: we compare the difference between generic GCN and PipeGCN with the same parameter set, i [Equation] e., θ = (cid:101) θ ( t ) . Lemma

Original: [Table]
Revised: [Equation]

Original: and
Revised: and

Original: [Table]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Table]
Revised: [Equation]

Revised: (cid:107) (cid:101) H ( t,(cid:96) − 1) − H ( (cid:96) − 1) (cid:107) F + B

Original: F + B ∆ H (cid:17)
Revised: ∆ H (cid:17)

Original: By induction, we assume that (cid:107) (cid:101) H ( t,(cid:96) − 1) − H ( (cid:96) − 1) (cid:107) F ≤ B ∆ H (cid:96) − 1 (cid:80) i =1 ( C σ B W B P ) i .
Revised: By induction, we assume that (cid:107) (cid:101) H ( t,(cid:96) − 1) − H ( (cid:96) − 1) (cid:107) F ≤ B ∆ H (cid:96) − 1 (cid:80) i =1 ( C σ B W B P ) i .

Original: Therefore, (cid:107) (cid:101) Z ( t,(cid:96) )
Revised: Therefore, (cid:107) (cid:101) Z ( t,(cid:96) )

Original: − Z ( (cid:96) ) (cid:107) F ≤ B W B P B ∆ H (cid:96) − 1 (cid:88) i
Revised: − Z ( (cid:96) ) (cid:107) F ≤ B W B P B ∆ H (cid:96) − 1 (cid:88) i

Original: =
Revised: =

Original: ( C σ B W B P ) i =
Revised: ( C σ B W B P ) i =

Original: B ∆ H (cid:96) (cid:88) i =
Revised: B ∆ H (cid:96) (cid:88) i =

Original: C i − 1 σ B iW B iP (cid:107) (cid:101) H ( t,(cid:96) )
Revised: C i − 1 σ B iW B iP (cid:107) (cid:101) H ( t,(cid:96) )

Original: − H ( (cid:96) )
Revised: − H ( (cid:96) )

Original: (cid:107) F = (cid:107) σ ( (cid:101) Z ( t,(cid:96) ) )
Revised: (cid:107) F = (cid:107) σ ( (cid:101) Z ( t,(cid:96) ) )

Original: − σ ( Z ( (cid:96) ) )
Revised: − σ ( Z ( (cid:96) ) )

Original: (cid:107) F ≤ C σ (cid:107) (cid:101) Z ( t,(cid:96) )
Revised: (cid:107) F ≤ C σ (cid:107) (cid:101) Z ( t,(cid:96) )

Original: − Z ( (cid:96) ) (cid:107) F ≤ B ∆ H (cid:96) (cid:88) i
Revised: − Z ( (cid:96) ) (cid:107) F ≤ B ∆ H (cid:96) (cid:88) i

Original: =
Revised: =

Original: ( C σ B W B P )
Revised: ( C σ B W B P )

Original: i
Revised: i

Original: Lemma A.5.
Revised: Lemma A.7.

Original: (cid:107) (cid:101) J ( t,(cid:96) )
Revised: (cid:107) (cid:101) J ( t,(cid:96) )

Original: − J ( (cid:96) ) (cid:107) F ≤ E J and (cid:107) (cid:102) M ( t,(cid:96) )
Revised: − J ( (cid:96) ) (cid:107) F ≤ E J and (cid:107) (cid:102) M ( t,(cid:96) )

Original: − M ( (cid:96) ) (cid:107) F ≤ E M with
Revised: − M ( (cid:96) ) (cid:107) F ≤ E M with

Original: E J = max  (cid:96)  ( B P B W C σ )  (cid:96) L loss E H + B P( B W ( B J E Z L σ + B ∆ M ) + B ∆ W B M ) L − 1 (cid:88) i =
Revised: E J = max 2 ≤ (cid:96) ≤ L ( B P B W C σ ) L − (cid:96) L loss E H + B P

Original: ( BP B W C σ ) i E M = C σ E J + L σ B J E Z 
Revised: ( B W ( B J E Z L σ + B ∆ M [Equation]

Original: Proof.
Revised: Proof.

Original: When (cid:96) =
Revised: When (cid:96) =

Original: L , (cid:107) (cid:101) J ( t,L )
Revised: L , (cid:107) (cid:101) J ( t,L )

Original: − J
Revised: − J

Original: ( L ) 
Revised: ( L ) (cid:107) F ≤ L loss E H . For any (cid:96)

Original: [Equation]
Revised: [Equation]

Original: F ≤ L loss E H .

Original: We assume that
Revised: we assume that

Original: [Equation]
Revised: [Equation]

Original: where U = B P ( B W B J E Z L σ + B ∆ W B M + B W B ∆ M ) 
Revised: where U = B P ( B W B J E Z L σ + B ∆ W B M + B W B ∆ M ) .

Original: [Equation]

Original: We prove them by induction as follows
Revised: We prove them by induction as follows

Original: [Equation]
Revised: [Equation]

Original: [Table] Here(cid:107) σ
Revised: Here

Original: [Equation]
Revised: [Equation]

Original: (cid:101) Z ( t,(cid:96) ) )

Original: − σ (cid:48)

Original: ( Z ( (cid:96) ) )

Original: (cid:107) F ≤ L σ E Z .

Original: With Equation 6,
Revised: With Equation 6,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: On the other hand,
Revised: On the other hand

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: =

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: The ﬁrst part is bounded by Equation 7.

Original: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: For the second part,

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: ≤

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation] [Table] The ﬁrst part is bounded by Equation 7. For the second part,
Revised: t

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: ) ] (cid:62)

Original: [Equation]
Revised: [Equation]

Original: [Table] ) [ (cid:102) W ( t,(cid:96) ) ] (cid:62) − (cid:102) M ( t, [Table] ) [ (cid:102) W ( t,
Revised: ≤ B

Original: [Table]
Revised: [Equation]

Original: ) ] (cid:62) (cid:107) F ≤ B ∆ W B M + B W B ∆ M
Revised: W B M + B W B ∆ M

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: [Equation] 1 ,(cid:96) ) ] (cid:62) − (cid:102)
Revised: Lemma A.8. (cid:107)

Original: [Equation]
Revised: [Equation]

Original: ( t,(cid:96) ) [ (cid:102) W ( t,(cid:96) ) ] (cid:62) (cid:107) F
Revised: G

Original: [Equation]
Revised: [Equation]

Original: = [Equation] Lemma A.6.
Revised: [Table]

Original: (cid:107) (cid:101) G ( t,(cid:96) )
Revised: [Equation] G [Equation] E

Original: − G ( (cid:96) ) (cid:107) F ≤ E G where E G = B P ( B H E M + B M E H )
Revised: [Equation] where E G = B P ( B H E M + B M E H )

Original: Proof.
Revised: Proof.

Original: (cid:107) (cid:101) G ( t,(cid:96) )

Original: − G ( (cid:96) ) (cid:107) F = (cid:13)(cid:13)(cid:13)

Original: [ P in (cid:101) H ( t,(cid:96) − 1) + P bd (cid:101)

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: − 1) )

Original: [Table] [Equation] [Table] [Equation]
Revised: (cid:107) F ) + B P B H E M ≤ B M B P ( E H + B ∆ H )

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: + B P B H E M

Original: By summing up from (cid:96) = 1 to (cid:96) =
Revised: By summing up from (cid:96) = 1 to (cid:96) =

Original: L to both sides, we have
Revised: L to both sides, we have

Original: Corollary A.3.
Revised: Corollary A.9.

Original: (cid:107)∇ (cid:101)
Revised: (cid:107)∇ (cid:101)

Original: L ( θ ) − ∇L ( θ ) (cid:107) 2 ≤ E loss := LE G .
Revised: L ( θ ) − ∇L ( θ ) (cid:107) 2 ≤ E loss where E loss = LE G .

Original: According to the derivation of E loss , we observe that E loss contains a factor η .
Revised: According to the derivation of E loss , we observe that E loss contains a factor η .

Original: We can rewrite E loss .
Revised: To simplify the expression of E loss , we assume that B P B W C σ ≤ 12 without loss of generality, and rewrite Corollary A.9 as the following.

Original: Corollary A.4.
Revised: Corollary A.10.

Original: (cid:107)∇ (cid:101)
Revised: (cid:107)∇ [Equation]

Original: L ( θ ) − ∇L ( θ )(cid:107) 2 ≤ ηE where
Revised: L [Equation] θ )

Original: [Equation]
Revised: [Equation]

Original: [-R = C loss (cid:32) 
 L σ S + U 2 m C 2 σ C loss B X L − 1 (cid:88) i = U i (cid:33)-]
Revised: [-R = C loss (cid:32) 
 L σ S + U 2 m C 2 σ C loss B X L − 1 (cid:88) i = U i (cid:33)-]θ )

Original: E = LB P (cid:32) U m B X (cid:32) C 2 σ S
Revised: [Equation] 2 ≤ ηE where

Original: ( C loss + L loss ) + R L (cid:88) i = U i (cid:33) + C 2 σ C loss S (cid:33)
Revised: [Equation]

Original: A. P ROOF OF THE M AIN T HEOREM
Revised: A. P ROOF OF THE M AIN T HEOREM

Original: We ﬁrst introduce a lemma before the proof of our main theorem.
Revised: We ﬁrst introduce a lemma before the proof of our main theorem.

Original: Lemma A.7 (Lemma 1 in (Cong et al., 2021)) .
Revised: Lemma A.11 (Lemma 1 in (Cong et al., 2021)) .

Original: An L -layer GCN is L f -Lipschitz smoothness, i
Revised: An L -layer GCN is L f -Lipschitz smoothness, i.e.,

Original: [Equation]

Original: e.,

Original: [Equation]
Revised: [Equation]

Original: Now we prove the main theorem
Revised: Now we prove the main theorem.

Revised: Theorem A.12 (Convergence of PipeGCN, formal) .

Revised: Under Assumptions A.1, A.2, and A.3, we can derive the following by choosing a learning rate η = √ εE and number of training iterations

Original: [Equation]
Revised: [Equation]

Revised: where E is deﬁned in Corollary A.10, ε > 0 is an arbitrarily small constant, L ( · ) is the loss function, θ ( t ) and θ ∗ represent the parameter vector at iteration t and the optimal parameter respectively.

Original: Proof.
Revised: Proof.

Original: With the smoothness of the model,
Revised: With the smoothness of the model,

Original: [Equation]
Revised: [Equation]

Original: Let δ ( t ) = ∇ (cid:101)
Revised: Let δ ( t ) = ∇ (cid:101)

Original: L ( θ ( t ) )
Revised: L ( θ ( t ) )

Original: − ∇L ( θ ( t ) ) and η ≤ 1 /L f , we have
Revised: − ∇L ( θ ( t ) ) and η ≤ 1 /L f , we have

Original: [Equation]
Revised: [Equation]

Original: From Corollary A.4 we know that (cid:107) δ ( t ) (cid:107) 2 < ηE .
Revised: From Corollary A.10 we know that (cid:107) δ ( t ) (cid:107) 2 < ηE .

Original: After rearranging the terms,(cid:107)∇L
Revised: After rearranging the terms,

Original: [Equation]
Revised: [Equation]

Original: Summing up from t = 1 to T and taking the average,
Revised: Summing up from t = 1 to T and taking the average,

Original: [Equation]
Revised: [Equation]

Original: where θ ∗ is the minimum point of L ( · ) .
Revised: where θ ∗ is the minimum point of L ( · ) .

Original: By taking η = √ εE and T =
Revised: By taking η = √ ε E and T =

Original: ( L ( θ (1) )
Revised: ( L ( θ (1) )

Original: − L ( θ ∗ ))
Revised: − L ( θ ∗ ))

Original: Eε − 32 with an arbitrarily small constant ε > 0 , we have
Revised: Eε − 32 with an arbitrarily small constant ε > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: BI MPROVING T RAINING T HROUGHPUT OVER F ULL -G RAPH T RAINING M ETHODS(A DDITIONAL E XPERIMENTS )
Revised: B T RAINING T IME B REAKDOWN OF F ULL -G RAPH T RAINING M ETHODS

Original: Figure 8 compares the training throughput between PipeGCN and the SOTA full-graph training methods (ROC (Jia et al., 2020) and CAGNET (Tripathy et al., 2020)) on more datasets under that same setting of Figure 3 of the main content.

Original: As can be seen, the advantage of PipeGCN consistently holds , which is similar to Figure 3 of the main content.
Revised: To understand why PipeGCN signiﬁcantly boosts the training throughput over full-graph training methods

Original: [Figure]
Revised: [Table]

Original: We provide the additional convergence curves on Yelp in Figure 9.
Revised: we provide the detailed time breakdown in Tab. 6 using the same model as Tab. 3 (4-layer

Original:  We can see that PipeGCN and its variants maintain the convergence speed w [Figure] r.t the number of epochs while substantially reducing the end-to-end training time
Revised: GraphSAGE, 256 hidden units), in which “GCN” denotes the vanilla partition-parallel training illustrated in Fig. 1(a). We observe that PipeGCN greatly saves communication time.

Original: [Figure]
Revised: [Table]

Original: similar convergence as the vanilla training (without staleness)

Original: but are twice as fast in terms of wall-clock time (see the Throughput improvement in Table 3 of the main content).

Original: D T RAINING T IME I MPROVEMENT B REAKDOWN (A DDITIONAL E XPERIMENTS )
Revised: C T RAINING T IME I MPROVEMENT B REAKDOWN OF P IPE GCN

Original: To understand the training time improvement offered by PipeGCN, we further breakdown the epoch time into three parts (intra-partition computation, inter-partition communication, and reduce for aggregating model gradient) and provide the result in Figure 10.
Revised: To understand the training time improvement offered by PipeGCN, we further breakdown the epoch time into three parts (intra-partition computation, inter-partition communication, and reduce for aggregating model gradient) and provide the result in Fig.

Original: We can observe that: 1) interpartition communication dominates the training time in vanilla distributed training (see GCN); 2)
Revised: We can observe that: 1) interpartition communication dominates the training time in vanilla partition-parallel training (GCN); 2)

Original: PipeGCN (with or without smoothing) greatly hides the communication overhead across different number of partitions and all datasets, e.g., the communication time is hidden completely in 2-partition
Revised: PipeGCN (with or without smoothing) greatly hides the communication overhead across different number of partitions and all datasets, e.g., the communication time is hidden completely in 2-partition

Original: Reddit and almost completely in 3-partition Yelp, thus the substantial reduction in training time; and 3) the proposed smoothing incurs only minimal overhead (i.e., minor difference between PipeGCN and PipeGCN-GF).
Revised: Reddit and almost completely in 3-partition Yelp, thus the substantial reduction in training time; and 3) the proposed smoothing incurs only minimal overhead (i.e., minor difference between PipeGCN and PipeGCN-GF).

Original: Lastly, we also notice that when communication ratio is extremely large (85%+),
Revised: Lastly, we also notice that when communication ratio is extremely large (85%+),

Original: PipeGCN hides communication signiﬁcantly but not completely (e.g., 10-partition ogbn-products), in which case we can employ those compression or quantization techniques from the area of general
Revised: PipeGCN hides communication signiﬁcantly but not completely (e.g., 10-partition ogbn-products), in which case we can employ those compression and quantization techniques (Alistarh et al. Seide et al. Wen et al.

Original:  distributed SGD for further reducing the communication, as the compression is orthogonal to the pipeline method.
Revised: ; Li et al. (2018a); Yu et al. from the area of general distributed SGD for further reducing the communication, as the compression is orthogonal to the pipeline method.

Original: Besides compression, we can also increase the pipeline depth of PipeGCN, e.g., using two iterations of compute to hide one iteration of communication, which is left to our future work.[Figure]
Revised: Besides compression, we can also increase the pipeline depth of PipeGCN, e.g., using two iterations of compute to hide one iteration of communication, which is left to our future work.

Original: [Figure]
Revised: [Figure]

Original: E p o c h t i m e ( s )
Revised: E p o c h t i m e ( s )

Original: G C N
Revised: G C N

Original: G C NP i p e G C N P i p e G C N P i p e G C N - G F
Revised: G C N

Original: P i p e G C NG F
Revised: P i p e G C N

Original: Reddit computation communication reduce 10 Number of partitions 0.

Original: E p o c h t i m e ( s ) G C N G C N P i p e G C N 
Revised: P i p e G C N P

Original: P i p e G C N P

Original: i p e G C N - G
Revised: i p e G C N - G

Original: F
Revised: F

Original: P i p e G C NG F obgn-products computation communication reduce 6 Number of partitions 0.
Revised: P i p e G C NG F obgn-products computation communication reduce 6 Number of partitions 0.

Original: E p o c h t i m e ( s )
Revised: E p o c h t i m e ( s )

Original: G C N
Revised: G C N

Original: G C N
Revised: G C N

Original: P i p e G C N P i p e G C N P
Revised: P i p e G C N P i p e G C N P

Original: i p
Revised: i p

Original: e G C NG F
Revised: e G C NG F

Original: P i p e G C NG F
Revised: P i p e G C NG F

Original: Yelp computation communication reduce
Revised: Yelp computation communication reduce

Revised: D M AINTAINING C ONVERGENCE S PEED (A DDITIONAL E XPERIMENTS )

Revised: We provide the additional convergence curves on Yelp in Fig.

Revised: We can see that PipeGCN and its variants maintain the convergence speed w.r.t the number of epochs while substantially reducing the end-to-end training time

Revised: [Figure]

Original: E P IPE GCN WITH M ULTIPLE C OMPUTATIONAL N ODES
Revised: E S CALING GCN T RAINING OVER M ULTIPLE GPU S ERVERS

Original: We evaluate PipeGCN on Reddit overdifferent number of partitions and multiple nodes (each contains 
Revised: We also scale up PipeGCN training over multiple GPU servers (each contains AMD Radeon Instinct

Original: AMD Radeon Instinct MI60 GPUs, an AMD EPYC 7642 CPU, and 48 lane PCI 3.0 connecting CPU-GPU and GPU-GPU) networked with 10Gbps Ethernet.
Revised: MI60 GPUs, an AMD EPYC 7642 CPU, and 48 lane PCI 3.0 connecting CPU-GPU and GPU-GPU) networked with 10Gbps Ethernet.

Original: Thecorresponding accuracy results of PipeGCN and its variants are summarized below:
Revised: The accuracy results of PipeGCN and its variants are summarized in Tab. 7:

Original: #partitions (#node*#gpus)

Original: PipeGCN

Original: [Table]
Revised: [Table]

Original: Furthermore
Revised: Furthermore

Original: [Table]
Revised: [Table]

Original: we provide the speedup against vanilla distributed GCN training below:
Revised: we provide PipeGCN’s speedup against vanilla partition-parallel training in

Original: [Table]
Revised: [Table]

Original: From the two tables above, we can observe that our PipeGCN family consistently maintains the accuracy of the full-graph training, while improving the throughput by 15% ∼ 66% regardless of the machine settings and number of partitions.
Revised: From the two tables above, we can observe that our PipeGCN family consistently maintains the accuracy of the full-graph training, while improving the throughput by 15% ∼ 66% regardless of the machine settings and number of partitions.

Original: F T RAINING T IME B REAKDOWN C OMPARISON WITH B ASELINES

Original: To understand where PipeGCN gains signiﬁcantly over baseline algorithms, we provide the detailed time breakdown of ROC and CAGNET on Reddit with the same model in Table 3 (4-layer GraphSAGE, 256 hidden units) in Table 8, in which ‘Dist GCN’ is the vanilla distributed GCN training illustrated in Figure 1 (a).

Original: We observe that PipeGCN greatly saves communication time

Original: [Table]

Original: [Table]

Original: G I MPLEMENTATION D ETAILS
Revised: F I MPLEMENTATION D ETAILS

Original: We discuss the details of the effective and efﬁcient implementation of PipeGCN in this section.
Revised: We discuss the details of the effective and efﬁcient implementation of PipeGCN in this section.

Original: First, for parallel communication and computation, a second cudaStream is required for communication besides the default cudaStream for computation.
Revised: First, for parallel communication and computation, a second cudaStream is required for communication besides the default cudaStream for computation.

Original: To also save memory buffers for communication, we batch all communication (e.g., from different layers) into this second cudaStream.
Revised: To also save memory buffers for communication, we batch all communication (e.g., from different layers) into this second cudaStream.

Original: When the popular communication backend, Gloo, is used, we parallelize the CPU-GPU transfer with CPU-CPU transfer.
Revised: When the popular communication backend, Gloo, is used, we parallelize the CPU-GPU transfer with CPU-CPU transfer.

Original: Second, when Dropout layer is used in GCN model, it should be applied after communication.
Revised: Second, when Dropout layer is used in GCN model, it should be applied after communication.

Original: The implementation of the dropout layer for PipeGCN should be considered carefully so that the dropout mask remains consistent for the input tensor and corresponding gradient.
Revised: The implementation of the dropout layer for PipeGCN should be considered carefully so that the dropout mask remains consistent for the input tensor and corresponding gradient.

Original: If the input feature passes through the dropout layer before being communicated, during the backward phase, the dropout mask is changed and the gradient of masked values is involved in the computation, which introduces noise to the calculation of followup gradients.
Revised: If the input feature passes through the dropout layer before being communicated, during the backward phase, the dropout mask is changed and the gradient of masked values is involved in the computation, which introduces noise to the calculation of followup gradients.

Original: As a result, the dropout layer can only be applied after receiving boundary features.
Revised: As a result, the dropout layer can only be applied after receiving boundary features.