Original: P IPE GCN:
Revised: P IPE GCN:

Original: E FFICIENT
Revised: E FFICIENT

Original: F ULL -G RAPH
Revised: F ULL -G RAPH

Original: T RAINING
Revised: T RAINING

Original: OF
Revised: OF

Original: G RAPH
Revised: G RAPH

Original: C ONVOLUTIONAL
Revised: C ONVOLUTIONAL

Original: N ETWORKS
Revised: N ETWORKS

Original: WITH
Revised: WITH

Original: P IPELINED
Revised: P IPELINED

Original: F EATURE C OMMUNICATION
Revised: F EATURE C OMMUNICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph Convolutional Networks (GCNs) is the state-of-the-art method for learning graph-structured data, and training large-scale GCNs requires distributed training across multiple accelerators such that each accelerator is able to hold a partitioned subgraph.
Revised: Graph Convolutional Networks (GCNs) is the state-of-the-art method for learning graph-structured data, and training large-scale GCNs requires distributed training across multiple accelerators such that each accelerator is able to hold a partitioned subgraph.

Original: However, distributed GCN training incurs prohibitive overhead of communicating node features and feature gradients among partitions for every
Revised: However, distributed GCN training incurs prohibitive overhead of communicating node features and feature gradients among partitions for every

Original: GCN layer in each training iteration, limiting the achievable training efﬁciency and model scalability.
Revised: GCN layer in each training iteration, limiting the achievable training efﬁciency and model scalability.

Original: To this end, we propose PipeGCN, a simple-yet-effective scheme that hides the communication overhead by pipelining inter-partition communication with intra-partition computation.
Revised: To this end, we propose PipeGCN, a simple-yet-effective scheme that hides the communication overhead by pipelining inter-partition communication with intra-partition computation.

Original: It is non-trivial to pipeline for efﬁcient GCN training, as communicated node features/gradients will become stale and thus can harm the convergence, negating the pipeline beneﬁt.
Revised: It is non-trivial to pipeline for efﬁcient GCN training, as communicated node features/gradients will become stale and thus can harm the convergence, negating the pipeline beneﬁt.

Original: Notably, little is known regarding the convergence rate of GCN training with both stale features and stale feature gradients.
Revised: Notably, little is known regarding the convergence rate of GCN training with both stale features and stale feature gradients.

Original: This work not only provides a theoretical convergence guarantee but also ﬁnds the convergence rate of PipeGCN to be close to that of the vanilla distributed GCN training without staleness.
Revised: This work not only provides a theoretical convergence guarantee but also ﬁnds the convergence rate of PipeGCN to be close to that of the vanilla distributed GCN training without staleness.

Original: Furthermore, we develop a smoothing method to further improve PipeGCN’s convergence.
Revised: Furthermore, we develop a smoothing method to further improve PipeGCN’s convergence.

Original: Extensive experiments show that PipeGCN can largely boost training throughput (up to 2.2 × ) while achieving the same accuracy as its vanilla counterpart and outperforming existing full-graph training methods.
Revised: Extensive experiments show that PipeGCN can largely boost training throughput (up to 2.2 × ) while achieving the same accuracy as its vanilla counterpart and outperforming existing full-graph training methods.

Original: All code will be released publicly upon acceptance.
Revised: All code will be released publicly upon acceptance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph convolutional networks (GCNs) (Kipf & Welling, 2016) have gained great popularity recently as they demonstrated the state-of-the-art (SOTA) performance in various graph-based learning tasks, including node classiﬁcation (Kipf & Welling, 2016), link prediction (Zhang & Chen, 2018), graph classiﬁcation (Xu et al., 2018), and recommendation systems (Ying et al., 2018).
Revised: Graph convolutional networks (GCNs) (Kipf & Welling, 2016) have gained great popularity recently as they demonstrated the state-of-the-art (SOTA) performance in various graph-based learning tasks, including node classiﬁcation (Kipf & Welling, 2016), link prediction (Zhang & Chen, 2018), graph classiﬁcation (Xu et al., 2018), and recommendation systems (Ying et al., 2018).

Original: The promising performance of GCNs is due to their diverse neighborhood connectivity, which provides a greater applicability to graph-based data than convolutional neural networks (CNNs) that adopt a ﬁxed regular neighborhood structure.
Revised: The promising performance of GCNs is due to their diverse neighborhood connectivity, which provides a greater applicability to graph-based data than convolutional neural networks (CNNs) that adopt a ﬁxed regular neighborhood structure.

Original: In particular, a GCN aggregates all features from the neighbor node set for a given node, the feature of which is then updated via a multi-layer perceptron.
Revised: In particular, a GCN aggregates all features from the neighbor node set for a given node, the feature of which is then updated via a multi-layer perceptron.

Original: Such a two-step process (i.e., aggregate and update ) empowers GCNs to better learn graph structures.
Revised: Such a two-step process (i.e., aggregate and update ) empowers GCNs to better learn graph structures.

Original: However, training GCNs at scale has been a challenging problem, as a prohibitive amount of compute and memory resources are required to train a real-world large-scale graph, let alone exploring deeper and more advanced models.
Revised: However, training GCNs at scale has been a challenging problem, as a prohibitive amount of compute and memory resources are required to train a real-world large-scale graph, let alone exploring deeper and more advanced models.

Original: To tackle this challenge, various sampling-based methods have been proposed to reduce the resource requirement at a cost of incurring feature approximation errors.
Revised: To tackle this challenge, various sampling-based methods have been proposed to reduce the resource requirement at a cost of incurring feature approximation errors.

Original: A straightforward instance is to create mini-batches by sampling neighbors (e.g., GraphSAGE (Hamilton et al., 2017) and VR-GCN (Chen et al., 2018)) or to extract subgraphs as training samples (e.g.,
Revised: A straightforward instance is to create mini-batches by sampling neighbors (e.g., GraphSAGE (Hamilton et al., 2017) and VR-GCN (Chen et al., 2018)) or to extract subgraphs as training samples (e.g.,

Original: Cluster-GCN (Chiang et al., 2019) and GraphSAINT (Zeng et al., 2020)).
Revised: Cluster-GCN (Chiang et al., 2019) and GraphSAINT (Zeng et al., 2020)).

Original: In addition to sampling-based methods, distributed GCN training has emerged as a promising alternative, as it enables large full graph training of GCNs across multiple accelerators such as GPUs.
Revised: In addition to sampling-based methods, distributed GCN training has emerged as a promising alternative, as it enables large full graph training of GCNs across multiple accelerators such as GPUs.

Original: The essence of this approach is to separate a giant graph into several small partitions, each of which is able to ﬁt into a single GPU, and then train these subgraphs locally on GPUs but with indispensable communication.
Revised: The essence of this approach is to separate a giant graph into several small partitions, each of which is able to ﬁt into a single GPU, and then train these subgraphs locally on GPUs but with indispensable communication.

Original: Following this direction, several recent works (Ma et al., 2019; Jia et al., 2020;
Revised: Following this direction, several recent works (Ma et al., 2019; Jia et al., 2020;

Original: Tripathy et al., 2020; Thorpe et al., 2021) have been proposed and veriﬁed the great potential of distributed GCN training.
Revised: Tripathy et al., 2020; Thorpe et al., 2021) have been proposed and veriﬁed the great potential of distributed GCN training.

Original: P 3 (Gandhi & Iyer, 2021) follows another direction that splits the data along the feature dimension and leverages intra-layer model parallelism for training, which shows superior performance on small models.
Revised: P 3 (Gandhi & Iyer, 2021) follows another direction that splits the data along the feature dimension and leverages intra-layer model parallelism for training, which shows superior performance on small models.

Original: In this work, we propose a new method of distributed GCN training, PipeGCN, which targets achieving a full-graph accuracy with boosted training efﬁciency.
Revised: In this work, we propose a new method of distributed GCN training, PipeGCN, which targets achieving a full-graph accuracy with boosted training efﬁciency.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: • We ﬁrst analyze two efﬁciency bottlenecks in distributed GCN training: signiﬁcant communication overhead and frequent synchronization , and then propose a simple-yet-effective technique called PipeGCN to address the above two bottlenecks by pipelining inter-partition communication with intra-partition computation to hide the communication overhead.
Revised: • We ﬁrst analyze two efﬁciency bottlenecks in distributed GCN training: signiﬁcant communication overhead and frequent synchronization , and then propose a simple-yet-effective technique called PipeGCN to address the above two bottlenecks by pipelining inter-partition communication with intra-partition computation to hide the communication overhead.

Original: • We address the challenge raised by PipeGCN,
Revised: • We address the challenge raised by PipeGCN,

Original: i
Revised: i

Original: e., staleness in communicated features and feature gradients, by providing a theoretical convergence guarantee of PipeGCN which ﬁnds the convergence rate to be O
Revised: e., staleness in communicated features and feature gradients, by providing a theoretical convergence guarantee of PipeGCN which ﬁnds the convergence rate to be O

Original: ( T − 23 ) ,
Revised: ( T − 23 ) ,

Original: i
Revised: i

Original: e., close to vanilla distributed GCN training without staleness.
Revised: e., close to vanilla distributed GCN training without staleness.

Original: To the best of our knowledge, this is the ﬁrst work providing theoretical proof for the convergence of GCN training with both stale feature and stale feature gradients.
Revised: To the best of our knowledge, this is the ﬁrst work providing theoretical proof for the convergence of GCN training with both stale feature and stale feature gradients.

Original: • We further propose a low-overhead smoothing method to further improve PipeGCN’s convergence by reducing the error incurred by the staleness.
Revised: • We further propose a low-overhead smoothing method to further improve PipeGCN’s convergence by reducing the error incurred by the staleness.

Original: • Extensive empirical and ablation studies consistently validate the advantages of PipeGCN over both vanilla distributed GCN training (boosting training throughput by up to 2.2 × while achieving the same or a better accuracy) as well as SOTA full-graph training methods.
Revised: • Extensive empirical and ablation studies consistently validate the advantages of PipeGCN over both vanilla distributed GCN training (boosting training throughput by up to 2.2 × while achieving the same or a better accuracy) as well as SOTA full-graph training methods.

Original:  B ACKGROUND AND R ELATED W ORKS
Revised:  B ACKGROUND AND R ELATED W ORKS

Original: Graph Convolutional Networks.
Revised: Graph Convolutional Networks.

Original: GCNs (Kipf & Welling, 2016) exhibit a powerful learning ability for graph-structured data.
Revised: GCNs (Kipf & Welling, 2016) exhibit a powerful learning ability for graph-structured data.

Original: They represent each node in a graph as a feature (embedding) vector and learn the feature vector via a two-step process ( neighbor aggregate and then update ) for each layer, which can be mathematically described as:
Revised: They represent each node in a graph as a feature (embedding) vector and learn the feature vector via a two-step process ( neighbor aggregate and then update ) for each layer, which can be mathematically described as:

Original: [Equation]
Revised: [Equation]

Original: where N ( v ) is the neighbor set of node v in the graph, h ( (cid:96) )
Revised: where N ( v ) is the neighbor set of node v in the graph, h ( (cid:96) )

Original: u represents the learned embedding vector of node u at the (cid:96) -th layer, z ( (cid:96) ) v is an intermediate aggregated feature calculated by an aggregation function ζ ( (cid:96) ) , and ﬁnally φ ( (cid:96) ) is the function for updating the feature of node v .
Revised: u represents the learned embedding vector of node u at the (cid:96) -th layer, z ( (cid:96) ) v is an intermediate aggregated feature calculated by an aggregation function ζ ( (cid:96) ) , and ﬁnally φ ( (cid:96) ) is the function for updating the feature of node v .

Original: The original GCN (Kipf & Welling, 2016) uses a mean aggregator for ζ ( (cid:96) ) and the update function φ ( (cid:96) ) is a singlelayer perceptron σ ( W ( (cid:96) )
Revised: The original GCN (Kipf & Welling, 2016) uses a mean aggregator for ζ ( (cid:96) ) and the update function φ ( (cid:96) ) is a singlelayer perceptron σ ( W ( (cid:96) )

Original: z ( (cid:96) ) v ) where σ ( · ) is a non-linear activation function and W ( (cid:96) ) is the weight matrix.
Revised: z ( (cid:96) ) v ) where σ ( · ) is a non-linear activation function and W ( (cid:96) ) is the weight matrix.

Original: Another famous GCN instance is GraphSAGE (Hamilton et al., 2017) in which φ ( (cid:96) ) is σ [Equation] [Equation]
Revised: Another famous GCN instance is GraphSAGE (Hamilton et al., 2017) in which φ ( (cid:96) ) is σ (cid:16) W ( (cid:96) )

Original:  CONCAT 
Revised: · CONCAT (cid:16) z ( (cid:96) ) v , h ( (cid:96) − 1) v (cid:17)(cid:17)

Original: [Equation]
Revised: [Equation]

Original: Distributed Training for GCNs.
Revised: Distributed Training for GCNs.

Original: Real-world graphs may contain hundreds of millions of nodes and billions of edges (Hu et al., 2020), for which a training feasible approach is to partition it into small subgraphs (to ﬁt each GPU’s resource), and train them in parallel, during which necessary communication is performed to exchange boundary node features and gradients to satisfy GCNs’ neighbor aggregation (Equ. 1).
Revised: Real-world graphs may contain hundreds of millions of nodes and billions of edges (Hu et al., 2020), for which a training feasible approach is to partition it into small subgraphs (to ﬁt each GPU’s resource), and train them in parallel, during which necessary communication is performed to exchange boundary node features and gradients to satisfy GCNs’ neighbor aggregation (Equ. 1).

Original: Such approach is called vanilla partition-parallel training and illustrated in
Revised: Such approach is called vanilla partition-parallel training and illustrated in

Original: Fig.
Revised: Fig.

Original: 1 (a).
Revised: 1 (a).

Original: Following this approach, pioneer works have been proposed recently.
Revised: Following this approach, pioneer works have been proposed recently.

Original: ROC (Jia et al., 2020), NeuGraph (Ma et al., 2019) and AliGraph (Zhu et al., 2019) perform the partition-parallel training but rely on CPU storage for all partitions and repeated swapping of a partial partition to GPUs.
Revised: ROC (Jia et al., 2020), NeuGraph (Ma et al., 2019) and AliGraph (Zhu et al., 2019) perform the partition-parallel training but rely on CPU storage for all partitions and repeated swapping of a partial partition to GPUs.

Original: Inevitably, prohibitive CPU-GPU swaps are incurred, plaguing the achievable training efﬁciency.
Revised: Inevitably, prohibitive CPU-GPU swaps are incurred, plaguing the achievable training efﬁciency.

Original: CAGNET (Tripathy et al., 2020) is different in that it further splits node feature vectors into tiny sub-vectors, which however demands broadcast of those sub-vectors per node and computing them sequentially, thus requires redundant communication and frequent synchronization.
Revised: CAGNET (Tripathy et al., 2020) is different in that it further splits node feature vectors into tiny sub-vectors, which however demands broadcast of those sub-vectors per node and computing them sequentially, thus requires redundant communication and frequent synchronization.

Original: More recently,
Revised: More recently,

Original: P 3 (Gandhi & Iyer, 2021) proposes to split feature dimension and partition the ﬁrst layer for parallel training with mitigated communication overhead, but it is based on a strong assumption that the hidden dimensions of a GCN should be considerably smaller than that of input features.
Revised: P 3 (Gandhi & Iyer, 2021) proposes to split feature dimension and partition the ﬁrst layer for parallel training with mitigated communication overhead, but it is based on a strong assumption that the hidden dimensions of a GCN should be considerably smaller than that of input features.

Original: A concurrent work Dorylus (Thorpe et al., 2021) proposes to build a ﬁne-grain pipeline along each compute operation in GCN training and supports asynchronous usage of stale features.
Revised: A concurrent work Dorylus (Thorpe et al., 2021) proposes to build a ﬁne-grain pipeline along each compute operation in GCN training and supports asynchronous usage of stale features.

Original: However, staleness of
Revised: However, staleness of

Original: [Figure]
Revised: [Figure]

Original: 4 Partition
Revised: 4 Partition

Original: Graph
Revised: Graph

Original: Inner Node Boundary Node
Revised: Inner Node Boundary Node

Original: Communicate Boundary Feature & Grad
Revised: Communicate Boundary Feature & Grad

Original: Pipeline
Revised: Pipeline

Original: Communicate Compute ...
Revised: Communicate Compute ...

Original: Iteration #2 Iteration # 

Revised: Iteration #2 Iteration #

Original: Part...
Revised: Part...

Original: Part...
Revised: Part...

Original: Communicate Compute ...
Revised: Communicate Compute ...

Original: Iteration # 56...
Revised: Iteration # 56...

Original: Communicate Compute
Revised: Communicate Compute

Original: Compute Communicate (b
Revised: Compute Communicate (b

Original: [Figure]
Revised: [Figure]

Original: Timeline of vanilla partition
Revised: Timeline of vanilla partition

Original: [Figure]
Revised: [Figure]

Original: parallel training (c) PipeGCN
Revised: parallel training (c) PipeGCN

Original: Part
Revised: Part

Original: Iteration #
Revised: Iteration #

Original: Timeline of (a) 3535...
Revised: Timeline of (a) 3535...

Original: Timeline of PipeGCN feature gradients is neither analyzed nor considered for convergence proof, let alone error reduction methods for the incurred staleness.
Revised: Timeline of PipeGCN feature gradients is neither analyzed nor considered for convergence proof, let alone error reduction methods for the incurred staleness.

Original: [Table]
Revised: [Table]

Original: Asynchronous Distributed Training.
Revised: Asynchronous Distributed Training.

Original: Many prior works have been proposed for asynchronous distributed training of DNNs, such as
Revised: Many prior works have been proposed for asynchronous distributed training of DNNs, such as

Original: Hogwild!
Revised: Hogwild!

Original: (Niu et al., 2011), SSP (Ho et al., 2013), and MXNet (Li et al., 2014).
Revised: (Niu et al., 2011), SSP (Ho et al., 2013), and MXNet (Li et al., 2014).

Original: Most are based on the parameter server architecture with multiple workers running asynchronously to hide communication overhead of each other, at a cost of using stale weight gradients from previous iterations.
Revised: Most are based on the parameter server architecture with multiple workers running asynchronously to hide communication overhead of each other, at a cost of using stale weight gradients from previous iterations.

Original: Similarly, other works like
Revised: Similarly, other works like

Original: Pipe-SGD (Li et al., 2018) pipeline communication with local computation of each worker, trading staleness of weight gradients for a better training efﬁciency (see details in Tab. 1).
Revised: Pipe-SGD (Li et al., 2018) pipeline communication with local computation of each worker, trading staleness of weight gradients for a better training efﬁciency (see details in Tab. 1).

Original: Nonetheless, these works are for large models with small data, where communication overhead of model weights/gradients are substantial but data feature communications are marginal, if not none.
Revised: Nonetheless, these works are for large models with small data, where communication overhead of model weights/gradients are substantial but data feature communications are marginal, if not none.

Original: Besides, most asynchronous DNN training focus on convergence with stale weight gradients of models, rather than stale features/feature gradients .
Revised: Besides, most asynchronous DNN training focus on convergence with stale weight gradients of models, rather than stale features/feature gradients .

Original: Another direction is to partition a large model along its layers across multiple GPUs and then stream in the data batch through the layer pipeline, e.g., PipeDream (Harlap et al., 2018) and PipeMare (Yang et al., 2021).
Revised: Another direction is to partition a large model along its layers across multiple GPUs and then stream in the data batch through the layer pipeline, e.g., PipeDream (Harlap et al., 2018) and PipeMare (Yang et al., 2021).

Original: However, it is also designed for large models with small data and thus not well suited for GCNs.
Revised: However, it is also designed for large models with small data and thus not well suited for GCNs.

Original: In a nutshell, little effort has been made to study pipelined training or asynchronous distributed training of GCNs, where feature communication is the major overhead , let alone corresponding theoretical convergence proofs.
Revised: In a nutshell, little effort has been made to study pipelined training or asynchronous distributed training of GCNs, where feature communication is the major overhead , let alone corresponding theoretical convergence proofs.

Original: GCNs with Stale Features/Feature Gradients.
Revised: GCNs with Stale Features/Feature Gradients.

Original: Several recent works have been proposed to adopt either stale features (Chen et al., 2018;
Revised: Several recent works have been proposed to adopt either stale features (Chen et al., 2018;

Original: Cong et al., 2020) or feature gradients (Cong et al., 2021) in
Revised: Cong et al., 2020) or feature gradients (Cong et al., 2021) in

Original: GCN training.
Revised: GCN training.

Original: Nevertheless, their convergence analysis considers only one of two kinds of staleness and derives a convergence rate of O ( T − 1 2 ) for pure sampling-based methods.
Revised: Nevertheless, their convergence analysis considers only one of two kinds of staleness and derives a convergence rate of O ( T − 1 2 ) for pure sampling-based methods.

Original: This is, however, limited in distributed GCN training as its convergence is simultaneously affected by both kinds of staleness .
Revised: This is, however, limited in distributed GCN training as its convergence is simultaneously affected by both kinds of staleness .

Original: PipeGCN proves such convergence with both stale features and feature gradients and offers a better rate of O ( T − 23 ) .
Revised: PipeGCN proves such convergence with both stale features and feature gradients and offers a better rate of O ( T − 23 ) .

Original: Furthermore, none of previous works has studied the errors incurred by staleness which harms the convergence speed, while PipeGCN develops a low-overhead smoothing method to reduce such errors.
Revised: Furthermore, none of previous works has studied the errors incurred by staleness which harms the convergence speed, while PipeGCN develops a low-overhead smoothing method to reduce such errors.

Original:  T HE P ROPOSED P IPE GCN F RAMEWORK
Revised:  T HE P ROPOSED P IPE GCN F RAMEWORK

Original: Overview.
Revised: Overview.

Original: To enable efﬁcient distributed GCN training, we ﬁrst identify the two bottlenecks associated with vanilla partition-parallel training: substantial communication overhead and frequently synchronized communication with computation, and then address them directly by proposing a novel strategy, PipeGCN, to pipeline the communication and computation stages across two adjacent iterations in each partition of distributed GCN training, thus breaking the synchrony and hiding the communication overhead, as shown in Fig.
Revised: To enable efﬁcient distributed GCN training, we ﬁrst identify the two bottlenecks associated with vanilla partition-parallel training: substantial communication overhead and frequently synchronized communication with computation, and then address them directly by proposing a novel strategy, PipeGCN, to pipeline the communication and computation stages across two adjacent iterations in each partition of distributed GCN training, thus breaking the synchrony and hiding the communication overhead, as shown in Fig.

Original: 1 (c).
Revised: 1 (c).

Original: It is non-trivial to achieve efﬁcient GCN training with such a pipeline method, as staleness is incurred in communicated features/feature gradients and more importantly little effort has been made to study the convergence guarantee of GCN training using stale feature gradients .
Revised: It is non-trivial to achieve efﬁcient GCN training with such a pipeline method, as staleness is incurred in communicated features/feature gradients and more importantly little effort has been made to study the convergence guarantee of GCN training using stale feature gradients .

Original: This work takes an initial effort to prove both the theoretical and
Revised: This work takes an initial effort to prove both the theoretical and

Original: [Figure]
Revised: [Figure]

Original: L

Original: [Figure]

Original: From

Original: [Figure]

Original: +

Original: [Figure]

Original: Iteration

Original: [Figure]

Original: +

Original: Received from other subgraphs

Original: Current Iteration Previous Iteration

Original: Communicate

Original: Communicate

Original: Communicate

Original: L

Original: [Figure]

Original: L

Original: [Figure]

Original: L

Original: [Figure]

Original: [-L1 Backward Update ... L
-]
Revised: [-L1 Backward Update ... L
-]

Original: For.

Original: [-L
-]
Revised: [-L
-]

Original: For.

Original: L

Original: [Figure]

Original: L

Original: [Figure]

Original: Up. ...

Original: [Figure] Communicate [Figure] empirical convergence of such a pipeline GCN training method, and for the ﬁrst time ﬁnds its convergence rate to be close to that of vanilla training without staleness.
Revised: empirical convergence of such a pipeline GCN training method, and for the ﬁrst time ﬁnds its convergence rate to be close to that of vanilla training without staleness.

Original: Furthermore, we propose a low-overhead smoothing method to reduce the errors due to stale features/feature gradients for further improving the convergence.
Revised: Furthermore, we propose a low-overhead smoothing method to reduce the errors due to stale features/feature gradients for further improving the convergence.

Original: 3. B OTTLENECKS IN V ANILLA P ARTITION -P ARALLEL T RAINING
Revised: 3. B OTTLENECKS IN V ANILLA P ARTITION -P ARALLEL T RAINING

Original: Signiﬁcant communication overhead.
Revised: Signiﬁcant communication overhead.

Original: Fig.
Revised: Fig.

Original: 1 (a) illustrates vanilla partition-parallel training , where each partition holds inner nodes that come from the original graph and boundary nodes that come from other subgraphs.
Revised: 1 (a) illustrates vanilla partition-parallel training , where each partition holds inner nodes that come from the original graph and boundary nodes that come from other subgraphs.

Original: These boundary nodes are demanded by the neighbor aggregate of GCNs across neighbor partitions.
Revised: These boundary nodes are demanded by the neighbor aggregate of GCNs across neighbor partitions.

Original: E.g., in Fig.
Revised: E.g., in Fig.

Original: 1 (a), node-5 needs nodes-[3,4,6] residing on other partitions for calculating Equ. 1.
Revised: 1 (a), node-5 needs nodes-[3,4,6] residing on other partitions for calculating Equ. 1.

Original: Therefore, it is the features/gradients of boundary nodes that dominate the communication overhead in distributed GCN training .
Revised: Therefore, it is the features/gradients of boundary nodes that dominate the communication overhead in distributed GCN training .

Original: Note that the amount of boundary nodes can be excessive and far exceeds the inner nodes (up to 5 .
Revised: Note that the amount of boundary nodes can be excessive and far exceeds the inner nodes (up to 5 .

Original: 5 × in our evaluations), as the boundary nodes are replicated across partitions and scale with more partitions.
Revised: 5 × in our evaluations), as the boundary nodes are replicated across partitions and scale with more partitions.

Original: Besides the sheer size, communication of boundary nodes occurs for each layer and for both forward and backward passes , making communication overhead substantial.
Revised: Besides the sheer size, communication of boundary nodes occurs for each layer and for both forward and backward passes , making communication overhead substantial.

Original: We evaluate such overhead in Tab.
Revised: We evaluate such overhead in Tab.

Original: 5 and ﬁnd communication to be dominant, which is consistent with CAGNET (Tripathy et al., 2020).
Revised: 5 and ﬁnd communication to be dominant, which is consistent with CAGNET (Tripathy et al., 2020).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Frequently synchronized communication.
Revised: Frequently synchronized communication.

Original: Note that this communication of boundary nodes must be ﬁnished before calculating Equ. 1 and Equ. 2, which forces a synchronization between communication and computation and results in a full sequential execution , as shown in Fig.
Revised: Note that this communication of boundary nodes must be ﬁnished before calculating Equ. 1 and Equ. 2, which forces a synchronization between communication and computation and results in a full sequential execution , as shown in Fig.

Original: 1 (b).
Revised: 1 (b).

Original: Thus, for most of training time, each partition is waiting for tedious communication to ﬁnish before doing the actual compute work, and this repeats frequently for each GCN layer and for both forward and backward passes.
Revised: Thus, for most of training time, each partition is waiting for tedious communication to ﬁnish before doing the actual compute work, and this repeats frequently for each GCN layer and for both forward and backward passes.

Original: 3. T HE P ROPOSED P IPE GCN M ETHOD
Revised: 3. T HE P ROPOSED P IPE GCN M ETHOD

Original: Fig.
Revised: Fig.

Original: 1 (c) illustrates the high-level overview of PipeGCN, which pipelines the communicate and compute stages spanning two iterations for each GCN layer .
Revised: 1 (c) illustrates the high-level overview of PipeGCN, which pipelines the communicate and compute stages spanning two iterations for each GCN layer .

Original: Fig.
Revised: Fig.

Original: 2 further provides the detailed end-to-end ﬂow, where PipeGCN removes the heavy communication overhead in the vanilla approach by breaking the synchronization between communicate and compute and hiding it with compute of each GCN layer.
Revised: 2 further provides the detailed end-to-end ﬂow, where PipeGCN removes the heavy communication overhead in the vanilla approach by breaking the synchronization between communicate and compute and hiding it with compute of each GCN layer.

Original: This is achieved by deferring the communicate to next iteration’s compute (instead of serving the current iteration) such that compute and communicate can run in parallel.
Revised: This is achieved by deferring the communicate to next iteration’s compute (instead of serving the current iteration) such that compute and communicate can run in parallel.

Original: Inevitably, staleness is introduced in the deferred communication and results in a mixture usage of fresh inner features/gradients and staled boundary features/gradients.
Revised: Inevitably, staleness is introduced in the deferred communication and results in a mixture usage of fresh inner features/gradients and staled boundary features/gradients.

Original: Analytically, PipeGCN is achieved by modifying Equ. 1.
Revised: Analytically, PipeGCN is achieved by modifying Equ. 1.

Original: For instance, when using a mean aggregator,
Revised: For instance, when using a mean aggregator,

Original: Equ. 1 and its corresponding backward formulation become:
Revised: Equ. 1 and its corresponding backward formulation become:

Original: − 1) W ( (cid:96) )
Revised: − 1) W ( (cid:96) )

Original: t − 1 ) (cid:46)
Revised: t − 1 ) (cid:46)

Original: end
Revised: end

Original: J ( L ) V
Revised: J ( L ) V

Original: i ← d Loss ( H ( L ) V
Revised: i ← d Loss ( H ( L ) V

Original: i ,Y i )
Revised: i ,Y i )

Original: d H ( L ) V i for (cid:96) :
Revised: d H ( L ) V i for (cid:96) :

Original: 1 do (cid:46)
Revised: 1 do (cid:46)

Original: ( (cid:96) )
Revised: ( (cid:96) )

Original: i
Revised: i

Original: (cid:104) P
Revised: (cid:104) P

Original: i
Revised: i

Original: ( (cid:96) − 1) (cid:105) (cid:62) (cid:16)
Revised: ( (cid:96) − 1) (cid:105) (cid:62) (cid:16)

Original: J
Revised: J

Original: ( (cid:96) )
Revised: ( (cid:96) )

Original: V i ◦ σ (cid:48)
Revised: V i ◦ σ (cid:48)

Original: ( P i
Revised: ( P i

Original: ( (cid:96) − 1)
Revised: ( (cid:96) − 1)

Original: ( (cid:96) )
Revised: ( (cid:96) )

Original: t −
Revised: t −

Original: (cid:17) (cid:46)
Revised: (cid:17) (cid:46)

Original: if (cid:96) > 1 then
Revised: if (cid:96) > 1 then

Original: J ( (cid:96) − 1) ← P (cid:62) i (cid:16) J ( (cid:96) )
Revised: J ( (cid:96) − 1) ← P (cid:62) i (cid:16) J ( (cid:96) )

Original: V i σ (cid:48)
Revised: V i σ (cid:48)

Original: ( P i H ( (cid:96) − 1)
Revised: ( P i H ( (cid:96) − 1)

Original: ) t −
Revised: ) t −

Original: ) t − 1 ] (cid:62)
Revised: ) t − 1 ] (cid:62)

Original: if then wait until thread ( completes for
Revised: if then wait until thread ( completes for

Original: → do
Revised: → do

Original: J ( (cid:96) − 1) S i,j ← J
Revised: J ( (cid:96) − 1) S i,j ← J

Original: − 1) S i,j
Revised: − 1) S i,j

Original: ) j (cid:46) accumulate
Revised: ) j (cid:46) accumulate

Original: end end with thread ( (cid:96) )
Revised: end end with thread ( (cid:96) )

Original: b (cid:46) communicate boundary feature gradient in parallel
Revised: b (cid:46) communicate boundary feature gradient in parallel

Original: Send [ J ( (cid:96) − 1) S 1 ,i , · · · , J ( (cid:96) − 1) S n,i ] to partition [1 , · · · , n ] and Receive [ C ( (cid:96) ) 1 , · · · , C ( (cid:96) ) n ] end end
Revised: Send [ J ( (cid:96) − 1) S 1 ,i , · · · , J ( (cid:96) − 1) S n,i ] to partition [1 , · · · , n ] and Receive [ C ( (cid:96) ) 1 , · · · , C ( (cid:96) ) n ] end end

Original: G ← AllReduce ( G
Revised: G ← AllReduce ( G

Original: [Equation]
Revised: [Equation]

Original: ) (cid:46) synchronize model gradient
Revised: ) (cid:46) synchronize model gradient

Original: W t ← W t − 1 − ηG (cid:46) update model 34 end 35 return W T
Revised: W t ← W t − 1 − ηG (cid:46) update model 34 end 35 return W T

Original: [Equation]
Revised: [Equation]

Original: where B is the set of boundary neighbors of v , d v is the degree of node v , and δ ( t,(cid:96) )
Revised: where B is the set of boundary neighbors of v , d v is the degree of node v , and δ ( t,(cid:96) )

Original: x is the gradient approximation of variable x at layer (cid:96) and iteration t .
Revised: x is the gradient approximation of variable x at layer (cid:96) and iteration t .

Original: Lastly, PipeGCN’s details are shown in Alg.
Revised: Lastly, PipeGCN’s details are shown in Alg.

Original: 3. P IPE GCN’ S C ONVERGENCE G UARANTEE
Revised: 3. P IPE GCN’ S C ONVERGENCE G UARANTEE

Original: Staleness of communicated boundary features/gradients is the major challenge of PipeGCN, due to its unknown impact to the convergence.
Revised: Staleness of communicated boundary features/gradients is the major challenge of PipeGCN, due to its unknown impact to the convergence.

Original: Here we provide convergence analysis under three assumptions:
Revised: Here we provide convergence analysis under three assumptions:

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: The loss function Loss ( · , · ) is C loss -Lipschitz continuous and L loss -smooth w.r.t.
Revised: The loss function Loss ( · , · ) is C loss -Lipschitz continuous and L loss -smooth w.r.t.

Original: to the input node embedding vector, i.e., | Loss ( h ( L ) , y ) − Loss
Revised: to the input node embedding vector, i.e., | Loss ( h ( L ) , y ) − Loss

Original: ( h (cid:48)
Revised: ( h (cid:48)

Original: ( L ) , y )
Revised: ( L ) , y )

Original: | ≤ C loss (cid:107) h ( L ) − h (cid:48)
Revised: | ≤ C loss (cid:107) h ( L ) − h (cid:48)

Original: ( L ) (cid:107) 2 and
Revised: ( L ) (cid:107) 2 and

Original: [Table]
Revised: [Table]

Original: 2 where h is the predicted label and y is the correct label vector.
Revised: 2 where h is the predicted label and y is the correct label vector.

Original: Assumption 3.2.
Revised: Assumption 3.2.

Original: The activation function σ ( · ) is C σ -Lipschitz continuous and L σ -smooth, i.e.,(cid:107) σ ( z ( (cid:96)
Revised: The activation function σ ( · ) is C σ -Lipschitz continuous and L σ -smooth, i.e.,

Original: [Table]
Revised: [Table]

Original: − σ ( z (cid:48)
Revised: σ ( z (

Original: [Table]
Revised: [Table]

Original: ≤
Revised: z

Original: [Table]
Revised: [Table]

Original: Assumption 3.3.
Revised: Assumption 3.3.

Original: For any (cid:96) ∈
Revised: For any (cid:96) ∈

Original: [ L ] , the norm of weight matrices, the propagation matrix, and the input feature matrix are bounded: (cid:107) W ( (cid:96) ) 
Revised: [ L ] , the norm of weight matrices, the propagation matrix, and the input feature matrix are bounded: (cid:107) W ( (cid:96) ) [Table]

Original: (cid:107) F ≤ B W , (cid:107) P (cid:107) F ≤ B P , (cid:107) X (cid:107) F ≤ B X .
Revised: F ≤ B W , (cid:107) P (cid:107) F ≤ B P , (cid:107) X (cid:107) F ≤ B X .

Original: (This generic assumption is also used in (Chen et al., 2018;
Revised: (This generic assumption is also used in (Chen et al., 2018;

Original: Liao et al., 2020; Garg et al., 2020;
Revised: Liao et al., 2020; Garg et al., 2020;

Original: Cong et al., 2021).)
Revised: Cong et al., 2021).)

Original: Then we provide the convergence rate of PipeGCN in the following theorem:
Revised: Then we provide the convergence rate of PipeGCN in the following theorem:

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Under Assumptions 3.1, 3.2, and 3.3, we can derive the following by choosing a learning rate η = √ εE and number of training iterations T =
Revised: Under Assumptions 3.1, 3.2, and 3.3, we can derive the following by choosing a learning rate η = √ εE and number of training iterations T =

Original: ( L ( θ (1) )
Revised: ( L ( θ (1) )

Original: − L ( θ ∗ ))
Revised: − L ( θ ∗ ))

Original: Eε − 32 :
Revised: Eε − 32 :

Original: T
Revised: T

Original: T (cid:88)
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where ε > 0 is an arbitrarily small constant, L ( · ) is the loss function, θ ( t ) is the parameter vector at iteration t , and
Revised: where ε > 0 is an arbitrarily small constant, L ( · ) is the loss function, θ ( t ) is the parameter vector at iteration t , and

Original: [Equation]
Revised: [Equation]

Original: Therefore the convergence rate of PipeGCN is O
Revised: Therefore the convergence rate of PipeGCN is O

Original: ( T − 23 ) , which is better than sampling-based method ( O ( T − 12 ) )
Revised: ( T − 23 ) , which is better than sampling-based method ( O ( T − 12 ) )

Original: (Chen et al., 2018; Cong et al., 2021) and close to full-graph training ( O ( T − 1 ) ) .
Revised: (Chen et al., 2018; Cong et al., 2021) and close to full-graph training ( O ( T − 1 ) ) .

Original: The detailed proof can be found in Appendix A.
Revised: The detailed proof can be found in Appendix A.

Original: 3. T HE P ROPOSED S MOOTHING M ETHOD
Revised: 3. T HE P ROPOSED S MOOTHING M ETHOD

Original: The proposed smoothing method aims at reducing errors incurred by stale features/(feature gradients) at a minimal overhead.
Revised: The proposed smoothing method aims at reducing errors incurred by stale features/(feature gradients) at a minimal overhead.

Original: Here we take the smoothing of feature gradient as an example, but the same formulation also applies to stale features.
Revised: Here we take the smoothing of feature gradient as an example, but the same formulation also applies to stale features.

Original: To improve the approximate gradients for each feature, ﬂuctuations in feature gradients between adjacent iterations should be reduced.
Revised: To improve the approximate gradients for each feature, ﬂuctuations in feature gradients between adjacent iterations should be reduced.

Original: Therefore, we apply a light-weight moving average for the feature gradients of each boundary node v as follow:
Revised: Therefore, we apply a light-weight moving average for the feature gradients of each boundary node v as follow:

Original: [Equation]
Revised: [Equation]

Original: where ˆ δ ( t,(cid:96) )
Revised: where ˆ δ ( t,(cid:96) )

Original: z v is the smoothed feature gradient at layer (cid:96) and iteration t , and γ is the decay rate.
Revised: z v is the smoothed feature gradient at layer (cid:96) and iteration t , and γ is the decay rate.

Original: To integrate the smoothed feature gradient into the backward pass, we rewrite Equ. 4 as:
Revised: To integrate the smoothed feature gradient into the backward pass, we rewrite Equ. 4 as:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: d

Original: [Table]

Original: [Table]

Original: z v + (cid:88)

Original: [Table]
Revised: [Table]

Original: ) d v

Original: [Table]

Original: [Table]

Original: v

Original: Such smoothing of stale features and gradients can thus be independently applied to PipeGCN.
Revised: Such smoothing of stale features and gradients can thus be independently applied to PipeGCN.

Original:  E XPERIMENT R ESULTS
Revised:  E XPERIMENT R ESULTS

Original: We evaluate PipeGCN on four large-scale datasets, Reddit (Hamilton et al., 2017), ogbn-products (Hu et al., 2020), Yelp (Zeng et al., 2020), and ogbn-papers100M (Hu et al., 2020).
Revised: We evaluate PipeGCN on four large-scale datasets, Reddit (Hamilton et al., 2017), ogbn-products (Hu et al., 2020), Yelp (Zeng et al., 2020), and ogbn-papers100M (Hu et al., 2020).

Original: More details are provided in Tab. 3.
Revised: More details are provided in Tab. 3.

Original: To ensure robustness and reproducibility, we ﬁx (i.e., do not tune) the hyperparameters and settings for PipeGCN and its variants throughout all experiments.
Revised: To ensure robustness and reproducibility, we ﬁx (i.e., do not tune) the hyperparameters and settings for PipeGCN and its variants throughout all experiments.

Original: To implement partition parallelism (for both vanilla distributed GCN training and PipeGCN), the widely used
Revised: To implement partition parallelism (for both vanilla distributed GCN training and PipeGCN), the widely used

Original: METIS (Karypis & Kumar, 1998) partitioning algorithm is adopted for graph partitioning with its objective set to minimize the communication volume.
Revised: METIS (Karypis & Kumar, 1998) partition algorithm is adopted for graph partition with its objective set to minimize the communication volume.

Original: We implement PipeGCN in PyTorch (Paszke et al., 2019) and DGL (Wang et al., 2019).
Revised: We implement PipeGCN in PyTorch (Paszke et al., 2019) and DGL (Wang et al., 2019).

Original: Experiments are conducted on a machine with 
Revised: Experiments are conducted on a machine with 10 RTX-2080Ti (11GB),

Original: RTX-2080Ti (11GB), Xeon 6230R@2.10GHz (187GB), and PCIe3x16 connecting CPU-GPU and GPU-GPU.
Revised: Xeon 6230R@2.10GHz (187GB), and PCIe3x16 connecting CPU-GPU and GPU-GPU.

Original: Only for ogbn-papers100M, we use 4 A100 (80GB) instead.
Revised: Only for ogbn-papers100M, we use 4 computational nodes (each contains 8 MI60 GPUs, an AMD EPYC

Revised: CPU, and 48 lane PCI 3.0 connecting CPU-GPU and GPU-GPU) networked with 10Gbps Ethernet.

Original: To support full-graph GCN training with the model sizes in Tab. 3, the minimum required partition numbers are 2, 3, 5, 4 for Reddit, ogbn-products, Yelp, and ogbn-papers100M, respectively.
Revised: To support full-graph GCN training with the model sizes in Tab. 3, the minimum required partition numbers are 2, 3, 5, 4 for Reddit, ogbn-products, Yelp, and ogbn-papers100M, respectively.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: For convenience, we here name all methods: vanilla partition-parallel training of GCNs ( GCN ),
Revised: For convenience, we here name all methods: vanilla partition-parallel training of GCNs ( GCN ),

Original: PipeGCN with feature gradient smoothing ( PipeGCN-G ), PipeGCN with feature smoothing ( PipeGCN-F ), and PipeGCN with both smoothing ( PipeGCN-GF ).
Revised: PipeGCN with feature gradient smoothing ( PipeGCN-G ), PipeGCN with feature smoothing ( PipeGCN-F ), and PipeGCN with both smoothing ( PipeGCN-GF ).

Original: The default decay rate γ for all smoothing methods is set to 0.95 .
Revised: The default decay rate γ for all smoothing methods is set to 0.95 .

Original: 4. I MPROVING T RAINING T HROUGHPUT OVER
Revised: 4. I MPROVING T RAINING T HROUGHPUT OVER

Original: [Figure]
Revised: [Figure]

Original: G
Revised: G

Original: [Figure]
Revised: [Figure]

Original: T 
Revised: T RAINING

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 3 compares the training throughput between PipeGCN and SOTA full-graph training methods (ROC (Jia et al., 2020) and CAGNET (Tripathy et al., 2020)).
Revised: 3 compares the training throughput between PipeGCN and SOTA full-graph training methods (ROC (Jia et al., 2020) and CAGNET (Tripathy et al., 2020)).

Original: We observe that both vanilla partition-parallel training (GCN) and
Revised: We observe that both vanilla partition-parallel training (GCN) and

Original: PipeGCN greatly outperform ROC and CATNET across different number of partitions, because they avoid both the expensive CPU-GPU swaps (ROC) and the redundant node broadcast (CAGNET).
Revised: PipeGCN greatly outperform ROC and CATNET across different number of partitions, because they avoid both the expensive CPU-GPU swaps (ROC) and the redundant node broadcast (CAGNET).

Original: Speciﬁcally, GCN is 3.1 ∼ 6.2 × faster than ROC and 2.1 ∼ 7.9 × faster than CAGNET ( c =2).
Revised: Speciﬁcally, GCN is 3.1 ∼ 6.2 × faster than ROC and 2.1 ∼ 7.9 × faster than CAGNET ( c =2).

Original: PipeGCN further improves upon GCN, achieving a throughput improvement of 5.6 ∼ 9.9 × over ROC and 3.9 ∼ 14.7 × over CAGNET ( c =2).
Revised: PipeGCN further improves upon GCN, achieving a throughput improvement of 5.6 ∼ 9.9 × over ROC and 3.9 ∼ 14.7 × over CAGNET ( c =2).

Original: The comparison on more datasets can be found in the Appendix B, which consistently show the advantages of PipeGCN.
Revised: The comparison on more datasets can be found in the Appendix B, which consistently show the advantages of PipeGCN.

Original: Furthermore, we further provide the epoch time breakdown of ROC and
Revised: Furthermore, we further provide the epoch time breakdown of ROC and

Original: CAGNET on Reddit in the Appendix E for understanding where PipeGCN gains signiﬁcant savings over the baseline algorithms.
Revised: CAGNET on Reddit in the Appendix F for understanding where PipeGCN gains signiﬁcant savings over the baseline algorithms.

Original: Considering the substantial performance gap between ROC/CAGNET and GCN, we focus on comparing GCN with PipeGCN for the reminder of the section.
Revised: Considering the substantial performance gap between ROC/CAGNET and GCN, we focus on comparing GCN with PipeGCN for the reminder of the section.

Original: Note that we are not able to compare with NeuGraph (Ma et al., 2019), AliGraph (Zhu et al., 2019), and
Revised: Note that we are not able to compare with NeuGraph (Ma et al., 2019), AliGraph (Zhu et al., 2019), and

Original: P 3 (Gandhi & Iyer, 2021) as their codes are not open source.
Revised: P 3 (Gandhi & Iyer, 2021) as their codes are not open source.

Original: 4. I MPROVING T RAINING T HROUGHPUT WITHOUT C OMPROMISING A CCURACY
Revised: 4. I MPROVING T RAINING T HROUGHPUT WITHOUT C OMPROMISING A CCURACY

Original: We compare the training performance of both test score and training throughput between GCN and
Revised: We compare the training performance of both test score and training throughput between GCN and

Original: PipeGCN in Tab. 4.
Revised: PipeGCN in Tab. 4.

Original: We can see that PipeGCN without smoothing already achieves a comparable test score with the vanilla GCN training on both Reddit and Yelp, and incurrs only a negligible accuracy drop (-0.08% ∼ -0.23%) on ogbn-products, while boosting the training throughput by 1.72 × ∼ 2.16 × across all datasets and different number of partitions, thus validating the effectiveness of PipeGCN.
Revised: We can see that PipeGCN without smoothing already achieves a comparable test score with the vanilla GCN training on both Reddit and Yelp, and incurrs only a negligible accuracy drop (-0.08% ∼ -0.23%) on ogbn-products, while boosting the training throughput by 1.72 × ∼ 2.16 × across all datasets and different number of partitions, thus validating the effectiveness of PipeGCN.

Original: With the proposed smoothing method plugged in, PipeGCN-G/F/GF is able to compensate the dropped score of vanilla PipeGCN, achieving an equal or even better test score as/than the vanilla
Revised: With the proposed smoothing method plugged in, PipeGCN-G/F/GF is able to compensate the dropped score of vanilla PipeGCN, achieving an equal or even better test score as/than the vanilla

Original: GCN training (without staleness), e.g., 97.14% vs. 97.11% on Reddit, 79.36% vs. 79.14% on ogbn-products and 65.28% vs. 65.26% on Yelp.
Revised: GCN training (without staleness), e.g., 97.14% vs. 97.11% on Reddit, 79.36% vs. 79.14% on ogbn-products and 65.28% vs. 65.26% on Yelp.

Original: Meanwhile, PipeGCN-G/F/GF enjoys a similar throughput improvement as vanilla PipeGCN, thus validating the negligible overhead of the proposed smoothing method.
Revised: Meanwhile, PipeGCN-G/F/GF enjoys a similar throughput improvement as vanilla PipeGCN, thus validating the negligible overhead of the proposed smoothing method.

Original: Therefore, pipelined transfer of features and gradients greatly improves the training throughput while maintaining the full-graph accuracy .
Revised: Therefore, pipelined transfer of features and gradients greatly improves the training throughput while maintaining the full-graph accuracy .

Original: Note that our distributed GCN training methods consistently achieve higher test scores than SOTA sampling-based methods for GraphSAGE-based models reported in (Zeng et al., 2020) and (Hu et al., 2020), conﬁrming that the full-graph training technique is preferred to obtain better GCN models.
Revised: Note that our distributed GCN training methods consistently achieve higher test scores than SOTA sampling-based methods for GraphSAGE-based models reported in (Zeng et al., 2020) and (Hu et al., 2020), conﬁrming that the full-graph training technique is preferred to obtain better GCN models.

Original: For example, the best sampling-based method achieves a 96.6% accuracy on Reddit (Zeng et al., 2020) while full-graph GCN training achieves 97.1%, and PipeGCN improves the accuracy by 0.28% over sampling-based GraphSAGE models on ogbn-products (Hu et al., 2020).
Revised: For example, the best sampling-based method achieves a 96.6% accuracy on Reddit (Zeng et al., 2020) while full-graph GCN training achieves 97.1%, and PipeGCN improves the accuracy by 0.28% over sampling-based GraphSAGE models on ogbn-products (Hu et al., 2020).

Original: Such an advantage of full-graph training is also validated by recent works (Jia et al., 2020; Tripathy et al., 2020).
Revised: Such an advantage of full-graph training is also validated by recent works (Jia et al., 2020; Tripathy et al., 2020).

Original: 4. M AINTAINING C ONVERGENCE S PEED
Revised: 4. M AINTAINING C ONVERGENCE S PEED

Original: To understand PipeGCN’s inﬂuence on the convergence speed, we compare the training curve among different methods in Fig.
Revised: To understand PipeGCN’s inﬂuence on the convergence speed, we compare the training curve among different methods in Fig.

Original: We observe that the convergence of PipeGCN without smoothing is still comparable with that of the vanilla GCN training, although PipeGCN converges slower at the
Revised: We observe that the convergence of PipeGCN without smoothing is still comparable with that of the vanilla GCN training, although PipeGCN converges slower at the

Original: [Table]
Revised: [Table]

Original: 1.82 ×
Revised: 1.82 ×

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: 1 × (2.00 epochs
Revised: 1 × (2.

Original: [Figure]
Revised: [Figure]

Original: s
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 65.27 ±

Original: [Figure]
Revised: [Figure]

Original: 2.16 ×
Revised: 2.16 ×

Original: PipeGCN
Revised: PipeGCN

Original: [Figure]
Revised: [Figure]

Original: 65.26 ±
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: 2.15 ×
Revised: 2.15 ×

Original: PipeGCN
Revised: PipeGCN

Original: [Figure]
Revised: [Figure]

Original: 65.26 ±
Revised: 65.26 ±

Original: [Figure]
Revised: [Figure]

Original: 2.15 ×
Revised: 2.15 ×

Original: PipeGCN
Revised: PipeGCN

Original: [Figure]
Revised: [Figure]

Original: 65.26 ±
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: 2.11 ×
Revised: 2.11 ×

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 26 ±

Original: [Figure]
Revised: [Figure]

Original: 1 × (2.25 epochs
Revised: 1 × (2.25 epochs

Original: [Figure]
Revised: [Figure]

Original: s)

Original: [Figure][Figure]
Revised: [Figure]

Original: 1.72 ×
Revised: 1.72 ×

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 1.69 ×
Revised: 1.69 ×

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: 65.25 ±
Revised: 25 ±

Original: [Figure]
Revised: [Figure]

Original: 1.68 ×
Revised: 1.68 ×

Original: [Figure]
Revised: [Figure]

Original: 65.26 ± 0.
Revised: 65.26 ± 0.

Original: 1.67 ×
Revised: 1.67 ×

Original: [Figure]
Revised: [Figure]

Original: variants (PipeGCN*), where PipeGCN and its variants achieve a similar convergence as the vanilla training (without staleness) but are twice as fast in wall-clock time (see Tab. 4).
Revised: variants (PipeGCN*), where PipeGCN and its variants achieve a similar convergence as the vanilla training (without staleness) but are twice as fast in wall-clock time (see Tab. 4).

Original: early phase of training and then catches up at the later phase, due to the staleness of boundary features/gradients.
Revised: early phase of training and then catches up at the later phase, due to the staleness of boundary features/gradients.

Original: With the proposed smoothing methods, PipeGCN-G/F boosts the convergence substantially and matches or even outperforms (esp.
Revised: With the proposed smoothing methods, PipeGCN-G/F boosts the convergence substantially and matches or even outperforms (esp.

Original: at late training phase) the convergence speed of vanilla GCN training .
Revised: at late training phase) the convergence speed of vanilla GCN training .

Original: There is no clear difference between PipeGCN-G and PipeGCN-F. Lastly, with combined smoothing of features and gradients, PipeGCN-GF can acheive the same convergence speed as vanilla GCN training (e.g., on Reddit) but can overﬁt gradually similar to the vanilla GCN training, which is further investigated in Sec. 4.4.
Revised: There is no clear difference between PipeGCN-G and PipeGCN-F. Lastly, with combined smoothing of features and gradients, PipeGCN-GF can acheive the same convergence speed as vanilla GCN training (e.g., on Reddit) but can overﬁt gradually similar to the vanilla GCN training, which is further investigated in Sec. 4.4.

Original: Therefore, PipeGCN maintains the convergence speed w.r.t the number of epochs while reduces the end-to-end training time by around 50% thanks to its boosted training throughput (see Tab. 4).
Revised: Therefore, PipeGCN maintains the convergence speed w.r.t the number of epochs while reduces the end-to-end training time by around 50% thanks to its boosted training throughput (see Tab. 4).

Original: 4. B ENEFIT OF P IPE GCN WITH S TALENESS S MOOTHING
Revised: 4. B ENEFIT OF P IPE GCN WITH S TALENESS S MOOTHING

Original: Error Reduction and Convergence Speedup.
Revised: Error Reduction and Convergence Speedup.

Original: To understand why the proposed smoothing technique (see Sec. 3.4) speeds up convergence, we compare the error incurred by the stale communication between PipeGCN and PipeGCN-G/F.
Revised: To understand why the proposed smoothing technique (see Sec. 3.4) speeds up convergence, we compare the error incurred by the stale communication between PipeGCN and PipeGCN-G/F.

Original: The error is calculated as the Frobenius-norm of the gap between the correct gradient/feature and the stale gradient/feature used in PipeGCN training.
Revised: The error is calculated as the Frobenius-norm of the gap between the correct gradient/feature and the stale gradient/feature used in PipeGCN training.

Original: Fig.
Revised: Fig.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: compares the error at each GCN layer.
Revised: compares the error at each GCN layer.

Original: We can see that the proposed smoothing technique (PipeGCNG/F) reduces the error of staleness substantially (from the base version of PipeGCN) and this beneﬁt consistently holds across different layers in terms of both feature and gradients errors, validating the effectiveness of our smoothing method and explains its improvement to the convergence speed.
Revised: We can see that the proposed smoothing technique (PipeGCNG/F) reduces the error of staleness substantially (from the base version of PipeGCN) and this beneﬁt consistently holds across different layers in terms of both feature and gradients errors, validating the effectiveness of our smoothing method and explains its improvement to the convergence speed.

Original: Overﬁtting Reduction.
Revised: Overﬁtting Reduction.

Original: To understand the effect of staleness smoothing on model overﬁtting, we also evaluate the test-accuracy convergence under different decay rates γ in Fig.
Revised: To understand the effect of staleness smoothing on model overﬁtting, we also evaluate the test-accuracy convergence under different decay rates γ in Fig.

Original: Here ogbn-products is adopted as the study case because the distribution of its test set largely differs from that of its training set.
Revised: Here ogbn-products is adopted as the study case because the distribution of its test set largely differs from that of its training set.

Original: From Fig.
Revised: From Fig.

Original: 6, we observe that smoothing with a large γ ( 0 .
Revised: 6, we observe that smoothing with a large γ ( 0 .

Original: 95 ) offers a fast convergence, i.e., close to the vanilla GCN training, but overﬁts rapidly.
Revised: 95 ) offers a fast convergence, i.e., close to the vanilla GCN training, but overﬁts rapidly.

Original: To understand this issue, we further provide detailed comparisons of the errors incurred under different γ in Fig.
Revised: To understand this issue, we further provide detailed comparisons of the errors incurred under different γ in Fig.

Original: We can see that a larger γ enjoys lower approximation errors and makes the gradients/features more stable, thus improving the convergence speed.
Revised: We can see that a larger γ enjoys lower approximation errors and makes the gradients/features more stable, thus improving the convergence speed.

Original: The increased stability on the training set, however, constrains the model from exploring a more general minimum point on the test set, thus leading to overﬁtting as the vanilla GCN training.
Revised: The increased stability on the training set, however, constrains the model from exploring a more general minimum point on the test set, thus leading to overﬁtting as the vanilla GCN training.

Original: In contrast, a small γ ( 0 ∼ 0 . 5 ) mitigates this overﬁtting and achieves a better accuracy (see Fig. 6).
Revised: In contrast, a small γ ( 0 ∼ 0 . 5 ) mitigates this overﬁtting and achieves a better accuracy (see Fig. 6).

Original: But a too-small γ (e.g., 0 ) gives a high error for both stale features and gradients (see
Revised: But a too-small γ (e.g., 0 ) gives a high error for both stale features and gradients (see

Original: Fig. 7), thus suffering from a slower convergence.
Revised: Fig. 7), thus suffering from a slower convergence.

Original: Therefore, a trade-off between convergence speed and achievable optimality exists between different smoothing decay rates, and γ = 0 .
Revised: Therefore, a trade-off between convergence speed and achievable optimality exists between different smoothing decay rates, and γ = 0 .

Original: 5 combines the best of both worlds in this study.
Revised: 5 combines the best of both worlds in this study.

Original: 4. T RAINING T IME I MPROVEMENT B REAKDOWN
Revised: 4. T RAINING T IME I MPROVEMENT B REAKDOWN

Original: [Table]
Revised: [Table]

Original: To further understand the training time improvement of
Revised: To further understand the training time improvement of

Original: PipeGCN, we breakdown the epoch time into three parts (intrapartition computation, inter-partition communication, and reduce of model gradient) and provide an example in Tab 6.
Revised: PipeGCN, we breakdown the epoch time into three parts (intrapartition computation, inter-partition communication, and reduce of model gradient) and provide an example in Tab. 8.

Original: More results can be found in Appendix D.
Revised: More results can be found in Appendix D.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we propose a new method, PipeGCN, for efﬁcient full-graph GCN training.
Revised: In this work, we propose a new method, PipeGCN, for efﬁcient full-graph GCN training.

Original: PipeGCN pipelines communication with computation in distributed GCN training to hide the substantial communication overhead.
Revised: PipeGCN pipelines communication with computation in distributed GCN training to hide the substantial communication overhead.

Original: Furthermore, we take an initial effort to understand the convergence of
Revised: Furthermore, we take an initial effort to understand the convergence of

Original: GCN training with both stale features and feature gradients, and further propose a smoothing method to speedup the convergence of vanilla PipeGCN.
Revised: GCN training with both stale features and feature gradients, and further propose a smoothing method to speedup the convergence of vanilla PipeGCN.

Original: Extensive experiments validate the advantages of
Revised: Extensive experiments validate the advantages of

Original: PipeGCN over both vanilla training (without staleness) and SOTA full-graph training.
Revised: PipeGCN over both vanilla training (without staleness) and SOTA full-graph training.

Original: A C ONVERGENCE P ROOF
Revised: A C ONVERGENCE P ROOF

Original: In this section, we prove the convergence of PipeGCN.
Revised: In this section, we prove the convergence of PipeGCN.

Original: The essential step is to prove that the bound of the gradient error is controlled by learning rate η .
Revised: The essential step is to prove that the bound of the gradient error is controlled by learning rate η .

Original: A. N OTATIONS
Revised: A. N OTATIONS

Original: For a given graph G =
Revised: For a given graph G =

Original: ( V , E ) with an adjacency matrix A , feature matrix X , we deﬁne the propagation matrix P as P :=
Revised: ( V , E ) with an adjacency matrix A , feature matrix X , we deﬁne the propagation matrix P as P :=

Original: ˜ D − 1 / 2 ˜
Revised: ˜ D − 1 / 2 ˜

Original: A ˜ D − 1 / 2 , where
Revised: A ˜ D − 1 / 2 , where

Original: ˜ A =
Revised: ˜ A =

Original: A + I, ˜ D u,u = (cid:80) v ˜
Revised: A + I, ˜ D u,u = (cid:80) v ˜

Original: A u,v .
Revised: A u,v .

Original: One GCN layer performs one step of feature propagation (Kipf & Welling, 2016) as formulated below
Revised: One GCN layer performs one step of feature propagation (Kipf & Welling, 2016) as formulated below

Original: [Equation]
Revised: [Equation]

Original: where H ( (cid:96) ) , W ( (cid:96) ) , and Z ( (cid:96) ) denote the embedding matrix, the trainable weight matrix, and the intermediate embedding matrix in the l -th layer, respectively, and σ denotes the non-linear function.
Revised: where H ( (cid:96) ) , W ( (cid:96) ) , and Z ( (cid:96) ) denote the embedding matrix, the trainable weight matrix, and the intermediate embedding matrix in the (cid:96) -th layer, respectively, and σ denotes the non-linear function.

Original: For an L -layer GCN, the loss function is denoted by L ( θ ) where θ = vec [ W (1) , W (2) , · · · , W ( L ) ] .
Revised: For an L -layer GCN, the loss function is denoted by L ( θ ) where θ = vec [ W (1) , W (2) , · · · , W ( L ) ] .

Original: We deﬁne the l -th layer as a function f ( (cid:96) ) ( · , · ) .
Revised: We deﬁne the (cid:96) -th layer as a function f ( (cid:96) ) ( · , · ) .

Original: [Equation]
Revised: [Equation]

Original: and its gradient w.r.t.
Revised: and its gradient w.r.t.

Original: the input embedding matrix can be represented as
Revised: the input embedding matrix can be represented as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: and its gradient w.r.t.
Revised: and its gradient w.r.t.

Original: the weight can be represented as
Revised: the weight can be represented as

Original: [Equation]
Revised: [Equation]

Original: where M ( (cid:96) )
Revised: where M ( (cid:96) )

Original: =
Revised: =

Original: J ( (cid:96) ) ◦ σ (cid:48)
Revised: J ( (cid:96) ) ◦ σ (cid:48)

Original: ( Z ( (cid:96) ) ) and ◦ denotes Hadamard product.
Revised: ( Z ( (cid:96) ) ) and ◦ denotes Hadamard product.

Original: For partition-parallel training, we can split P into two parts P = P in + P bd where P in represents interpartition propagation and P bd denotes intra-partition propagation.
Revised: For partition-parallel training, we can split P into two parts P = P in + P bd where P in represents interpartition propagation and P bd denotes intra-partition propagation.

Original: For PipeGCN, we can represent one GCN layer as below
Revised: For PipeGCN, we can represent one GCN layer as below

Original: [Equation]
Revised: [Equation]

Original: where t is the epoch number and (cid:102) W ( t,(cid:96) ) is the weight at epoch t layer l .
Revised: where t is the epoch number and (cid:102) W ( t,(cid:96) ) is the weight at epoch t layer (cid:96) .

Original: We deﬁne the loss function for this setting as (cid:101) L ( (cid:101) θ ( t ) ) where (cid:101) θ ( t ) = vec [ (cid:102) W
Revised: We deﬁne the loss function for this setting as (cid:101) L ( (cid:101) θ ( t ) ) where (cid:101) θ ( t ) = vec [ (cid:102) W

Original: ( t, 1) , (cid:102) W ( t, 2) , · · · , (cid:102) W ( t,L ) ] .
Revised: ( t, 1) , (cid:102) W ( t, 2) , · · · , (cid:102) W ( t,L ) ] .

Original: We can also summarize the layer as a function (cid:101) f ( t,(cid:96) ) ( · , · )
Revised: We can also summarize the layer as a function (cid:101) f ( t,(cid:96) ) ( · , · )

Original: [Equation]
Revised: [Equation]

Original: Note that (cid:101) H ( t − 1 ,(cid:96) − 1) is not a part of the input of (cid:101) f
Revised: Note that (cid:101) H ( t − 1 ,(cid:96) − 1) is not a part of the input of (cid:101) f

Original: ( t,(cid:96) )
Revised: ( t,(cid:96) )

Original: ( · , · ) because it is a constant for the t -th epoch.
Revised: ( · , · ) because it is a constant for the t -th epoch.

Original: The corresponding backward propagation follows the following computation
Revised: The corresponding backward propagation follows the following computation

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: ,(cid:96) ) ] (cid:62) + P (cid:62) bd (cid:102) M
Revised: ) [ (cid:102) W ( t,(cid:96) ) ] (cid:62) + P (cid:62) bd (cid:102) M

Original: ( t − 1 ,(cid:96) )
Revised: ( t − 1 ,(cid:96) )

Original: [ (cid:102) W ( t − 1 ,(cid:96) ) ] (cid:62) ∇ W (cid:101) f ( t,(cid:96) )
Revised: [ (cid:102) W ( t − 1 ,(cid:96) ) ] (cid:62) ∇ W (cid:101) f ( t,(cid:96) )

Original: ( (cid:101) J ( t,(cid:96) ) , (cid:101) H ( t,(cid:96) − 1) , (cid:102) W ( t,(cid:96) ) ) :
Revised: ( (cid:101) J ( t,(cid:96) ) , (cid:101) H ( t,(cid:96) − 1) , (cid:102) W ( t,(cid:96) ) ) :

Original: = [ P in (cid:101) H ( t,(cid:96) − 1) + P bd (cid:101) H ( t − 1 ,(cid:96) − 1) ] (cid:62) (cid:102) M ( t,(cid:96) )
Revised: = [ P in (cid:101) H ( t,(cid:96) − 1) + P bd (cid:101) H ( t − 1 ,(cid:96) − 1) ] (cid:62) (cid:102) M ( t,(cid:96) )

Original: Again, (cid:101) J ( t − 1 ,(cid:96) ) is not a part of the input of ∇ H (cid:101) f ( t,(cid:96) ) ( · , · , · ) or ∇ W (cid:101) f ( t,(cid:96) ) ( · , · , · ) because it is a constant for epoch t .
Revised: Again, (cid:101) J ( t − 1 ,(cid:96) ) is not a part of the input of ∇ H (cid:101) f ( t,(cid:96) ) ( · , · , · ) or ∇ W (cid:101) f ( t,(cid:96) ) ( · , · , · ) because it is a constant for epoch t .

Original: Finally, we deﬁne ∇ (cid:101) L ( (cid:101) θ ( t ) )
Revised: Finally, we deﬁne ∇ (cid:101) L ( (cid:101) θ ( t ) )

Original: =
Revised: =

Original: vec [ (cid:101) G ( t, 1) , (cid:101) G ( t, 2) , · · · , (cid:101) G ( t,L ) ] .
Revised: vec [ (cid:101) G ( t, 1) , (cid:101) G ( t, 2) , · · · , (cid:101) G ( t,L ) ] .

Original: It should be highlighted that the ‘gradient’ ∇ H (cid:101) f ( t,(cid:96) ) ( · , · , · ) , ∇ W (cid:101) f ( t,(cid:96) ) ( · , · , · ) and ∇ (cid:101) L ( (cid:101) θ ( t ) ) are not the correct gradient for the corresponding forward process due to the stale communication.
Revised: It should be highlighted that the ‘gradient’ ∇ H (cid:101) f ( t,(cid:96) ) ( · , · , · ) , ∇ W (cid:101) f ( t,(cid:96) ) ( · , · , · ) and ∇ (cid:101) L ( (cid:101) θ ( t ) ) are not the correct gradient for the corresponding forward process due to the stale communication.

Original: Properties of gradient cannot be directly applied to these variables.
Revised: Properties of gradient cannot be directly applied to these variables.

Original: The deﬁnitions of other constants C loss , L loss , C σ , L σ , B W , B P , B X can be found in Assumption3.1-3.3 of the main content.
Revised: The deﬁnitions of other constants C loss , L loss , C σ , L σ , B W , B P , B X can be found in Assumption

Original: A. B OUNDED M ATRICES AND C HANGES
Revised: A. B OUNDED M ATRICES AND C HANGES

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For any l ∈ [ L ] , the Frobenius norm of node embedding matrices, gradient passing from the l -th layer node embeddings to the ( (cid:96) − 1) -th, gradient matrices are bounded, i.e., 
Revised: For any (cid:96) ∈ [ L ] , the Frobenius norm of node embedding matrices, gradient passing from the (cid:96) -th layer node embeddings to the ( (cid:96) − 1) -th, gradient matrices are bounded, i.e., (cid:107) H ( (cid:96) ) (cid:107) F , (cid:107) (cid:101) [Equation]

Revised: ( t

Original: [Equation]
Revised: [Equation]

Revised: (cid:96) )

Revised: (cid:107)

Revised: [Equation]

Revised: ( (cid:96) )

Revised: (cid:107)

Revised: [Equation]

Revised: (cid:107) (cid:102)

Revised: [Equation]

Revised: ( t

Revised: [Equation]

Revised: (cid:96) )

Revised: (cid:107)

Revised: [Equation]

Revised: (cid:107) G ( (cid:96) )

Revised: (cid:107) F , (cid:107) (cid:101) G ( t,(cid:96) )

Revised: (cid:107)

Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof of (cid:107) H ( (cid:96) )
Revised: The proof of (cid:107) H ( (cid:96) )

Original: (cid:107) F ≤ B H and (cid:107) J
Revised: (cid:107) F ≤ B H and (cid:107) J

Original: ( (cid:96) )
Revised: ( (cid:96) )

Original: (cid:107) F ≤ B J can be found in Proposition 2 in (Cong et al., 2021).
Revised: (cid:107) F ≤ B J can be found in Proposition 2 in (Cong et al., 2021).

Original: By induction,
Revised: By induction,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Because the gradient matrices are bounded, the weight change is bounded.
Revised: Because the gradient matrices are bounded, the weight change is bounded.

Original: Corollary A.1.
Revised: Corollary A.1.

Original: For any t, l, (cid:107) (cid:102) W ( t,(cid:96) ) − (cid:102) W ( t − 1 ,(cid:96) )
Revised: For any t , (cid:96) , (cid:107) (cid:102) W ( t,(cid:96) ) − (cid:102) W ( t − 1 ,(cid:96) )

Original: (cid:107) F ≤ B ∆ W = ηB G where η is the learning rate.
Revised: (cid:107) F ≤ B ∆ W = ηB G where η is the learning rate.

Original: Now we can analyze the changes of intermediate variables.
Revised: Now we can analyze the changes of intermediate variables.

Original: Lemma A.2.
Revised: Lemma A.2.

Original: For any t, (cid:96) , we have (cid:107) (cid:101) Z ( t,(cid:96) ) − (cid:101)
Revised: For any t, (cid:96) , we have (cid:107) (cid:101) Z ( t,(cid:96) ) − [Equation] Z

Revised: [Equation]

Original: Z ( t − 1 ,(cid:96) ) (cid:107) F ≤ B ∆ Z ,(cid:107) (cid:101) H ( t,(cid:96) ) − (cid:101) H
Revised: t − 1 , [Equation] [Equation] F ≤ B ∆ Z ,

Revised: [Equation]

Original:  ( t − 1 ,(cid:96) )
Revised: H [Equation] t,(cid:96) ) − (cid:101) H ( t − 1 ,(cid:96) )

Original: (cid:107) F ≤ B ∆ H , where B ∆ Z =
Revised: (cid:107) F ≤ B ∆ H , where B ∆ Z =

Original: L − 1 (cid:80) i =

Revised: L − 1 (cid:80) i =

Original: C i σ B i +

Revised: C i σ B i +

Original: P B
Revised: P B

Original: i W B H B ∆ W and B ∆ H =
Revised: i W B H B ∆ W and B ∆ H =

Original: C σ B ∆ Z .
Revised: C σ B ∆ Z .

Original: Proof.
Revised: Proof.

Original: When (cid:96) = [Equation]
Revised: When (cid:96) = 0 , (cid:107) (cid:101)

Original: H
Revised: H

Original: [Equation]
Revised: [Equation]

Original: [Equation] −
Revised: t

Original: [Equation]
Revised: [Equation]

Original:  H ( t − [Equation] 0) (cid:107) F = (cid:107) X − X (cid:107) F = 0 .
Revised: 0) − (cid:101) H ( t − 1 , 0) (cid:107) F = (cid:107) X − X (cid:107) F = 0 .

Original: Now we consider l > 0 by induction.
Revised: Now we consider (cid:96) > 0 by induction.

Original: [Equation]
Revised: [Equation]

Original: Then we analyze the bound of s 
Revised: Then we analyze the bound of s (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: :=
Revised: :=

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: According to A.1,
Revised: According to A

Revised: [Table]

Revised: [Table]

Revised: W

Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: By induction,
Revised: By induction,

Original: [Table]
Revised: [Table]

Revised: H

Revised: [Table]

Original: C
Revised: C

Revised: [Table]

Revised: +

Revised: [Table]

Revised: B iW B

Original: [Table]
Revised: [Table]

Original: B
Revised: B

Original: [Table]
Revised: [Table]

Revised: W .

Original: Combining these inequalities,
Revised: Combining these inequalities,

Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Plugging it back, we have
Revised: Plugging it back, we have

Original: [Equation]
Revised: [Equation]

Original: Lemma A.3.
Revised: Lemma A.3.

Original: [Equation] J where
Revised: (cid:107) (cid:101) J ( t,(cid:96) ) − (cid:101) J ( t − 1 ,(cid:96) ) (cid:107) F ≤ B ∆ J where

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: =
Revised: max

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For the last layer ( (cid:96) =
Revised: For the last layer ( (cid:96) =

Original: L ) , [Equation] J [Equation]
Revised: L ) , (cid:107) (cid:101) J ( t,L ) − (cid:101) J ( t − 1 ,L ) (cid:107) F ≤ L loss (cid:107) (cid:101)

Original:  t
Revised: H ( t

Original: [Equation]
Revised: [Equation]

Original: L ) − [Equation] J ( t [Equation]
Revised: L ) − (cid:101) H ( t − 1 ,L ) (cid:107) F ≤ L loss B ∆ H .

Original: [Equation]
Revised: For the case of (cid:96) <

Original: L [Equation] F ≤ L loss (cid:107) (cid:101) H ( t
Revised: L , we prove the lemma by using induction.

Original: [Equation]
Revised: [Equation]

Original: L ) − (cid:101) H ( t −
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: L ) (cid:107) F ≤ L loss B ∆ H . For the case of l < L
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: we prove the lemma by using induction

Original:  [Equation]
Revised: [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: According to Corollary A.
Revised: According to Corollary A.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: For the second term,
Revised: For the second term,

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: According to the smoothness of σ and Lemma A.2, (cid:107) σ [Equation] [Equation] Z
Revised: According to the smoothness of σ and Lemma A.

Original: [Equation]
Revised: [Equation]

Original: t [Equation] ) − σ(cid:48) (
Revised: − σ

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: t −
Revised: ≤ L σ B

Original: [Equation]
Revised: [Equation]

Original: (cid:96) ) ) (cid:107) F ≤ L σ B ∆ Z .
Revised: Z .

Original: By induction,
Revised: By induction,

Original: [Equation]
Revised: [Equation]

Original: As a result,
Revised: As a result

Original: [Equation]
Revised: [Equation]

Original: J
Revised: From Equation

Original: [Equation]
Revised: [Equation]

Original: From Equation 5, we can also conclude that
Revised: we can also conclude that

Original: Corollary A.2.
Revised: Corollary A.2.

Original: (cid:107) (cid:102) M
Revised: (cid:107) (cid:102) M

Original: ( t,(cid:96) )
Revised: ( t,(cid:96) )

Original: − (cid:102) M
Revised: − (cid:102) M

Original: ( t − 1 ,(cid:96) )
Revised: ( t − 1 ,(cid:96) )

Original: (cid:107) F ≤ B ∆ M with B ∆ M = B J L σ B ∆ Z + C σ B ∆ J .
Revised: (cid:107) F ≤ B ∆ M with B ∆ M = B J L σ B ∆ Z + C σ B ∆ J .

Original: A. B OUNDED F EATURE E RROR AND G RADIENT E RROR
Revised: A. B OUNDED F EATURE E RROR AND G RADIENT E RROR

Original: In this subsection, we compare the difference between generic GCN and PipeGCN with the same parameter set, i.e., θ = (cid:101) θ ( t ) .
Revised: In this subsection, we compare the difference between generic GCN and PipeGCN with the same parameter set, i.e., θ = (cid:101) θ ( t ) .

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: [Table]
Revised: [Table]

Original: Proof.
Revised: Proof.

Original: [Table]
Revised: [Table]

Original: F + B ∆ H (cid:17)
Revised: F + B ∆ H (cid:17)

Original: By induction, we assume that (cid:107) (cid:101) H ( t,(cid:96) − 1) − H ( (cid:96) − 1) (cid:107) F ≤ B ∆ H l − 1 (cid:80) i [-=
-] ( C σ B W B P ) i .
Revised: By induction, we assume that (cid:107) (cid:101) H ( t,(cid:96) − 1) − H ( (cid:96) − 1) (cid:107) F ≤ B ∆ H (cid:96) − 1 (cid:80) i [-=
-]=1 ( C σ B W B P ) i .

Original: Therefore, (cid:107) (cid:101) Z ( t,(cid:96) )
Revised: Therefore, (cid:107) (cid:101) Z ( t,(cid:96) )

Original: − Z ( (cid:96) ) (cid:107) F ≤ B W B P B ∆ H l − 1 (cid:88) i
Revised: − Z ( (cid:96) ) (cid:107) F ≤ B W B P B ∆ H (cid:96) − 1 (cid:88) i

Original: =
Revised: =

Original: ( C σ B W B P ) i =
Revised: ( C σ B W B P ) i =

Original: B ∆ H l (cid:88) i =
Revised: B ∆ H (cid:96) (cid:88) i =

Original: C i − 1 σ B iW B iP (cid:107) (cid:101) H ( t,(cid:96) )
Revised: C i − 1 σ B iW B iP (cid:107) (cid:101) H ( t,(cid:96) )

Original: − H ( (cid:96) )
Revised: − H ( (cid:96) )

Original: (cid:107) F = (cid:107) σ ( (cid:101) Z ( t,(cid:96) ) )
Revised: (cid:107) F = (cid:107) σ ( (cid:101) Z ( t,(cid:96) ) )

Original: − σ ( Z ( (cid:96) ) )
Revised: − σ ( Z ( (cid:96) ) )

Original: (cid:107) F ≤ C σ (cid:107) (cid:101) Z ( t,(cid:96) )
Revised: (cid:107) F ≤ C σ (cid:107) (cid:101) Z ( t,(cid:96) )

Original: − Z ( (cid:96) ) (cid:107) F ≤ B ∆ H l (cid:88) i =
Revised: − Z ( (cid:96) ) (cid:107) F ≤ B ∆ H (cid:96) (cid:88) i =

Original: ( C σ B W B P )
Revised: ( C σ B W B P )

Original: i
Revised: i

Original: Lemma A.5.
Revised: Lemma A.5.

Original: (cid:107) (cid:101) J ( t,(cid:96) )
Revised: (cid:107) (cid:101) J ( t,(cid:96) )

Original: − J ( (cid:96) ) (cid:107) F ≤ E J and (cid:107) (cid:102) M ( t,(cid:96) )
Revised: − J ( (cid:96) ) (cid:107) F ≤ E J and (cid:107) (cid:102) M ( t,(cid:96) )

Original: − M ( (cid:96) ) (cid:107) F ≤ E M with
Revised: − M ( (cid:96) ) (cid:107) F ≤ E M with

Original: E J = max l ( B P B W C σ ) l L loss E H + B P
Revised: E J = max (cid:96) ( B P B W C σ ) (cid:96) L loss E H + B P

Original: ( B W ( B J E Z L σ + B ∆ M ) + B ∆ W B M )
Revised: ( B W ( B J E Z L σ + B ∆ M ) + B ∆ W B M )

Original: L − 1 (cid:88) i =
Revised: L − 1 (cid:88) i =

Original: ( B P B W C σ )
Revised: ( B P B W C σ )

Original: i
Revised: i

Original: E M = C σ E J + L σ B J E Z
Revised: E M = C σ E J + L σ B J E Z

Original: Proof.
Revised: Proof.

Original: When (cid:96) =
Revised: When (cid:96) =

Original: L , (cid:107) (cid:101) J [Equation] t
Revised: L , (cid:107) (cid:101) J ( t,L )

Original: [Equation] L ) [Equation]
Revised: − J

Original:  L )
Revised: ( L )

Original: [Equation]
Revised: [Equation]

Original:  ≤ L loss E H .
Revised: F ≤ L loss E H .

Original: We assume that
Revised: We assume that

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where U = B P ( B W B J E Z L σ + B ∆ W B M + B W B ∆ M )
Revised: where U = B P ( B W B J E Z L σ + B ∆ W B M + B W B ∆ M )

Original: [Equation]
Revised: [Equation]

Original: We prove them by induction as follows
Revised: We prove them by induction as follows

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Here (cid:107) σ
Revised: Here (cid:107) σ

Original: [Equation]
Revised: [Equation]

Original:  Z ( t,(cid:96) ) ) − σ (cid:48)
Revised: (cid:101) Z ( t,(cid:96) ) ) − σ (cid:48)

Original: ( Z ( (cid:96) ) )
Revised: ( Z ( (cid:96) ) )

Original: (cid:107) F ≤ L σ E Z .
Revised: (cid:107) F ≤ L σ E Z .

Original: With Equation 6,
Revised: With Equation 6,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: On the other hand,
Revised: On the other hand,

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: The ﬁrst part is bounded by Equation 7.
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: For the second part,
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: The ﬁrst part is bounded by Equation 7.

Original: [Equation] [Table] [Equation] [Table]
Revised: For the second part,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: ) [ (cid:102) W ( t,(cid:96) ) ] (cid:62) − (cid:102) M ( t,

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: ) [ (cid:102) W ( t,

Original: [Table]
Revised: [Table]

Original: W ( t,(cid:96) ) ] (cid:62) − (cid:102) M ( t,(cid:96) )

Original: [ (cid:102) W ( t,(cid:96) ) ] (cid:62) (cid:107) F [Equation]
Revised: ) ] (cid:62) (cid:107) F ≤ B ∆ W B M + B W B ∆ M

Original: Therefore,
Revised: Therefore,

Original: [Equation][Equation]
Revised: [Equation]

Original: ( t,(cid:96) )

Original: [Equation]
Revised: [Equation]

Original: (cid:102) W ( t,(cid:96) ) ] (cid:62) (cid:107) F [Equation] [Equation] [Equation]
Revised: 1 ,(cid:96) ) ] (cid:62) − (cid:102)

Original: l

Original: [Equation]
Revised: [Equation]

Original: loss E H + U

Original: L − l (cid:88) i [Equation] ( B
Revised: ( t,(cid:96) )

Original: [Equation]

Original: B

Original: [Equation]

Original: i

Original: [Equation] U = ( B [Equation]
Revised: [ (cid:102) W ( t,(cid:96) ) ] (cid:62) (cid:107) F

Original: B

Original: [Equation]
Revised: [Equation]

Original: C

Original: [Equation]

Original: [Equation]

Original: H + U

Original: L

Original: [Equation]

Original: l (cid:88)

Original: i [Equation] ( B [Equation]
Revised: =

Original: B W C

Original: [Equation]
Revised: [Equation]

Original: i

Original: Lemma A.6.
Revised: Lemma A.6.

Original: (cid:107) (cid:101) G ( t,(cid:96) )
Revised: (cid:107) (cid:101) G ( t,(cid:96) )

Original: − G ( (cid:96) ) (cid:107) F ≤ E G where E G = B P ( B H E M + B M E H )
Revised: − G ( (cid:96) ) (cid:107) F ≤ E G where E G = B P ( B H E M + B M E H )

Original: Proof.
Revised: Proof.

Original: (cid:107) [Equation] G 
Revised: (cid:107) (cid:101) G ( t,(cid:96) )

Original: [Equation] ) − G ( (cid:96) ) (cid:107) F = (cid:13)(cid:13)(cid:13) [ P in 
Revised: − G ( (cid:96) ) (cid:107) F = (cid:13)(cid:13)(cid:13) [ P in (cid:101) H ( t,(cid:96) − 1) + P bd (cid:101)

Original: [Equation]
Revised: [Table]

Original: H

Original: [Equation]
Revised: [Equation]

Original: t
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: − 1) + P bd (cid:101) H ( t − 1 ,(cid:96) − 1) ] (cid:62) (cid:102)
Revised: [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Original: By summing up from (cid:96) = 1 to (cid:96) =
Revised: By summing up from (cid:96) = 1 to (cid:96) =

Original: L to both sides, we have
Revised: L to both sides, we have

Original: Corollary A.3.
Revised: Corollary A.3.

Original: (cid:107)∇ (cid:101)
Revised: (cid:107)∇ (cid:101)

Original: L ( θ ) − ∇L ( θ ) (cid:107) 2 ≤ E loss := LE G .
Revised: L ( θ ) − ∇L ( θ ) (cid:107) 2 ≤ E loss := LE G .

Original: According to the derivation of E loss [Table] we observe that E loss contains a factor η .
Revised: According to the derivation of E loss , we observe that E loss contains a factor η .

Original: We can rewrite E loss .
Revised: We can rewrite E loss .

Original: Corollary A.4.
Revised: Corollary A.4.

Original: (cid:107)∇ (cid:101)
Revised: (cid:107)∇ (cid:101)

Original: L ( θ ) − ∇L ( θ ) (cid:107) 2 ≤ ηE where
Revised: L ( θ ) − ∇L ( θ ) (cid:107) 2 ≤ ηE where

Original: [Equation]
Revised: [Equation]

Original: R = C loss (cid:32) 

Revised: R = C loss (cid:32)

Original: L σ S + U 2 m C 2 σ C loss B X
Revised: L σ S + U 2 m C 2 σ C loss B X

Original: L − 1 (cid:88) i =
Revised: L − 1 (cid:88) i =

Original: U i (cid:33)
Revised: U i (cid:33)

Original: E = LB P (cid:32)
Revised: E = LB P (cid:32)

Original: U m B X (cid:32)
Revised: U m B X (cid:32)

Original: C 2 σ S
Revised: C 2 σ S

Original: ( C loss + L loss )
Revised: ( C loss + L loss )

Original: + R
Revised: + R

Original: L (cid:88) i =
Revised: L (cid:88) i =

Original: U i (cid:33) + C 2 σ C loss S (cid:33)
Revised: U i (cid:33) + C 2 σ C loss S (cid:33)

Original: A. P ROOF OF THE M AIN T HEOREM
Revised: A. P ROOF OF THE M AIN T HEOREM

Original: We ﬁrst introduce a lemma before the proof of our main theorem.
Revised: We ﬁrst introduce a lemma before the proof of our main theorem.

Original: Lemma A.7 (Lemma 1 in (Cong et al., 2021)) .
Revised: Lemma A.7 (Lemma 1 in (Cong et al., 2021)) .

Original: An L -layer GCN is L f -Lipschitz
Revised: An L -layer GCN is L f -Lipschitz

Original: smoothness,
Revised: smoothness,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e.,
Revised: e.,

Original: [Equation]
Revised: [Equation]

Original: Now we prove the main theorem
Revised: Now we prove the main theorem

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: With the smoothness of the model,
Revised: With the smoothness of the model,

Original: [Equation]
Revised: [Equation]

Original: Let δ ( t ) = ∇ (cid:101)
Revised: Let δ ( t ) = ∇ (cid:101)

Original: L ( θ ( t ) )
Revised: L ( θ ( t ) )

Original: − ∇L ( θ ( t ) ) and η ≤ 1 /L f , we have
Revised: − ∇L ( θ ( t ) ) and η ≤ 1 /L f , we have

Original: [Equation]
Revised: [Equation]

Original: From Corollary A.4 we know that (cid:107) δ ( t ) (cid:107) 2 < ηE .
Revised: From Corollary A.4 we know that (cid:107) δ ( t ) (cid:107) 2 < ηE .

Original: After rearranging the terms, (cid:107)∇L
Revised: After rearranging the terms, (cid:107)∇L

Original: [Equation]
Revised: [Equation]

Original: Summing up from t = 1 to T and taking the average,
Revised: Summing up from t = 1 to T and taking the average,

Original: [Equation]
Revised: [Equation]

Original: where θ ∗ is the minimum point of L ( · ) .
Revised: where θ ∗ is the minimum point of L ( · ) .

Original: By taking η = √ εE and T =
Revised: By taking η = √ εE and T =

Original: ( L ( θ (1) )
Revised: ( L ( θ (1) )

Original: − L ( θ ∗ ))
Revised: − L ( θ ∗ ))

Original: Eε − 32 with an arbitrarily small constant ε > 0 , we have
Revised: Eε − 32 with an arbitrarily small constant ε > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: B I MPROVING T RAINING T HROUGHPUT OVER F ULL -G RAPH T RAINING M ETHODS (A DDITIONAL E XPERIMENTS )
Revised: B I MPROVING T RAINING T HROUGHPUT OVER F ULL -G RAPH T RAINING M ETHODS (A DDITIONAL E XPERIMENTS )

Original: Figure 8 compares the training throughput between PipeGCN and the SOTA full-graph training methods (ROC (Jia et al., 2020) and CAGNET (Tripathy et al., 2020)) on more datasets under that same setting of Figure 3 of the main content.
Revised: Figure 8 compares the training throughput between PipeGCN and the SOTA full-graph training methods (ROC (Jia et al., 2020) and CAGNET (Tripathy et al., 2020)) on more datasets under that same setting of Figure 3 of the main content.

Original: As can be seen, the advantage of PipeGCN consistently holds , which is similar to Figure 3 of the main content.
Revised: As can be seen, the advantage of PipeGCN consistently holds , which is similar to Figure 3 of the main content.

Original: [Figure]
Revised: [Figure]

Original: C M AINTAINING C ONVERGENCE S PEED (A DDITIONAL E XPERIMENTS )
Revised: C M AINTAINING C ONVERGENCE S PEED (A DDITIONAL E XPERIMENTS )

Original: We provide the additional convergence curves on Yelp in Figure 9.
Revised: We provide the additional convergence curves on Yelp in Figure 9.

Original: We can see that PipeGCN and its variants maintain the convergence speed w
Revised: We can see that PipeGCN and its variants maintain the convergence speed w

Original: [Figure]
Revised: [Figure]

Original: r.t the number of epochs while substantially reducing the end-to-end training time
Revised: r.t the number of epochs while substantially reducing the end-to-end training time

Original: [Figure]
Revised: [Figure]

Original: similar convergence as the vanilla training (without staleness)
Revised: similar convergence as the vanilla training (without staleness)

Original: but are twice as fast in terms of wall-clock time (see the Throughput improvement in Table 3 of the main content).
Revised: but are twice as fast in terms of wall-clock time (see the Throughput improvement in Table 3 of the main content).

Original: D T RAINING T IME I MPROVEMENT B REAKDOWN (A DDITIONAL E XPERIMENTS )
Revised: D T RAINING T IME I MPROVEMENT B REAKDOWN (A DDITIONAL E XPERIMENTS )

Original: To understand the training time improvement offered by PipeGCN, we further breakdown the epoch time into three parts (intra-partition computation, inter-partition communication, and reduce for aggregating model gradient) and provide the result in Figure 10.
Revised: To understand the training time improvement offered by PipeGCN, we further breakdown the epoch time into three parts (intra-partition computation, inter-partition communication, and reduce for aggregating model gradient) and provide the result in Figure 10.

Original: We can observe that: 1) interpartition communication dominates the training time in vanilla distributed training (see GCN); 2)
Revised: We can observe that: 1) interpartition communication dominates the training time in vanilla distributed training (see GCN); 2)

Original: PipeGCN (with or without smoothing) greatly hides the communication overhead across different number of partitions and all datasets, e.g., the communication time is hidden completely in 2-partition
Revised: PipeGCN (with or without smoothing) greatly hides the communication overhead across different number of partitions and all datasets, e.g., the communication time is hidden completely in 2-partition

Original: Reddit and almost completely in 3-partition Yelp, thus the substantial reduction in training time; and 3) the proposed smoothing incurs only minimal overhead (i.e., minor difference between PipeGCN and PipeGCN-GF).
Revised: Reddit and almost completely in 3-partition Yelp, thus the substantial reduction in training time; and 3) the proposed smoothing incurs only minimal overhead (i.e., minor difference between PipeGCN and PipeGCN-GF).

Original: Lastly, we also notice that when communication ratio is extremely large (85%+),
Revised: Lastly, we also notice that when communication ratio is extremely large (85%+),

Original: PipeGCN hides communication signiﬁcantly but not completely (e.g., 10-partition ogbn-products), in which case we can employ those compression or quantization techniques from the area of general
Revised: PipeGCN hides communication signiﬁcantly but not completely (e.g., 10-partition ogbn-products), in which case we can employ those compression or quantization techniques from the area of general

Original: distributed SGD for further reducing the communication, as the compression is orthogonal to the pipeline method.
Revised: distributed SGD for further reducing the communication, as the compression is orthogonal to the pipeline method.

Original: Besides compression, we can also increase the pipeline depth of PipeGCN, e.g., using two iterations of compute to hide one iteration of communication, which is left to our future work.
Revised: Besides compression, we can also increase the pipeline depth of PipeGCN, e.g., using two iterations of compute to hide one iteration of communication, which is left to our future work.

Original: 4 Number of partitions 0.
Revised: [Figure] [Figure]

Original: E p o c h t i m e ( s )
Revised: E p o c h t i m e ( s )

Original: G C N
Revised: G C N

Original: G C N
Revised: G C N

Original: P i p e G C N P i p e G C N P
Revised: P i p e G C N P i p e G C N P

Original: i p e G C N - G F
Revised: i p e G C N - G F

Original: P i p e G C NG F
Revised: P i p e G C NG F

Original: Reddit computation communication reduce 10 Number of partitions 0.
Revised: Reddit computation communication reduce 10 Number of partitions 0.

Original: E p o c h t i m e ( s )
Revised: E p o c h t i m e ( s )

Original: G C N
Revised: G C N

Original: G C N
Revised: G C N

Original: P i p e G C N P i p e G C N P
Revised: P i p e G C N P i p e G C N P

Original: i p e G C N - G
Revised: i p e G C N - G

Original: F
Revised: F

Original: P i p e G C NG F obgn-products computation communication reduce 6 Number of partitions 0.
Revised: P i p e G C NG F obgn-products computation communication reduce 6 Number of partitions 0.

Original: E p o c h t i m e ( s )
Revised: E p o c h t i m e ( s )

Original: G C N
Revised: G C N

Original: G C N
Revised: G C N

Original: P i p e G C N P i p e G C N P
Revised: P i p e G C N P i p e G C N P

Original: i p
Revised: i p

Original: e G C NG F
Revised: e G C NG F

Original: P i p e G C NG F
Revised: P i p e G C NG F

Original: Yelp computation communication reduce
Revised: Yelp computation communication reduce

Original: E T RAINING T IME B REAKDOWN C OMPARISON WITH B ASELINES
Revised: E P IPE GCN WITH M ULTIPLE C OMPUTATIONAL N ODES

Revised: We evaluate PipeGCN on Reddit over different number of partitions and multiple nodes (each contains

Revised: AMD Radeon Instinct MI60 GPUs, an AMD EPYC 7642 CPU, and 48 lane PCI 3.0 connecting

Revised: CPU-GPU and GPU-GPU) networked with 10Gbps Ethernet.

Revised: The corresponding accuracy results of PipeGCN and its variants are summarized below:

Original: To understand where PipeGCN gains signiﬁcantly over baseline algorithms, we provide the detailed time breakdown of ROC and CAGNET in Table 6, from which we observe that PipeGCN greatly saves communication time.
Revised: #partitions (#node*#gpus) PipeGCN [Table] Furthermore

Original: [Table]
Revised: [Table]

Original: (c=
Revised: we provide the speedup against vanilla distributed GCN training below:

Original: [Table]
Revised: [Table]

Original: 2 GPUs) 2. CAGNET (c=2, 2 GPUs) 5. Dist GCN (2 GPUs) 0. PipeGCN (2 GPUs) 0.
Revised: From the two tables above, we can observe that our PipeGCN family consistently maintains the accuracy of the full-graph training, while improving the throughput by 15% ∼ 66% regardless of the machine settings and number of partitions.

Original:  ROC (4 GPUs) 3. CAGNET (c=1, 4 GPUs) 2. CAGNET (c=2, 4 GPUs) 2.
Revised: To understand where PipeGCN gains signiﬁcantly over baseline algorithms, we provide the detailed time breakdown of ROC and CAGNET on Reddit with the same model in Table 3 (4-layer GraphSAGE, 256 hidden units) in Table 8, in which ‘Dist GCN’ is the vanilla distributed GCN training illustrated in Figure 1 (a).

Original: Dist GCN (4 GPUs) 0. PipeGCN (4 GPUs) 0.
Revised: We observe that PipeGCN greatly saves communication time [Table] [Table]

Original: F I MPLEMENTATION D ETAILS
Revised: G I MPLEMENTATION D ETAILS

Original: We discuss the details of the effective and efﬁcient implementation of PipeGCN in this section.
Revised: We discuss the details of the effective and efﬁcient implementation of PipeGCN in this section.

Original: First, for parallel communication and computation, a second cudaStream is required for communication besides the default cudaStream for computation.
Revised: First, for parallel communication and computation, a second cudaStream is required for communication besides the default cudaStream for computation.

Original: To also save memory buffers for communication, we batch all communication (e.g., from different layers) into this second cudaStream.
Revised: To also save memory buffers for communication, we batch all communication (e.g., from different layers) into this second cudaStream.

Original: When the popular communication backend, Gloo, is used, we parallelize the CPU-GPU transfer with CPU-CPU transfer.
Revised: When the popular communication backend, Gloo, is used, we parallelize the CPU-GPU transfer with CPU-CPU transfer.

Original: Second, when Dropout layer is used in GCN model, it should be applied after communication.
Revised: Second, when Dropout layer is used in GCN model, it should be applied after communication.

Original: The implementation of the dropout layer for PipeGCN should be considered carefully so that the dropout mask remains consistent for the input tensor and corresponding gradient.
Revised: The implementation of the dropout layer for PipeGCN should be considered carefully so that the dropout mask remains consistent for the input tensor and corresponding gradient.

Original: If the input feature passes through the dropout layer before being communicated, during the backward phase, the dropout mask is changed and the gradient of masked values is involved in the computation, which introduces noise to the calculation of followup gradients.
Revised: If the input feature passes through the dropout layer before being communicated, during the backward phase, the dropout mask is changed and the gradient of masked values is involved in the computation, which introduces noise to the calculation of followup gradients.

Original: As a result, the dropout layer can only be applied after receiving boundary features.
Revised: As a result, the dropout layer can only be applied after receiving boundary features.